Yacy on an Olinuxino board

Discussion in English language.
Forumsregeln
You can start and continue with posts in english language in all other forums as well, but if you are looking for a forum to start a discussion in english, this is the right choice.

Yacy on an Olinuxino board

Beitragvon layst » Di Okt 28, 2014 10:34 pm

Hi everybody,

I come to you today about running Yacy on Olimex boards. I have been running Yacy on my laptop for a little time now, but I want to switch to a permanent and low energy consuming solution in order to index more pages and have my contribution available at any time. I have experienced the Olimex Olinuxino board for another project so I figured out that it could be a good choice to run Yacy 24/7.

However Yacy keeps on crashing on my Olinuxino board. I thought that the hardware could be the cause, but the board I am using is the A20-LIME2 (dual core A7 at 1GHz with 1GB DDR3) and Yacy is the only app I intend to run on it, so I guess that the technical specs are sufficient for my need, since Yacy could be run on less powerful Raspberry Pi systems.

I also noticed (from the log) that Yacy always dies when performing the action matching the following log line:
org.apache.solr.update.SolrIndexConfig IndexWriter infoStream solr logging is enabled

I have seen from my laptop that the next step would be to intialize the data directories, but these directories do exist on the Olinuxino board (they are the same as on my laptop, including their permissions).

Right now I am stuck, but I still want to contribute actively to this very nice project. Hence it would be great if you had any suggestions.

Cheers,
layst
 
Beiträge: 11
Registriert: Di Okt 28, 2014 7:59 pm

Re: Yacy on an Olinuxino board

Beitragvon smokingwheels » Mi Okt 29, 2014 11:09 am

layst hat geschrieben:I have been running Yacy on my laptop for a little time now, but I want to switch to a permanent and low energy consuming solution in order to index more pages and have my contribution available at any time.

I switched to a VPS on https://my.vultr.com for $8.00 USD/month with a power saving of approx $40/month powering my pc at home.
My Yacy now has an age of 25 days and is 5 GB.
I read some where if you backup the DATA folder and transfer it to another machine of a fresh install you have every thing in your index again, I will have to look into that though.
I use Debian 7 64bit and have a robot keyboard that types all the commands in A typical Yacy install takes 4 min.
https://www.youtube.com/watch?v=UQzC9DvBDf0
I can help you set it up for free just PM me. Try my VPS Yacy at http://smokingwheels.evils.in:8090
smokingwheels
 
Beiträge: 130
Registriert: Sa Aug 31, 2013 7:16 am

Re: Yacy on an Olinuxino board

Beitragvon layst » Mi Okt 29, 2014 4:08 pm

I read some where if you backup the DATA folder and transfer it to another machine of a fresh install you have every thing in your index again, I will have to look into that though.


I did save my DATA folder from the laptop :) I guessed that it should be the case.

I switched to a VPS on https://my.vultr.com for $8.00 USD/month with a power saving of approx $40/month powering my pc at home.


Renting a VPS could be a solution. I might switch to that if I cannot achieve setting up Yacy on my Olinuxino board. I shall come to you then, if I meet any problems :P thank you!

Right now I still would like to have my installation at home, to save a little money (especially in a long-term perspective) and to be able to upgrade easily my installation, for instance by replacing the HDD from 320 Gb to 1 Tb or whatever when it is close to be full. Maybe I am mistaken, but I feel like the needed storage can increase quite quickly (my DATA folder is about 8 Gb after less than 10 searches), and I would not like to depend on an external service to upgrade it.
layst
 
Beiträge: 11
Registriert: Di Okt 28, 2014 7:59 pm

Re: Yacy on an Olinuxino board

Beitragvon smokingwheels » Do Okt 30, 2014 3:55 am

DATA folder is about 8 Gb after less than 10 searches

You transfered the DATA folder how many Documents show up on the Olinuxino board install?
Maybe a reindex is needed /IndexReIndexMonitor_p.html.

Yacy keeps on crashing on my Olinuxino board

See link for automatic restart instructions schedule, just point the steering url to IP and port add /Steering.html?restart= to it.
http://mantis.tokeek.de/view.php?id=414
smokingwheels
 
Beiträge: 130
Registriert: Sa Aug 31, 2013 7:16 am

Re: Yacy on an Olinuxino board

Beitragvon layst » Fr Okt 31, 2014 3:37 pm

smokingwheels hat geschrieben:
DATA folder is about 8 Gb after less than 10 searches

You transfered the DATA folder how many Documents show up on the Olinuxino board install?
Maybe a reindex is needed /IndexReIndexMonitor_p.html.

Yacy keeps on crashing on my Olinuxino board

See link for automatic restart instructions schedule, just point the steering url to IP and port add /Steering.html?restart= to it.
http://mantis.tokeek.de/view.php?id=414


Both of these implie to get to the point where yacy actually run, at least for a few minutes, right? My problem is that I don't even get there; the process crashes before it can set the server up (from what I gathered).

Sorry if it was not clear enough.
layst
 
Beiträge: 11
Registriert: Di Okt 28, 2014 7:59 pm

Re: Yacy on an Olinuxino board

Beitragvon smokingwheels » Sa Nov 01, 2014 12:46 pm

Ask questions in forums that deal with the Olinuxino board. eg https://www.olimex.com/forum/
Contact Olinuxino Developers also.

Can you unplug your USB Drive and receive the Yacy log files on another PC?
If so I suggest you can report a bug report at http://mantis.tokeek.de/main_page.php for future reference.
Note: After you report a bug you have to view it and then click Monitor button to get email updates.

I'm really interested in it to have approx 1.2 Watt/hour Yacy Search Engine at home.
smokingwheels
 
Beiträge: 130
Registriert: Sa Aug 31, 2013 7:16 am

Re: Yacy on an Olinuxino board

Beitragvon smokingwheels » Sa Nov 01, 2014 2:13 pm

I was looking into getting one but I was reading the specs and they only 512 mb of RAM for $85 board.
If you have not made a system Swap file yet for it yet then I suggest you find out how to.

My first attempt on a 512 mb system kept crashing until I made a system swap file, It crashed again not so often though, so I sprung the extra $2.00 /month for a 1 GB of ram its 27 days old now.
smokingwheels
 
Beiträge: 130
Registriert: Sa Aug 31, 2013 7:16 am

Re: Yacy on an Olinuxino board

Beitragvon layst » Sa Nov 01, 2014 3:36 pm

I might ask Olimex about this issue, juts like I can fill a bug report.

About the hardware, the board I am using (A20-Lime2, as said in the first post) cost me something like 65€ including VAT, shipping and SATA cables. In addition to the 1Gb of RAM, I set up 2Gb of SWAP, just in case.

But before doing anything, I will try another thing. I have noticed this morning that my laptop kernel is 3.14, while the one shipped on Raspbian nowadays is 3.12. Yacy seems to run on both. My Olinuxino board has only a 3.4 kernel; so I will try to upgrade it or compile a kernel (3.12 minimum) to see if it can solve the issue.

In any case, I will let you know about this try, and will fill a bug report and/or contact Olimex, depending on my findings.

Thanks for your links, especially the one to fill a report.

EDIT: it seems there are no images available for my boards with kernel 3.12 or higher. Since I will not have much time to work on it in the next days, I have sent an email to Olimex. I shall keep you informed.
layst
 
Beiträge: 11
Registriert: Di Okt 28, 2014 7:59 pm

Re: Yacy on an Olinuxino board

Beitragvon layst » Do Nov 06, 2014 1:43 pm

As promised, here is the relevant part of the email I had from Olimex (I got it yesterday (5th November) in the morning):
Unfortunately, I don’t see how Olimex can be of any help with the issue you are facing. If the creators of the Yacy server are unable to give instructions for installation under Debian 3.4.90+ then I would suggest that the default images provided by Olimex are compiled as armhf and Yacy seems to only work under armel distributions (especially since you probably used armel distribution for the RPi that ran Yacy successfully). Something more, since Yacy is written in java the platform also needs to be suitable for hard-float – there are separate java distributions for soft-float and hard-float.

Make sure you are using the proper java and the settings of yacy are set properly. If there is no java installed nothing would happen.

To be more specific why RPi uses armel and OLinuXino armhf distributions – there are hardware reasons for that – the older ARMv6 processors had no support for hard float point; while the OLinuXino is ARMv7 and there is such support. The CPU in the Raspberry Pi implements the ARMv6 ISA (with VFP2) and is thus incompatible with the Debian armhf port baseline of ARMv7+VFP3 and ARM hardware-floating-point ports for other distributions, which all have the same baseline.

The only ways out would be:

1. Find a similar program that works under Debian 3.4.90+ and armhf (hardware-floating) – quite unlikely to find stable alternative to Yacy

2. Ask around the Yacy community to provide better armhf support

3. Compile an OLinuXino image with soft-float

The 3.15 kernel support you saw is incomplete and it is created by an user. It is not an official release. The information might be found at this blog post: http://olimex.wordpress.com/2014/06/30/ ... rnel-3-15/ We are updating the kernel in the official releases when it reaches a stable state. The progress for each kernel version might be found here: http://linux-sunxi.org/Linux_Kernel

Best regards,
Lub/OLIMEX

------------------------------------------------------
From: Bertrand HER
Sent: Saturday, November 01, 2014 4:56 PM
To: support@olimex.com
Subject: Support Question

Hello,

I write to you today about some issues I have with your boards. I have, at the moment, three of your boards: iMX233-MINI, A13 and A20-Lime2. I use the A13 as a server perfectly, and tried to use the iMX233 as a Yacy server. I could not achieve it, since the server kept on crashing before it even started. But this software run on a raspberryPi, which has better hardware spec than iMX233. Hence I tried to run Yacy on my A13, but it also failed. I then ordered a A20-Lime2 to get rid f the hardware specs uncertainty, but Yacy just crash the same as on less powerful boards.

I noticed that I could run Yacy on my laptop. Since my laptop has a kernel 3.14 and the raspbian distribution has a 3.12 kernel (whiile iMX233 is only 3.11) I wondered if it could be the cause.

I opened a thread on the Yacy forums; it can be useful if you want some more details.

Do you think it possible to deliver a Debian image based on kernel 3.12 or higher for the Lime2? I saw that you did a 3.15 kernel based image for A10-Lime, so I guess it would be possible (and easier, I am not used to compile kernels) for you to deliver it.


As I said before, I will not have much time to spend on this topic for a few days at least, so for now I leave it the way it is. What I am planning to do next:
* try to install the softwares from jessie repositories if possible: I noticed that Yacy would not run on a fresh install of Debian Wheezy on anthoer laptop, but would once getting upgrades from jessie repos. It did not throw the same error, but make a try will not hurt.
* find or compile an Olinuxino image with armel instead of armhf.

Once again, if I come to any relevant results, I shall let you know.
layst
 
Beiträge: 11
Registriert: Di Okt 28, 2014 7:59 pm

Re: Yacy on an Olinuxino board

Beitragvon layst » So Nov 09, 2014 6:59 pm

Hi,

I finally had my board working for Yacy. As in many problems the issue was not far from under my nose.

The issue had nothing to do with the kernel version, the type of used compilation (hard float or soft float, from what I have understood) or Debian version (wheezy or jessie). In fact the problem was the Java virtual machines that were available. Indeed a
java -version
in console on my board would give
OpenJDK Runtime Environment (IcedTea 2.5.3) (7u71-2.5.3-1+b1)
OpenJDK Zero VM (build 24.65-b04, mixed mode)

while my laptop gives
OpenJDK Runtime Environment (IcedTea 2.5.3) (7u71-2.5.3-1+b1)
OpenJDK 64-Bit Server VM (build 24.65-b04, mixed mode)


Given the second line and the fact that Yay use a server, it could very be the problem, and it proved to be it.

So finally, the steps I followed to get it working (from a fresh Debian install on an Olinuxino board):
#1 add Yacy to the sources.list file (cf Debian install from the wiki: http://www.yacy-websuche.de/wiki/index.php/En:DebianInstall)
#2 apt-get update && apt-get install yacy (installing yacy will pull openjdk-7-jre-headless, we shall replace the link in /usr/bin/, so we have to do it before installing the Sun's package of Java)
#3 install the Sun's package of Java from their website, like described on the wiki page for an install on a Raspberry Pi (http://www.yacy-websearch.net/wiki/index.php/En:Raspberry_Pi). I picked the 7.71 version of Java (filename as of today : ejre-7u71-fcs-b14-linux-arm-vfp-hflt-client_headless-26_sep_2014.tar.gz). When making the symbolic link in '/usr/bin/', first move the binary from openjdk. I do not know enough but I moved it so that I can put it back easily if OpenJDK was needed again.

A last note about starting from a previous index. I had an 8 Gb folder from previous experiments of Yacy, and I could use it easily again, but I was tricked in the first place by permissions. So do check that this folder is owned by Yacy. This also implies a slightly slower start the first time.

I now run Yacy on a Olinuxino A13 board, which has only 256Mb of RAM. I have put something like 2Gb of SWAP but it might completely irrelevent, since it will write a lot on the SD card. I might tel you in a few months if my set up has become slower, or if it is not sufficient to run Yacy, or on the contrary if it is okay.

Thanks you who answered me, and to Olimex who put me on the rigth track by speaking of Java versions.

I will ask to edit the wiki so that it is easier for other who would try the same as I did.

Cheers!

EDIT (14/12/2014): I now have run Yacy on my board for a little more than a month. I quickly switched back to the A20-Lime2, the A13 was not sufficient (regarding RAM). I do not run anything else on this board, and I added a 1TB hard drive, powered by an external power supply. I am not expert in Yacy and I did not take much time to optimize my settings, so long as it worked without crashing every hour. Now it stands easily weeks without restarting and allows me to have my contribution done without paying much more attention (performing only a few crawling from time to time). I might learn later to fine tune my install.

Till then, thanks again to the community and the developpers for this great piece of software.
layst
 
Beiträge: 11
Registriert: Di Okt 28, 2014 7:59 pm


Zurück zu English

Wer ist online?

Mitglieder in diesem Forum: 0 Mitglieder und 3 Gäste