Search Engine YaCy on RaspberryPi - First Image

Ereignisse, Vorschläge und Aktionen

Search Engine YaCy on RaspberryPi - First Image

Beitragvon fherb » So Dez 29, 2013 1:30 am

Hi friends!

There is a thread about use of yacy on RasPi (and CubieTruck; both are low cost ARM systems) since some months: http://forum.yacy-websuche.de/viewtopic.php?f=15&t=3363. After some tests on RasPi I would now publish a first prepared image for Raspberry Pi.

But note: This is not an official "Orbiter"-image. Orbiter promised to prepare a image. But he had not enough time last weeks. Somethings of YaCy are more important as an use on RasPi. I'm only a user of YaCy with some experiences in RasPi and CubieTruck (official page of CubieBoards and CubieTruck: http://cubieboard.org/).

First information:

I used the acutal Wheezy image (also named Raspian) for RasPi from http://www.raspberrypi.org/ in version 2013-12-20 (the Christmas state ;) ). You will find my changes and adds of this image to prepare YaCy below.

I used a fast 4GB SD-card from SanDisk to prepare this image. When you copy this image to another SD-card ist's possible that this image is too big. Please, use any 8GB (or more) card for this image, when you have problems with a 4GB sd-card. The database needs also some place, of course.

It's common known how you can copy the image to the SD-card. I use Windows with Win32DiskManager.

First steps (and some information) after first start of this image:

------------------------------------------------------------------------------------------------

- Put the SD-card into you RasPi and power it on.

- WLAN is not prepared. You has to use keyboard and monitor or ssh terminal via the LAN interface. LAN is configured for DHCP. (Don't forget to allow this device for a connection with your home network in case you has a MAC-configured blocking.) The root user has no password. Start with user "pi" and password "raspberry". This is the configuration of Wheezy. pi is in the admin-group and you can use sudo for root rights.

- enlarge the disk space:

Code: Alles auswählen
sudo raspi-config


1) Expand Filesystem -> Enter -> Ok

All other configurations are not so important for YaCy. You can read some documents and threads for RasPi. And when you want to use RasPi for more than YaCy you can play with it.

Note: You can increase performance via 7) Overclocking.
I use "Medium" on my board. I never heart that the hardware went wrong by using this. But in case of a fault you have to start with the start-image. In most cases the data on sd-card break. But, you will see that the CPU is the bottleneck. So an overclocking can really help a bit to get a better performance.

- After enlarge the file system (and overclocking configuration): reboot via "Finish"ing the configuration tool.

- Now you are prepared to start YaCy. But note, there are 2 steps. The first start used some configuration of the startYACY.sh script. My suggestion is to start as user pi with debug mode:

Code: Alles auswählen
yacy/startYACY.sh -d


at first time. So you can observe the start.

Try http://xxx.xxx.xxx.xxx:8090 with your browser. Take the ip-address which your dhcp server gave the RasPi-board. After a short time you should get the YaCy web page.

- Go to "Administration" and click now on "Shutdown". -> This was the first start. Now the DATA directory was created and the defaults/yacy.init is copied to yacy/DATA/SETTINGS/yacy.conf with all the important configurations inclusive the memory for the jave environment. I set this value (360MB) as a good value for RasPi. So the most of RAM will be used and the Swap is only used in case of some peaks.

- Now you should restart YaCy without the debug argument:

Code: Alles auswählen
yacy/startYACY.sh


- Now feel free to configure your YaCy on Raspberry Pi! (But be carefully with more memory for Java. The Java engine uses some MByte more than what you give by this parameter.)

------------------------------------------------------------------------------------------------

Following some information about my changes in order to prepare YaCy on this RasPi image. This are the same information as in readme.txt in the pi-user-home-directory:

Code: Alles auswählen
Following was done with the Wheezy image:
-----------------------------------------

           fherb, 2013/12/28

The YaCy installation is in folder /home/pi/yacy.
This installation was not yet started. It's fresh.

Please, read the following comments in order to set
your RaspberryPi with YaCy in the best start position.

* For this image I started with a blank Wheezy for RaspberryPi.

* The python-games in pi's home are removed.

* Oracel Java ejre1.7.0_45 is installed in /usr/local/java.
Version:
Java(TM) SE Runtime Environment (build 1.7.0_40-b43)

The binary is in /usr/local/java/bin. This path is also
included in the PATH initialization in /etc/profile.

* Additional installed are htop and iotop.

* With sudo raspi-config the RAM for grafic is reduced to 16MB.
See "Advanced Options -> Memory Split".
So we have now 485MB of RAM for our use.

* After start this image you should start raspi-config and
enlarge the disk space to the full SD card size. After this,
with a 4GB SD-card this image is big enough to do a first test
of yacy inclusive crawling.

* Swapping is switched on by this Wheezy version. But the
swappiness is configured for YACY in /etc/sysctl.conf with

vm.swappiness = 2

2% of 485MB RAM are about 14MB.

So a short peak should not kill a running system. But note,
the swap file has only 100MB.


* Following is changed in /home/pi/yacy/defaults/yacy.init:

javastart_Xmx=Xmx360m
javastart_Xms=Xms360m
disk.free = 300
disk.free.hardlimit = 150

* Following changed in home/yacy/yacy/startYACY.sh:

Argument -server removed from JAVA_ARGS
xmx-argument is redruced from 600MB to 320MB


* The origin gzip-file yacy_v1.66_20131205_9294.tar.gz is in
the home folder of user pi replaced by

yacy_v1.66_20131205_9294_RasPi.tar.gz

So you can refresh the yacy folder to the original state each
time. Only the yacy.init and startYACY.sh was changed in
comparision to the origin distribution file from http://yacy.net

* The WLAN interface is not yet initialized. If you want to use this,
change the /etc/network/interfaces. There are a lot docs to find
in the internet. Additional you have to run raspi-config in order
to enable ssh also for use with WLAN.
But note, the WLAN interface produces CPU load. For such a
program like YaCy, which communicates very much, it's better to
use the LAN interface.


Best regards, whishes and fun,
Frank

Please, report your experiences in http://forum.yacy-websuche.de
Yes, this looks like a German forum. But, of course, you can post
also in English! Use folder "Fragen und Antworten".
In case of questions of use, so use "Hilfe fuer Einsteiger und Anwender".

(My original readme is speaking about a swap partition. Thats false. It's a swap-file.)

The last point:

You will find yacy_v1.66_20131205_9294_RasPi.tar.gz in the home folder of the pi user. This includes the original YaCy in Version 1.66 R9294. Only the file yacy.init and startYACY.sh are changed from me to get a good initialized YaCy on RasPi.

Please, read this readme.txt in case you doesn't speak German. You will find the right folder of this forum to can communicate also in English. That's a German forum but the most important persons of this project can also communicate in English.

And where is this image now?

Please, go to the next posting. In the moment my Windows computer is heavy working to compress this image to a "very small file of some Gigabytes". ;)

Additional information:

The yacy directory in the pi user home contains a fresh an unused YaCy. In order to recreate this fresh directory you can use the yacy tar-zip-file in pi user home. In order to refresh your YaCy do following when you are user pi in your home directory ~:

Code: Alles auswählen
rm -R yacy/*
rmdir yacy
tar xvf yacy_*.gz


After this you can start a fresh YaCy.
Zuletzt geändert von fherb am Mo Dez 30, 2013 2:15 pm, insgesamt 2-mal geändert.
fherb
 
Beiträge: 111
Registriert: Di Nov 26, 2013 10:02 am
Wohnort: Dresden (Germany)

Re: Search Engine YaCy on RaspberryPi - First Image

Beitragvon fherb » So Dez 29, 2013 2:37 am

Image-Download-Link

https://www.dropbox.com/s/6jin632obd6nzc2/2013-12-20-wheezy-raspbian-plus-yacy1.66.zip

-> When all is running, don't forget to update YaCy via web GUI. The used version in this image is some weeks old.

If YaCy doesn't start after such an update, so edit the yacy/startYACY.sh: Remove the -server argument in one of the first rows in this file.

It looks like my Java heap memory configuration of 360MByte is too high. After the start of YaCy (this Image) you should reduce this value to 250 or 270 MByte. (-> Admin Console -> Performance) It's difficulty to say what's the right value because it's only the heap part of working memory what Java needs.

Frank
fherb
 
Beiträge: 111
Registriert: Di Nov 26, 2013 10:02 am
Wohnort: Dresden (Germany)


Zurück zu Mitmachen

Wer ist online?

Mitglieder in diesem Forum: 0 Mitglieder und 2 Gäste

cron