Memory problems

Hier finden YaCy User Hilfe wenn was nicht funktioniert oder anders funktioniert als man dachte. Bei offensichtlichen Fehlern diese bitte gleich in die Bugs (http://bugs.yacy.net) eintragen.
Forumsregeln
In diesem Forum geht es um Benutzungsprobleme und Anfragen für Hilfe. Wird dabei ein Bug identifiziert, wird der thread zur Bearbeitung in die Bug-Sektion verschoben. Wer hier also einen Thread eingestellt hat und ihn vermisst, wird ihn sicherlich in der Bug-Sektion wiederfinden.

Memory problems

Beitragvon snadno » Di Apr 03, 2012 8:50 am

Hi I've been running yacy for a while... its been getting bigger, my index now at 21,000,000 Data at 34Gb, but I often have problems restarting, with out of memory problems from the JVM... I'm using openJDK 6... I was using the sun/orical but reached the limit it seemed and it was getting really slow, switching to openJDK everything sped up and worked better, but that was back at 19,000,000 words. As the index gets bigger i keep running into these problems. I'm running SolusOs RC3 (based from Debian Squeeze) Linux 32b version, have 4G memory and I5.

I've got Xms and Xmx set as 1450m... and can't seem to start up the server with anything higher. Any advice from people with larger indexs would be really helpful. I'm running yacy 1.02/9016

Thanks
N
snadno
 
Beiträge: 2
Registriert: Di Apr 03, 2012 8:35 am

Re: Memory problems

Beitragvon Vega » Di Apr 03, 2012 1:12 pm

Hallo,

"SolusOs RC3 (based from Debian Squeeze) Linux version 32b, have 4G memory and I5" - this is exactly the problem. Under a 32-bit operating system, you can assign a client process no longer than 2 GB of memory. Please proceed to a 64 bit operating system and a 64 bit version of Java.

Grettings from Dresden, Germany.
Thomas
Vega
 
Beiträge: 824
Registriert: Mi Jun 27, 2007 3:34 pm
Wohnort: Dresden

Re: Memory problems

Beitragvon snadno » Mi Apr 04, 2012 8:10 am

Thanks, I was hoping that not the problem, as SolusOS and Bodhi linux don't yet have 64 bit versions... But I'll give it a try for a few days with linuxMint 64big

Kind regards
Neil

Vega hat geschrieben:Hallo,

"SolusOs RC3 (based from Debian Squeeze) Linux version 32b, have 4G memory and I5" - this is exactly the problem. Under a 32-bit operating system, you can assign a client process no longer than 2 GB of memory. Please proceed to a 64 bit operating system and a 64 bit version of Java.

Grettings from Dresden, Germany.
Thomas
snadno
 
Beiträge: 2
Registriert: Di Apr 03, 2012 8:35 am

Re: Memory problems- Me Too

Beitragvon as_above_so_below » Do Aug 14, 2014 1:12 pm

(I am still using xp32bit out of habit.)
I generally like to set it and forget it. (let yacy run 24/7)
But...
After several days continuous running all my computer programs become slow and then non functional; requiring restart.

in earlier versions of yacy uptime was not such a problem.

I tried using 64bit windows and linux some time ago but found them lacking in various ways (compatible drivers for example)and so I switched back to 32bit. That was some time ago.

The question would naturally become. What OS set up will be best suited for running yacy?

I am probably just like the next gal who just wants better results.

Give us some pointers on best practices so that we might engage on the bleeding edge.



Thank you for playing.
as_above_so_below
 
Beiträge: 1
Registriert: Do Aug 14, 2014 12:36 pm

Re: Memory problems- Me Too

Beitragvon David » Mi Aug 20, 2014 6:14 pm

as_above_so_below hat geschrieben:I tried using 64bit windows and linux some time ago but found them lacking in various ways (compatible drivers for example)and so I switched back to 32bit. That was some time ago.

The question would naturally become. What OS set up will be best suited for running yacy?

In my opinion, YaCy runs pretty good on Linux Mint, and regarding the drivers, it mostly works out of the box. At the moment, it's one of the most popular and easiest to use linux distros.

http://www.linuxmint.com/
https://en.wikipedia.org/wiki/Linux_Mint
David
 
Beiträge: 170
Registriert: Di Mär 05, 2013 5:35 pm

Re: Memory problems

Beitragvon fherb » Di Nov 25, 2014 11:03 pm

Hello,

I would like to move this topic again back as an actual agenda item.

Last winter time I checked Yacy at different systems and now I restartet this process. The main reason is to use a vServer with a blog also as a YaCy system. The blog doesn't use the power of two Intel E5 cores, 2/4GByte RAM and 200GByte HDD. The second is to let run YaCy at home. In combination of an desktop (crawling and indexing during it is on) and an ARM system to host the solr and to host the YaCy user interface for search questions and Index/DHT-Transfer with other Peers.

My problem is that after some days/weeks of work the Java system with YaCy begins to produce out of memory problems. I thought to understand that YaCy stops crawl and DHT-receive in case the memory runs short. But it looks like that it isn't so:

I tried the last two days to adjust the RAM disable DHT-in below value at the admin page Performance_p.html (OS is Ubuntu)

I can set the value to a very, very small value. Also to a middle value (half of the physical RAM) or to a value higher than my physical memory space is. Never YaCy stops the DHT-in! I don't know if YaCy has also to stop crawling in case of to less RAM. But the trigger für DHT-in came never. (used with the actual release)

The second question is more principle: How I have to adjust the memory parameters?

For example:


* My system should have 2GByte RAM. (It's not much, but in this case only theoretical and a real value in comparison of the standard of 600MB for YaCy-Java-Runtime-Environment after installation.)
* I would like be sure that my operating system inclusive all small tasks and admistrations tools has ever 400MByte for this purpose. So YaCy inclusive solr should never use more RAM than 1,6GByte.

How I have to adjust the parameter of YaCy to get such a configuration?

In my current opinion (as an example),

* I would set the xmx-value in the yacy.conf to about 1200MByte. I saw that we have to calculate, that Java will use about 20% more than the value of xmx. Why ever.

* Now, to be save with this configuration, I would set the RAM disable DHT-in below value at 1000MByte. So I hope that Yacy will stop to increase the DHT-Table in RAM at this moment where it starts to need more than 1GByte RAM. My hope would be that YaCy would have a RAM-buffer of additional 200Mbyte starting with the DHT-in-stop (1GByte) until the worst value for the Java engine of 1,2GByte in order to keep stable.

But the last time shows that either I missunderstood this all or somthing doesn't work like as provided.

I think, somethig would help the user if the adjustment of the parameters for special RAM configurations would be explained in the Wiki.

May somebody help?

Best regards,
Frank
fherb
 
Beiträge: 111
Registriert: Di Nov 26, 2013 10:02 am
Wohnort: Dresden (Germany)

Re: Memory problems

Beitragvon fherb » Mi Nov 26, 2014 12:17 am

...additional to the last post, following observation:

Yacy runs in this case at a RaspberryPi since some days. Without crawling and without solr. This Yacy is connected to an other solr at an other Linux system. This Yacy runs really fine and the response times at the user web front end is suprising good, but:

- Without Yacy, the OS inclusive a running htop needs 82MByte of RAM.

- If I look with htop and subtract this basic load from the current used 378MByte of RAM, so Yacy uses in this moment 296MByte of physical RAM (484MByte).

- xmx was set to 220MByte

The page status.html shows:
- RAM used 92,12MB
- RAM max 212,75MB

At performance_p.html the parameter "RAM disable DHT-in below" was set to 150MB. The physical RAM of this system is 484MByte but the DHT-in disable is not yet triggered.

How I have to understand these values?

Does it be important that this system knows a swap file of 999MByte? (but it uses in this moment only 2MByte of this)

How are the really coherences between these memory values?

Best regards, Frank
fherb
 
Beiträge: 111
Registriert: Di Nov 26, 2013 10:02 am
Wohnort: Dresden (Germany)

Re: Memory problems

Beitragvon zottel » Fr Apr 17, 2015 9:05 pm

Not a direct answer to the last comments, but a general observation:

At least when run with OpenJDK, memory management seems to work better with v7 than with v6 (of OpenJDK).

On my old VPS, I had come to the limit where the database had grown so large that yacy wouldn't even start. As I had already planned to move to a larger VPS, I just stopped my node for a while. Shortly before I actually moved to the new VPS, I realized that I had still openjdk6 installed and moved to openjdk7.

With v7, yacy did start without problems. I didn't test if it lasted long at that time, but it was an unexpected improvement just by moving to the newer openjdk.

On my new VPS, I can now dedicate ~7 GB of RAM to yacy alone, so everything is working like a charm, of course. :-)
zottel
 
Beiträge: 51
Registriert: Mi Jan 16, 2013 3:04 pm


Zurück zu Fragen und Antworten

Wer ist online?

Mitglieder in diesem Forum: Yahoo [Bot] und 1 Gast

cron