Sudden very high CPU consumption

Keine Scheu, hier darf alles gefragt und diskutiert werden. Das ist das Forum für YaCy-Anfänger. Hier kann man 'wo muss man klicken' fragen und sich über Grundlagen zur Suchmaschinentechnik unterhalten.
Forumsregeln
Hier werden Fragen beantwortet und wir versuchen die Probleme von YaCy-Newbies zu klären. Bitte beantwortete Fragen im YaCy-Wiki http://wiki.yacy.de dokumentieren!

Sudden very high CPU consumption

Beitragvon oneaty » Fr Okt 31, 2014 2:28 pm

Hi,

Today, starting at around 10:00 AM, Yacy suddenly started to voratiously eat CPU.

Currently, I'm at 100%, Load Average of around 3% (and my server only has two cpu's)

By looking at the logs, the only noticeable thing is that from 10:00 AM onwards it only shows the following:

Code: Alles auswählen
W 2014/10/31 11:19:25 NormalizeDistributor adding of decoded rows to workers ended with timeout = 10000
W 2014/10/31 11:19:25 NormalizeDistributor adding of decoded rows to workers ended with timeout = 10000
W 2014/10/31 11:19:25 NormalizeDistributor adding of decoded rows to workers ended with timeout = 10000
W 2014/10/31 11:24:48 NormalizeDistributor adding of decoded rows to workers ended with timeout = 10000
W 2014/10/31 11:26:41 NormalizeDistributor adding of decoded rows to workers ended with timeout = 10000
W 2014/10/31 11:26:41 NormalizeDistributor adding of decoded rows to workers ended with timeout = 10000
W 2014/10/31 11:26:41 NormalizeDistributor adding of decoded rows to workers ended with timeout = 10000
W 2014/10/31 11:26:42 NormalizeDistributor adding of decoded rows to workers ended with timeout = 10000
W 2014/10/31 11:26:44 NormalizeDistributor adding of decoded rows to workers ended with timeout = 10000
W 2014/10/31 11:28:21 NormalizeDistributor adding of decoded rows to workers ended with timeout = 10000


I still didn't resarted Yacy, in case someone wants more up-to-date data
Zuletzt geändert von oneaty am Mo Nov 03, 2014 10:09 am, insgesamt 1-mal geändert.
oneaty
 
Beiträge: 66
Registriert: Mo Feb 04, 2013 12:47 pm
Wohnort: Rio de Janeiro

Re: Very High sudden CPU consumption

Beitragvon oneaty » Fr Okt 31, 2014 2:54 pm

At this exact moment, Yacy started logging some of its usual messages, but then other messages that seem to indicate a serious internal issue, like some java timeouts, then changed my type from senior to junior and a bunch of other messages
oneaty
 
Beiträge: 66
Registriert: Mo Feb 04, 2013 12:47 pm
Wohnort: Rio de Janeiro

Re: Very High sudden CPU consumption

Beitragvon Orbiter » So Nov 02, 2014 1:21 pm

while the "NormalizeDistributor adding of decoded rows to workers ended with timeout = 10000" message looks actually scary, I could not find a bug there. But I found another one which currently reproducible creates 100% CPU load: the new network history graph. Hold on, I first fix this;
Orbiter
 
Beiträge: 5778
Registriert: Di Jun 26, 2007 10:58 pm
Wohnort: Frankfurt am Main

Re: Very High sudden CPU consumption

Beitragvon Orbiter » So Nov 02, 2014 1:45 pm

emergency fix in 1.81.9135; may not apply to your problem, but this problem is there anyway since the new network/index statistic graphics had been made (ca. 1 week ago).
Orbiter
 
Beiträge: 5778
Registriert: Di Jun 26, 2007 10:58 pm
Wohnort: Frankfurt am Main

Re: Very High sudden CPU consumption

Beitragvon oneaty » So Nov 02, 2014 9:22 pm

ok, sorry about the late answer.

Here's what's happening from the server cpu consumption standpoint:

Bild

The issue I initially reported is clearly visible in the green (Nice CPU usage) spike.

Late in the evening, I decide to reboot the server with shutdown -r. I expected that this command would shutdown Yacy with service yacy stop, since my Yacy client startup/shutdown is automated by a LSB Init script.

But it wouldn't stop after half an hour. So I decided to force the server shutdown with my finger.

When I restarted the server, it tried to start Yacy automatically, but it didn't succeed.

Today, I manually simulated an attempt to start Yacy by issueing ./ startYACY -d > logyacy, and the log and screen output are:

1 - Log - https://drive.google.com/file/d/0B8FDexrYy_FKQTcybTRDWGk2Rnc/view?usp=sharing

2 - screen output - https://drive.google.com/file/d/0B8FDexrYy_FKNkRSalVPbnU2S2s/view?usp=sharing

What should I do?
oneaty
 
Beiträge: 66
Registriert: Mo Feb 04, 2013 12:47 pm
Wohnort: Rio de Janeiro

Re: Sudden very high CPU consumption

Beitragvon oneaty » Mo Nov 03, 2014 5:35 pm

I made some progress but I still need advice.

The progress: now I can access Yacy web front-end.

What I did?

By looking at the logs I sent, I suspect that this could be a memory issue, so I changed these two yacy.conf parameters:

Code: Alles auswählen
javastart_Xmx=Xmx1000m
javastart_Xms=Xms300m


The initial values for those were 600m and 90m, respectively.

Also, I'm almost certain that all these issues began after I started a crawl on a very big and popular site. Now that my Yacy server is back, that crawl restarted again. If my suspicion is correct, what should I do to prevent that it harms the server once more? What signs should I look for?

Also, by looking at the log of this run, I noticed this:

Code: Alles auswählen
I 2014/11/03 13:56:31 SWITCHBOARD postprocessing deactivated: no enough ram (148797808), needed 536870912, to force change field postprocessing.minimum_ram
I 2014/11/03 13:56:31 SWITCHBOARD postprocessing deactivated: too high load (5.3) > 2.5, to force change field postprocessing.maximum_load
I 2014/11/03 13:56:31 SWITCHBOARD postprocessing deactivated: constraints violated


Is this something that needs action? What is the SWITCHBOARD?
oneaty
 
Beiträge: 66
Registriert: Mo Feb 04, 2013 12:47 pm
Wohnort: Rio de Janeiro

Re: Sudden very high CPU consumption

Beitragvon oneaty » Mo Nov 03, 2014 6:11 pm

oneaty hat geschrieben:Also, I'm almost certain that all these issues began after I started a crawl on a very big and popular site. Now that my Yacy server is back, that crawl restarted again. If my suspicion is correct, what should I do to prevent that it harms the server once more? What signs should I look for?


Trying to answer my own question "What signs should I look for?", I think these graphs give a hint:

Bild

The monthly graph shows a clear decrease of free memory starting at the end of week 43, exactly when I started the crawl on that huge web site.

The yearly graph is definitive in showing that only in October the free memory presented a significant decrease.

So, are these graphs the ones that I should monitor to know when to either stop crawling, stop Yacy and increase memory allocation (but this is a limited option), or take whatever action needed that will allow to adjust Yacy to a unusual burden on the fly?
oneaty
 
Beiträge: 66
Registriert: Mo Feb 04, 2013 12:47 pm
Wohnort: Rio de Janeiro

Re: Sudden very high CPU consumption

Beitragvon oneaty » Mo Nov 03, 2014 6:47 pm

According to Sixcooler, in another forum thread(http://forum.yacy-websuche.de/viewtopic.php?p=27723#p27723),

Yes, YaCy needs a lot of RAM (heap) if the index grows.
And since we use solr for index-storage, there is also a heavy usage of virtual RAM.
Virtual RAM does not need to be available physically, but needs to be addressable - that's why 64Bit is the way to go.


Considering that the server where Yacy is running is an Ubuntu 14.04 Server (64 bit), has 2 GB RAM and 2GB of Swap, can I increase javastart_Xmx and javastart_Xms above the 2GB limit of RAM and use up to 4 GB (2 GB RAM + 2 GB Swap)?

Based on Sixcooler words, I would expect to see a heavy usage of my server's swap partition, but that's not what these graphs show:

Bild

Should some setting be turned on so that solr makes use of virtual RAM?
oneaty
 
Beiträge: 66
Registriert: Mo Feb 04, 2013 12:47 pm
Wohnort: Rio de Janeiro


Zurück zu Hilfe für Einsteiger und Anwender

Wer ist online?

Mitglieder in diesem Forum: 0 Mitglieder und 2 Gäste

cron