CPU usage (GNU, like Slackware64, old PC) 100%--can I limit?

Keine Scheu, hier darf alles gefragt und diskutiert werden. Das ist das Forum für YaCy-Anfänger. Hier kann man 'wo muss man klicken' fragen und sich über Grundlagen zur Suchmaschinentechnik unterhalten.
Forumsregeln
Hier werden Fragen beantwortet und wir versuchen die Probleme von YaCy-Newbies zu klären. Bitte beantwortete Fragen im YaCy-Wiki http://wiki.yacy.de dokumentieren!

CPU usage (GNU, like Slackware64, old PC) 100%--can I limit?

Beitragvon dchmelik » Fr Mai 27, 2016 5:27 am

I've been trying YaCy a few years on Slackware64, wanting to make public node. However, YaCy eventually uses 100% or maybe even '101%' CPU resources, according to the 'top' command. I have YaCy on Slackware64 14.1 GNU/Linux (stable, but with official updates) with AMD Athlon X2 5600+ (about 2x2.9, or 5.8 GHz) (that I wasn't running much else on but HTTPD for a rather plain, low-traffic site) and a want to try it on a virtual private server with newer, Intel E5-2860 (but only 1x2.8 GHz) running several other programs. It's okay if YaCy uses most my bigger server's CPU (preferably not all) but of course shouldn't on the less powerful one. YaCY uses so much crawling my sites, but then doesn't usually decrease usage. So, I'm wondering how to limit YaCy's system resource usage without crashing.

I've edited the version number in slackbuilds.org's JDK build script, so installed JDK 8u92, but haven't tried it for YaCy yet (maybe other stuff) as the intense usage happened for years and didn't seem merely a Java problem.

Someone said use Docker. It just seems too complicated, and the version someone already set up wouldn't use my configuration. Is there a simpler way, perhaps built-in to a recent YaCy version?

I saw a similar Windows thread but didn't want to 'hijack' that. I don't understand much Deutsch, but can copy your replies into translate.google.com (like translate.google.de), and people can discuss other POSIX-based operating systems (GNU/Linux, etc.) and hardware in this thread. It'd be interesting to see how to do the same on BSDs, though I don't use those so much anymore.
dchmelik
 
Beiträge: 3
Registriert: Fr Mai 27, 2016 4:43 am

Re: CPU usage (GNU, like Slackware64, old PC) 100%--can I li

Beitragvon luc » Fr Mai 27, 2016 9:09 am

Hi, maybe you can make use of cgroups. What's more it is actually used by docker to control containers resources, and also by Systemd for example.

For now I didn't really played with it, and I am not sure if is easier to use it directly or to learn using Docker tools. At least cgroups is a kernel feature, so it is surely not loosing time to learn using it...

PS : I experimented with YaCy on Docker recently and I think it is not as complex as it may look. A Dockerfile is even now on YaCy main repository, and is used to build a docker image from latest YaCy sources here.
luc
 
Beiträge: 276
Registriert: Mi Aug 26, 2015 1:04 am

Re: CPU usage (GNU, like Slackware64, old PC) 100%--can I li

Beitragvon luc » Mo Jul 18, 2016 9:59 pm

Hello, for information I performed some tests trying to control YaCy CPU load on Debian with Systemd. So, according to SystemD documentation, and because currently Debian services automatically rely on SystemD, it should be very easy to set a maximum limit on YaCy CPU load using this control configuration : "CPUQuota".

Indeed it is, and it worked very fine with only one command line (as root):
Code: Alles auswählen
systemctl set-property yacy CPUQuota=30%


You can set any value you wish to CPUQuota, it is immediately used, and saved for eventual later restart. I checked this with a YaCy peer crawling, using "top" command. Note : the value meaning is not so obvious if your processor has multiple cores. For example with 4 cores :
- CPUQuota=400% : allow full load of all cores
- CPUQuota=200% : allow at most full use of two cores
- CPUQUota=50% : allow at most 50% use of one core

But : currently (2016-07-18) this only works on Debian Stretch (testing) which has systemd version 230 (you can check with "systemctl --version").
On Debian Jessie (stable, embedding systemd version 215) this should works, but for some reason it does not! I guess the CPUQuota feature is quite young, and maybe has some bug in this version (only appear starting from systemd version 213).
luc
 
Beiträge: 276
Registriert: Mi Aug 26, 2015 1:04 am

Re: CPU usage (GNU, like Slackware64, old PC) 100%--can I li

Beitragvon dchmelik » Mi Aug 03, 2016 7:14 am

Thanks, I'll try one or both of those things... but I thought people would be discussing similar operating system (OS) distributions (distro.) Slackware is a very Unix-like OS/distro, since it uses sysvinit; Debian is not anymore, because it uses systemd.
dchmelik
 
Beiträge: 3
Registriert: Fr Mai 27, 2016 4:43 am

Re: CPU usage (GNU, like Slackware64, old PC) 100%--can I li

Beitragvon sixcooler » Mi Aug 03, 2016 7:30 pm

Hi,

if you want to limit the System-usage of YaCy on a Linux-System you can use a build-in feature to set the max systemload at Performance Settings of Queues and Processes (/PerformanceQueues_p.html) on the right column:
There you can set for each task of YaCy a maximum Systemload - of the system has an higher load this task stops untill the load is lower than the value.

For eg. I use a Max Systemload of 1.0 for Local crawl - whenever there is load from other processes, or tasks of yacy this stops and the System has recources left for other things.
The Systemload is much more precise than the cpu-usage for tuning in order to have an always usable system. On the other hand this is not available for Windows-Systems.

Cu, sixcooler.
sixcooler
 
Beiträge: 487
Registriert: Do Aug 14, 2008 5:22 pm

Re: CPU usage (GNU, like Slackware64, old PC) 100%--can I li

Beitragvon luc » Fr Aug 05, 2016 8:38 am

Ah yes thanks sixcooler! Again YaCy had an already available feature, one just has to know where to find it ;)
Effectively these options provide finer tuning inside YaCy. I am playing a little with them and it looks like they do the job on Debian Linux. But on my machine they do not prevent high CPU peak usages... Not really a problem, but when you wish to have a hard CPU usage limit, OS specific features like cgroups on Linux may be more reliable. Of course the best is to test what fits the most one's needs.
luc
 
Beiträge: 276
Registriert: Mi Aug 26, 2015 1:04 am


Zurück zu Hilfe für Einsteiger und Anwender

Wer ist online?

Mitglieder in diesem Forum: 0 Mitglieder und 1 Gast

cron