Suspend crawler during a time-window every day

Discussion in English language.
You can start and continue with posts in english language in all other forums as well, but if you are looking for a forum to start a discussion in english, this is the right choice.

Suspend crawler during a time-window every day

Beitragvon davide » So Mai 24, 2015 3:02 am

Is it possible to configure the process scheduler to periodically pause the crawler(s) during a specific time-window, every day?
Looking at my /Table_API_p.html, I see listed in the table the entry for starting the crawler, as it was automatically recorded from my manual interactions onto /CrawlStartExpert.html. (thank you orbiter for teaching me this :) However there's no entry for stopping it, even if I did manually stop the crawler from /Crawler_p.html.

Also, I noticed that every time I stop the crawler, YaCy seizes a serious amount of upload bandwidth for several minutes while the number of entries in the local crawler queue gradually reduce from a few millions to zero, by which point the stopping procedure appears to be complete. So, by this, I'm not even sure if stopping the crawler for two or three hours a day on a specific time-window would be effective at preserving the bandwidth for other purposes.
Maybe it is possible to just slow down the crawler periodically, without stopping it completely?
Beiträge: 78
Registriert: Fr Feb 15, 2013 8:03 am

Re: Suspend crawler during a time-window every day

Beitragvon Orbiter » Mi Mai 27, 2015 11:05 pm

there is no slow-down scheduler-option built-in, however, you can slow down using the PPM setting in /Crawler_p.html
This slow-down setting might be triggered using a cronjob; however, I never tried that. But that should work and serve your demand.
Beiträge: 5771
Registriert: Di Jun 26, 2007 10:58 pm
Wohnort: Frankfurt am Main

Zurück zu English

Wer ist online?

Mitglieder in diesem Forum: Giorgos und 1 Gast