Dokumentation des Expertcrawl Formular

Forum for developers

Dokumentation des Expertcrawl Formular

Beitragvon Micki » So Aug 07, 2016 12:49 pm

Gibt es eine Dokumentation für das Expert Crawlformular oder eine Api um anderen Anwendungen das Einstellen von Crawls zu ermöglichen?
Micki
 
Beiträge: 86
Registriert: Sa Feb 21, 2015 10:38 pm

Re: Dokumentation des Expertcrawl Formular

Beitragvon Micki » So Aug 07, 2016 4:05 pm

Hat sich erledigt. Bin fündig geworden:

http://www.yacy-websearch.net/wiki/inde ... APICrawler
Micki
 
Beiträge: 86
Registriert: Sa Feb 21, 2015 10:38 pm

Re: Dokumentation des Expertcrawl Formular

Beitragvon luc » Mo Aug 08, 2016 5:40 pm

Hi Micki, I also initially missed this page. So I added a link in the Dev:API wiki page.
If you see other relevants pages which should link to this doc, dont' hesitate to update it.
luc
 
Beiträge: 300
Registriert: Mi Aug 26, 2015 1:04 am

Re: Dokumentation des Expertcrawl Formular

Beitragvon Micki » Di Aug 09, 2016 8:48 pm

The Question is, how many crawljobs can be insert per minute? Are there any further limitations than discspace?
Micki
 
Beiträge: 86
Registriert: Sa Feb 21, 2015 10:38 pm

Re: Dokumentation des Expertcrawl Formular

Beitragvon luc » Mi Aug 10, 2016 8:35 am

I thinks this depends on your crawler and performance settings. I am not aware of /CrawlStartSite.html or /CrawlStartExpert.html setting time restrictions to submit new crawls.
So if you submit many new crawls in a short time, I guess you would eventually end up with your crawl queue being full because reaching memory limits. The best is even to test this.
luc
 
Beiträge: 300
Registriert: Mi Aug 26, 2015 1:04 am

Re: Dokumentation des Expertcrawl Formular

Beitragvon Micki » Fr Aug 12, 2016 7:10 pm

Using the form it thaks about 17 minutes until 1 one job is insert.
Micki
 
Beiträge: 86
Registriert: Sa Feb 21, 2015 10:38 pm

Re: Dokumentation des Expertcrawl Formular

Beitragvon luc » Di Aug 16, 2016 1:26 pm

Wow, 17 minutes, to only insert a crawl job? Something may have gone wrong. On my YaCy peer running only with 600MB RAM on a 2,4GHz processor this only takes a few secons to add a new crawl job with the form...
Do you use some options other than defaults?
luc
 
Beiträge: 300
Registriert: Mi Aug 26, 2015 1:04 am

Re: Dokumentation des Expertcrawl Formular

Beitragvon Micki » Di Aug 16, 2016 8:23 pm

I have 5 GB Ram 14 Mio documents 4 core and 343 crawlobs in the queu
Micki
 
Beiträge: 86
Registriert: Sa Feb 21, 2015 10:38 pm

Re: Dokumentation des Expertcrawl Formular

Beitragvon luc » Mi Aug 17, 2016 9:09 am

Ok, maybe this would be more efficient to start a crawl job with a list of start URLs instead of starting multiple crawl jobs...
By the way, there is probably room for performance improvements so it may be valuable to create a mantis issue. Did you noticed at which number of crawljobs it started to become unreasonnably long to insert new jobs?
luc
 
Beiträge: 300
Registriert: Mi Aug 26, 2015 1:04 am

Re: Dokumentation des Expertcrawl Formular

Beitragvon Micki » Fr Aug 19, 2016 6:27 pm

I'm not shure but i think it was between 200 an 250.
Micki
 
Beiträge: 86
Registriert: Sa Feb 21, 2015 10:38 pm


Zurück zu YaCy Coding & Architecture

Wer ist online?

Mitglieder in diesem Forum: 0 Mitglieder und 4 Gäste

cron