Minimum requirements for YaCy

Discussion in English language.
Forumsregeln
You can start and continue with posts in english language in all other forums as well, but if you are looking for a forum to start a discussion in english, this is the right choice.

Minimum requirements for YaCy

Beitragvon krzyszp » Mi Jan 22, 2014 12:14 pm

I have done dedicated server for YaCy to test stability.
The machine is Athlon64 x2, 2GB RAM, 160GB HDD (Sata) in small factor Shuttle box. I have installed Debian 7 (stable) on it and I install YaCy based on English wiki. Also, I have redirected port on my router to it to make it visible from outside. Also I have enabled network crawling.

I have discovered that after just few hours my YaCy site is not accessible at all.

Do I made something wrong?
Apart of network access for YaCy, I don't change any memory settings, system is completely clear, nothing else installed on it...
krzyszp
 
Beiträge: 21
Registriert: So Jan 12, 2014 2:40 pm

Re: Minimum requirements for YaCy

Beitragvon Yududi » Mi Jan 22, 2014 3:50 pm

Which version do you use?
How many sites do you crawl at the same time?
How big was your index?
How many memory do you gave YaCy during the setup process?
How many memory was free when YaCy was crawling?
What about the load average before YaCy crashed (5 min / 15 min value)?
Yududi
 
Beiträge: 64
Registriert: Di Dez 10, 2013 12:30 pm

Re: Minimum requirements for YaCy

Beitragvon krzyszp » Mi Jan 22, 2014 5:32 pm

YaCy version 1.67/9553
Max 4 sites simultaneously + what sends other hosts (can't check as status site not load).
Index size: 2.2GB
1GB Ram for YaCy

Now values after restart YaCy (I have done this before I read your post, so can't check before).
0.5GB free memory during crawling
Load average 0.5 for 1/5/15 minutes (very stable).

I will supply more data on next crash, but... I have to say, that it looks like not all YaCy crash - just the web site. In system under top I still seem java processes (only java processes on this system are belongs to YaCy).

Edit:
Ok, YaCy not responding again...
Load 1.10/1.03/0.91
Mem: 1332/1897 (in use/total)
krzyszp
 
Beiträge: 21
Registriert: So Jan 12, 2014 2:40 pm

Re: Minimum requirements for YaCy

Beitragvon gTSj » Mo Jan 27, 2014 11:33 pm

Do you have a stacktrace logged somewhere? Should be visible on terminal or in syslog.
gTSj
 
Beiträge: 21
Registriert: Mo Jan 27, 2014 10:49 pm

Re: Minimum requirements for YaCy

Beitragvon krzyszp » Di Jan 28, 2014 12:02 am

I had to delete yacy00.log file then restart yacy again.
End part of new log attached (complete file are 340kB).

Edit:

Can't add file, so last lanes are here:
Code: Alles auswählen
I 2014/01/27 22:50:00 REJECTED http://pl.wikipedia.org/w/index.php?title=Roman_Abraham&oldid=38468188 - post url not allowed
I 2014/01/27 22:50:00 REJECTED http://pl.wikipedia.org/w/index.php?title=Specjalna:Cytuj&page=Roman_Abraham&id=38468188 - post url not allowed
I 2014/01/27 22:50:00 REJECTED http://pl.wikipedia.org/w/index.php?title=Roman_Abraham&action=edit&section=10 - post url not allowed
I 2014/01/27 22:50:00 REJECTED http://pl.wikipedia.org/w/index.php?title=Roman_Abraham&veaction=edit&section=8 - post url not allowed
I 2014/01/27 22:50:00 REJECTED http://pl.wikipedia.org/w/index.php?title=Roman_Abraham&action=edit&section=9 - post url not allowed
I 2014/01/27 22:50:00 REJECTED http://pl.wikipedia.org/w/index.php?title=Roman_Abraham&action=edit - post url not allowed
I 2014/01/27 22:50:00 REJECTED http://pl.wikipedia.org/w/index.php?title=Roman_Abraham&veaction=edit&section=7 - post url not allowed
I 2014/01/27 22:50:00 REJECTED http://www.worldcat.org/identities/viaf-50073800 - url does not match must-match filter \Qhttp://pl.wikipedia.org/\E.*
I 2014/01/27 22:50:00 REJECTED http://pl.wikipedia.org/w/index.php?title=Roman_Abraham&action=edit&section=6 - post url not allowed
I 2014/01/27 22:50:00 REJECTED http://pl.wikipedia.org/w/index.php?title=Specjalna:Ksi%C4%85%C5%BCka&bookcmd=book_creator&referer=Roman%20Abraham - post url not allowed
I 2014/01/27 22:50:00 REJECTED http://www.lwow.home.pl/abraham.html - url does not match must-match filter \Qhttp://pl.wikipedia.org/\E.*
I 2014/01/27 22:50:00 LOADER Forcing sleep of 497 ms for host pl.wikipedia.org
I 2014/01/27 22:50:00 REJECTED http://upload.wikimedia.org/wikipedia/commons/thumb/e/e6/Legion_Honneur_Chevalier_ribbon.svg/40px-Legion_Honneur_Chevalier_ribbon.svg.png - url does not match must-match filter \Qhttp://pl.wikipedia.org/\E.*
I 2014/01/27 22:50:00 REJECTED http://pl.wikipedia.org/w/index.php?title=Roman_Abraham&action=edit&section=1 - post url not allowed
I 2014/01/27 22:50:00 REJECTED http://upload.wikimedia.org/wikipedia/commons/thumb/e/e6/POL_Medal_10-lecia_Odzyskania_Niepodleg%C5%82o%C5%9Bci_BAR.svg/40px-POL_Medal_10-lecia_Odzyskania_Niepodleg%C5%82o%C5%9Bci_BAR.svg.png - url does not match must-match filter \Qhttp://pl.wikipedia.org/\E.*
I 2014/01/27 22:50:00 REJECTED http://pl.wikipedia.org/w/index.php?title=Specjalna:Zaloguj&returnto=Roman%2BAbraham&type=signup - post url not allowed
I 2014/01/27 22:50:00 BALANCER forcing crawl-delay of 486 milliseconds for pl.wikipedia.org: minimumDelta = 500, flux = 0, host.average = 535, robots.delay = 0, ((waitig = 500) - (timeSinceLastAccess = 15)) = 485, domainStacks.size() = 1, domainStacksInitSize = 1
I 2014/01/27 22:50:00 REJECTED http://upload.wikimedia.org/wikipedia/commons/thumb/8/82/Gen_Kutrzeba-_Gen_Abraham.gif/200px-Gen_Kutrzeba-_Gen_Abraham.gif - url does not match must-match filter \Qhttp://pl.wikipedia.org/\E.*
I 2014/01/27 22:50:00 REJECTED http://upload.wikimedia.org/wikipedia/commons/thumb/0/0f/POL_Z%C5%82oty_Krzy%C5%BC_Zas%C5%82ugi_BAR.svg/40px-POL_Z%C5%82oty_Krzy%C5%BC_Zas%C5%82ugi_BAR.svg.png - url does not match must-match filter \Qhttp://pl.wikipedia.org/\E.*
I 2014/01/27 22:50:00 REJECTED http://pl.wikipedia.org/w/index.php?title=Roman_Abraham&action=edit&section=12 - post url not allowed
I 2014/01/27 22:50:00 REJECTED http://pl.wikipedia.org/w/index.php?title=Roman_Abraham&action=edit&section=8 - post url not allowed
I 2014/01/27 22:50:00 REJECTED http://creativecommons.org/licenses/by-sa/3.0/ - url does not match must-match filter \Qhttp://pl.wikipedia.org/\E.*
I 2014/01/27 22:50:00 REJECTED http://upload.wikimedia.org/wikipedia/commons/thumb/c/c8/POL_Virtuti_Militari_Z%C5%82oty_BAR.svg/40px-POL_Virtuti_Militari_Z%C5%82oty_BAR.svg.png - url does not match must-match filter \Qhttp://pl.wikipedia.org/\E.*
I 2014/01/27 22:50:00 REJECTED http://donate.wikimedia.org/wiki/Special:FundraiserRedirector?utm_source=donate&utm_medium=sidebar&utm_campaign=C13_pl.wikipedia.org&uselang=pl - url does not match must-match filter \Qhttp://pl.wikipedia.org/\E.*
I 2014/01/27 22:50:00 REJECTED http://upload.wikimedia.org/wikipedia/commons/thumb/d/d8/Commander_Ordre_de_Leopold.png/40px-Commander_Ordre_de_Leopold.png - url does not match must-match filter \Qhttp://pl.wikipedia.org/\E.*
I 2014/01/27 22:50:00 REJECTED http://upload.wikimedia.org/wikipedia/commons/thumb/c/c7/POL_Virtuti_Militari_Srebrny_BAR.svg/40px-POL_Virtuti_Militari_Srebrny_BAR.svg.png - url does not match must-match filter \Qhttp://pl.wikipedia.org/\E.*
I 2014/01/27 22:50:00 REJECTED http://pl.wikipedia.org/w/index.php?title=Specjalna:Ksi%C4%85%C5%BCka&bookcmd=render_article&arttitle=Roman%20Abraham&oldid=38468188&writer=rl - post url not allowed
I 2014/01/27 22:50:00 REJECTED http://upload.wikimedia.org/wikipedia/commons/thumb/5/59/PL_Epolet_gen_bryg.svg/50px-PL_Epolet_gen_bryg.svg.png - url does not match must-match filter \Qhttp://pl.wikipedia.org/\E.*
I 2014/01/27 22:50:01 REJECTED http://upload.wikimedia.org/wikipedia/commons/thumb/4/4a/Commons-logo.svg/20px-Commons-logo.svg.png - url does not match must-match filter \Qhttp://pl.wikipedia.org/\E.*
I 2014/01/27 22:50:01 REJECTED http://upload.wikimedia.org/wikipedia/commons/thumb/8/84/Gr%C3%B3b_gen._bryg._WP_dr._Romana_Abrahama.JPG/200px-Gr%C3%B3b_gen._bryg._WP_dr._Romana_Abrahama.JPG - url does not match must-match filter \Qhttp://pl.wikipedia.org/\E.*
I 2014/01/27 22:50:01 REJECTED http://pl.wikipedia.org/w/index.php?title=Roman_Abraham&action=edit&section=11 - post url not allowed
D 2014/01/27 22:50:01 SWITCHBOARD processResourceStack processCase=LOCAL_CRAWLING, depth=7, maxDepth=8, must-match=\Qhttp://pl.wikipedia.org/\E.*, must-not-match=(.*Recentchangeslinked.*)|(.*Whatlinkshere.*)|(.*MediaWiki.*), initiatorHash=DiKwX6E3hLcG, url=http://pl.wikipedia.org/wiki/De-Lovely
I 2014/01/27 22:50:01 REJECTED http://pl.wikipedia.org/w/index.php?title=Roman_Abraham&action=edit&section=4 - post url not allowed
I 2014/01/27 22:50:01 STACKCRAWL URL 'http://pl.wikipedia.org/w/opensearch_desc.php' is double registered in 'ERRORS', previous cause: FINAL_ROBOTS_RULE denied by robots.txt
I 2014/01/27 22:50:01 REJECTED http://upload.wikimedia.org/wikipedia/commons/thumb/6/6f/Roman_Abraham_1938.jpg/159px-Roman_Abraham_1938.jpg - url does not match must-match filter \Qhttp://pl.wikipedia.org/\E.*
I 2014/01/27 22:50:01 SWITCHBOARD CRAWL: ADDED 131 LINKS FROM http://pl.wikipedia.org/wiki/De-Lovely, STACKING TIME = 23, PARSING TIME = 32
I 2014/01/27 22:50:01 REJECTED https://wikimediafoundation.org/wiki/Warunki_korzystania - url does not match must-match filter \Qhttp://pl.wikipedia.org/\E.*
I 2014/01/27 22:50:01 REJECTED http://pl.wikipedia.org/w/index.php?title=Roman_Abraham&veaction=edit&section=6 - post url not allowed
I 2014/01/27 22:50:01 SWITCHBOARD Excluded 0 words in URL http://pl.wikipedia.org/wiki/De-Lovely
I 2014/01/27 22:50:01 REJECTED http://pl.wikipedia.org/w/api.php?action=rsd - post url not allowed
I 2014/01/27 22:50:01 REJECTED http://meta.wikimedia.org/ - url does not match must-match filter \Qhttp://pl.wikipedia.org/\E.*
I 2014/01/27 22:50:01 Fulltext indexing: 0MV031oEaihR http://pl.wikipedia.org/wiki/De-Lovely
I 2014/01/27 22:50:01 SWITCHBOARD *Indexed 368 words in URL http://pl.wikipedia.org/wiki/De-Lovely [0MV031oEaihR]
        Description:  De-Lovely – Wikipedia, wolna encyklopedia
        MimeType: text/html | Charset: UTF-8 | Size: 3439 bytes |
        LinkStorageTime: 54 ms | indexStorageTime: 4 ms
I 2014/01/27 22:50:01 REJECTED http://pl.wikipedia.org/w/index.php?title=Roman_Abraham&veaction=edit&section=2 - post url not allowed
I 2014/01/27 22:50:01 REJECTED http://pl.wikipedia.org/w/index.php?title=Roman_Abraham&veaction=edit&section=3 - post url not allowed
I 2014/01/27 22:50:01 REJECTED http://pl.wikipedia.org/w/index.php?title=Roman_Abraham&veaction=edit&section=9 - post url not allowed
I 2014/01/27 22:50:01 REJECTED http://sl.wikipedia.org/wiki/Roman_Abraham - url does not match must-match filter \Qhttp://pl.wikipedia.org/\E.*
I 2014/01/27 22:50:01 REJECTED http://upload.wikimedia.org/wikipedia/commons/thumb/8/8e/POL_Krzy%C5%BC_Walecznych_BAR.svg/40px-POL_Krzy%C5%BC_Walecznych_BAR.svg.png - url does not match must-match filter \Qhttp://pl.wikipedia.org/\E.*
I 2014/01/27 22:50:01 REJECTED http://upload.wikimedia.org/wikipedia/commons/thumb/e/e0/POL_Polonia_Restituta_Oficerski_BAR.svg/40px-POL_Polonia_Restituta_Oficerski_BAR.svg.png - url does not match must-match filter \Qhttp://pl.wikipedia.org/\E.*
I 2014/01/27 22:50:01 REJECTED http://pl.wikipedia.org/w/index.php?title=Roman_Abraham&action=edit&section=13 - post url not allowed
I 2014/01/27 22:50:01 LOADER Forcing sleep of 497 ms for host pl.wikipedia.org
I 2014/01/27 22:50:01 BALANCER forcing crawl-delay of 485 milliseconds for pl.wikipedia.org: minimumDelta = 500, flux = 0, host.average = 528, robots.delay = 0, ((waitig = 500) - (timeSinceLastAccess = 17)) = 483, domainStacks.size() = 1, domainStacksInitSize = 1
I 2014/01/27 22:50:01 REJECTED http://pl.wikipedia.org/w/index.php?title=Specjalna:Ksi%C4%85%C5%BCka&bookcmd=book_creator&referer=Harald%20Hagen - post url not allowed
I 2014/01/27 22:50:01 REJECTED http://pl.wikipedia.org/w/index.php?title=Harald_Hagen&action=edit&section=1 - post url not allowed
I 2014/01/27 22:50:01 REJECTED http://pl.wikipedia.org/w/index.php?title=Specjalna:Ostatnie_zmiany&feed=atom - post url not allowed
I 2014/01/27 22:50:01 REJECTED http://www.sports-reference.com/olympics/athletes/ha/harald-hagen-1.html - url does not match must-match filter \Qhttp://pl.wikipedia.org/\E.*
I 2014/01/27 22:50:01 REJECTED http://pl.wikipedia.org/w/index.php?title=Harald_Hagen&veaction=edit&section=1 - post url not allowed
I 2014/01/27 22:50:01 REJECTED https://www.mediawiki.org/wiki/Special:MyLanguage/How_to_contribute - url does not match must-match filter \Qhttp://pl.wikipedia.org/\E.*
I 2014/01/27 22:50:01 REJECTED http://no.wikipedia.org/wiki/Harald_Hagen - url does not match must-match filter \Qhttp://pl.wikipedia.org/\E.*
I 2014/01/27 22:50:01 REJECTED http://pl.wikipedia.org/w/index.php?title=Harald_Hagen&printable=yes - post url not allowed
I 2014/01/27 22:50:01 REJECTED http://pl.m.wikipedia.org/wiki/Harald_Hagen - url does not match must-match filter \Qhttp://pl.wikipedia.org/\E.*
I 2014/01/27 22:50:01 REJECTED http://wikimediafoundation.org/ - url does not match must-match filter \Qhttp://pl.wikipedia.org/\E.*
I 2014/01/27 22:50:01 REJECTED http://upload.wikimedia.org/wikipedia/commons/thumb/a/a9/Olympic_rings_with_white_rims.svg/16px-Olympic_rings_with_white_rims.svg.png - url does not match must-match filter \Qhttp://pl.wikipedia.org/\E.*
I 2014/01/27 22:50:01 REJECTED http://en.wikipedia.org/wiki/Harald_Hagen - url does not match must-match filter \Qhttp://pl.wikipedia.org/\E.*
I 2014/01/27 22:50:01 REJECTED https://creativecommons.org/licenses/by-sa/3.0/deed.pl - url does not match must-match filter \Qhttp://pl.wikipedia.org/\E.*
I 2014/01/27 22:50:01 REJECTED http://pl.wikipedia.org/w/index.php?title=Kongelig_Norsk_Seilforening&action=edit&redlink=1 - post url not allowed
I 2014/01/27 22:50:01 REJECTED http://www.mediawiki.org/ - url does not match must-match filter \Qhttp://pl.wikipedia.org/\E.*
I 2014/01/27 22:50:01 STACKCRAWL URL 'http://pl.wikipedia.org/w/opensearch_desc.php' is double registered in 'ERRORS', previous cause: FINAL_ROBOTS_RULE denied by robots.txt
I 2014/01/27 22:50:01 REJECTED http://www.wikidata.org/wiki/Q5653794 - url does not match must-match filter \Qhttp://pl.wikipedia.org/\E.*
I 2014/01/27 22:50:01 REJECTED http://pl.wikipedia.org/w/index.php?title=Harald_Hagen&action=info - post url not allowed
I 2014/01/27 22:50:01 REJECTED https://wikimediafoundation.org/wiki/Warunki_korzystania - url does not match must-match filter \Qhttp://pl.wikipedia.org/\E.*
I 2014/01/27 22:50:01 REJECTED http://pl.wikipedia.org/w/index.php?title=Specjalna:Ksi%C4%85%C5%BCka&bookcmd=render_article&arttitle=Harald%20Hagen&oldid=36126222&writer=rl - post url not allowed
I 2014/01/27 22:50:01 REJECTED http://pl.wikipedia.org/w/index.php?title=Specjalna:Cytuj&page=Harald_Hagen&id=36126222 - post url not allowed
I 2014/01/27 22:50:01 REJECTED http://wikimediafoundation.org/wiki/Zasady_zachowania_poufno%C5%9Bci - url does not match must-match filter \Qhttp://pl.wikipedia.org/\E.*
I 2014/01/27 22:50:01 REJECTED http://pl.wikipedia.org/w/index.php?title=Harald_Hagen&action=edit - post url not allowed
I 2014/01/27 22:50:01 REJECTED http://pl.wikipedia.org/w/api.php?action=rsd - post url not allowed
I 2014/01/27 22:50:01 REJECTED http://pl.wikipedia.org/w/index.php?title=Specjalna:Zaloguj&returnto=Harald%2BHagen&type=signup - post url not allowed
I 2014/01/27 22:50:01 REJECTED http://meta.wikimedia.org/ - url does not match must-match filter \Qhttp://pl.wikipedia.org/\E.*
I 2014/01/27 22:50:01 REJECTED http://pl.wikipedia.org/w/index.php?title=Harald_Hagen&action=history - post url not allowed
I 2014/01/27 22:50:01 REJECTED http://creativecommons.org/licenses/by-sa/3.0/ - url does not match must-match filter \Qhttp://pl.wikipedia.org/\E.*
I 2014/01/27 22:50:01 REJECTED http://pl.wikipedia.org/wiki/Special:CentralAutoLogin/start?type=1x1 - post url not allowed
I 2014/01/27 22:50:01 REJECTED http://donate.wikimedia.org/wiki/Special:FundraiserRedirector?utm_source=donate&utm_medium=sidebar&utm_campaign=C13_pl.wikipedia.org&uselang=pl - url does not match must-match filter \Qhttp://pl.wikipedia.org/\E.*
I 2014/01/27 22:50:01 REJECTED http://pl.wikipedia.org/w/index.php?title=Specjalna:Zaloguj&returnto=Harald%2BHagen - post url not allowed
I 2014/01/27 22:50:01 REJECTED http://upload.wikimedia.org/wikipedia/commons/thumb/d/d9/Flag_of_Norway.svg/22px-Flag_of_Norway.svg.png - url does not match must-match filter \Qhttp://pl.wikipedia.org/\E.*
I 2014/01/27 22:50:01 REJECTED http://pl.wikipedia.org/w/index.php?title=Harald_Hagen&veaction=edit - post url not allowed
I 2014/01/27 22:50:01 REJECTED http://pl.wikipedia.org/w/index.php?title=Harald_Hagen&oldid=36126222 - post url not allowed
I 2014/01/27 22:50:01 REJECTED http://bits.wikimedia.org/static-1.23wmf10/skins/vector/images/search-ltr.png?303-4 - url does not match must-match filter \Qhttp://pl.wikipedia.org/\E.*
I 2014/01/27 22:50:01 REJECTED http://upload.wikimedia.org/wikipedia/commons/thumb/0/06/Poland_location_map.svg/238px-Poland_location_map.svg.png - url does not match must-match filter \Qhttp://pl.wikipedia.org/\E.*
I 2014/01/27 22:50:01 REJECTED http://pl.wikipedia.org/w/index.php?title=Specjalna:Ostatnie_zmiany&feed=atom - post url not allowed
I 2014/01/27 22:50:01 REJECTED https://www.mediawiki.org/wiki/Special:MyLanguage/How_to_contribute - url does not match must-match filter \Qhttp://pl.wikipedia.org/\E.*
I 2014/01/27 22:50:01 REJECTED http://upload.wikimedia.org/wikipedia/commons/thumb/e/e9/Geographylogo.svg/20px-Geographylogo.svg.png - url does not match must-match filter \Qhttp://pl.wikipedia.org/\E.*
I 2014/01/27 22:50:01 REJECTED http://pl.wikipedia.org/w/index.php?title=Specjalna:Ksi%C4%85%C5%BCka&bookcmd=book_creator&referer=Broniszewo%20(powiat%20wrzesi%C5%84ski) - post url not allowed
I 2014/01/27 22:50:01 REJECTED http://pl.wikipedia.org/w/index.php?title=Broniszewo_(powiat_wrzesi%C5%84ski)&action=edit&section=1 - post url not allowed
I 2014/01/27 22:50:01 REJECTED http://upload.wikimedia.org/wikipedia/commons/thumb/0/0c/Red_pog.svg/6px-Red_pog.svg.png - url does not match must-match filter \Qhttp://pl.wikipedia.org/\E.*
I 2014/01/27 22:50:01 REJECTED https://creativecommons.org/licenses/by-sa/3.0/deed.pl - url does not match must-match filter \Qhttp://pl.wikipedia.org/\E.*
I 2014/01/27 22:50:01 REJECTED http://www.mediawiki.org/ - url does not match must-match filter \Qhttp://pl.wikipedia.org/\E.*
I 2014/01/27 22:50:01 REJECTED http://upload.wikimedia.org/wikipedia/commons/thumb/2/26/POL_Wrze%C5%9Bnia_COA.svg/50px-POL_Wrze%C5%9Bnia_COA.svg.png - url does not match must-match filter \Qhttp://pl.wikipedia.org/\E.*
I 2014/01/27 22:50:01 REJECTED http://pl.wikipedia.org/w/index.php?title=Broniszewo_(powiat_wrzesi%C5%84ski)&action=info - post url not allowed
I 2014/01/27 22:50:01 REJECTED http://toolserver.org/~geohack/geohack.php?language=pl&params=52.349722222222_N_17.688611111111_E_type:city - url does not match must-match filter \Qhttp://pl.wikipedia.org/\E.*
I 2014/01/27 22:50:01 REJECTED http://wikimediafoundation.org/wiki/Zasady_zachowania_poufno%C5%9Bci - url does not match must-match filter \Qhttp://pl.wikipedia.org/\E.*
I 2014/01/27 22:50:01 REJECTED http://toolserver.org/~geohack/geohack.php?language=pl&pagename=Broniszewo_(powiat_wrzesi%C5%84ski)&params=52_20_59_N_17_41_19_E_type:city - url does not match must-match filter \Qhttp://pl.wikipedia.org/\E.*
I 2014/01/27 22:50:01 REJECTED http://creativecommons.org/licenses/by-sa/3.0/ - url does not match must-match filter \Qhttp://pl.wikipedia.org/\E.*
I 2014/01/27 22:50:01 REJECTED http://pl.wikipedia.org/w/index.php?title=Dyskusja:Broniszewo_(powiat_wrzesi%C5%84ski)&action=edit&redlink=1 - post url not allowed
I 2014/01/27 22:50:01 REJECTED http://pl.wikipedia.org/wiki/Special:CentralAutoLogin/start?type=1x1 - post url not allowed
I 2014/01/27 22:50:01 REJECTED http://pl.wikipedia.org/w/index.php?title=Specjalna:Ksi%C4%85%C5%BCka&bookcmd=render_article&arttitle=Broniszewo%20(powiat%20wrzesi%C5%84ski)&oldid=36888806&writer=rl - post url not allowed
I 2014/01/27 22:50:01 REJECTED http://pl.wikipedia.org/w/index.php?title=Broniszewo_(powiat_wrzesi%C5%84ski)&veaction=edit - post url not allowed
I 2014/01/27 22:50:01 REJECTED http://donate.wikimedia.org/wiki/Special:FundraiserRedirector?utm_source=donate&utm_medium=sidebar&utm_campaign=C13_pl.wikipedia.org&uselang=pl - url does not match must-match filter \Qhttp://pl.wikipedia.org/\E.*
I 2014/01/27 22:50:01 REJECTED http://pl.wikipedia.org/w/index.php?title=Specjalna:Cytuj&page=Broniszewo_(powiat_wrzesi%C5%84ski)&id=36888806 - post url not allowed
I 2014/01/27 22:50:01 REJECTED http://bits.wikimedia.org/static-1.23wmf10/skins/vector/images/search-ltr.png?303-4 - url does not match must-match filter \Qhttp://pl.wikipedia.org/\E.*
I 2014/01/27 22:50:01 REJECTED http://upload.wikimedia.org/wikipedia/commons/thumb/9/9f/Wielkopolskie_mapa_administracyjna.png/238px-Wielkopolskie_mapa_administracyjna.png - url does not match must-match filter \Qhttp://pl.wikipedia.org/\E.*
I 2014/01/27 22:50:01 REJECTED http://pl.wikipedia.org/w/index.php?title=Specjalna:Zaloguj&returnto=Broniszewo%20(powiat%20wrzesi%C5%84ski) - post url not allowed
I 2014/01/27 22:50:01 REJECTED http://pl.wikisource.org/wiki/Polskie_tablice_rejestracyjne - url does not match must-match filter \Qhttp://pl.wikipedia.org/\E.*
I 2014/01/27 22:50:01 REJECTED http://en.wikipedia.org/wiki/Broniszewo%2C_Wrze%C5%9Bnia_County - url does not match must-match filter \Qhttp://pl.wikipedia.org/\E.*
I 2014/01/27 22:50:01 REJECTED http://pl.wikipedia.org/w/index.php?title=Broniszewo_(powiat_wrzesi%C5%84ski)&printable=yes - post url not allowed
I 2014/01/27 22:50:01 REJECTED http://pl.wikipedia.org/w/index.php?title=Szablon:Gmina_Wrze%C5%9Bnia&action=edit - post url not allowed
I 2014/01/27 22:50:01 REJECTED http://wikimediafoundation.org/ - url does not match must-match filter \Qhttp://pl.wikipedia.org/\E.*
I 2014/01/27 22:50:01 STACKCRAWL URL 'http://pl.wikipedia.org/w/opensearch_desc.php' is double registered in 'ERRORS', previous cause: FINAL_ROBOTS_RULE denied by robots.txt
I 2014/01/27 22:50:01 REJECTED http://pl.wikipedia.org/w/index.php?title=Broniszewo_(powiat_wrzesi%C5%84ski)&oldid=36888806 - post url not allowed
I 2014/01/27 22:50:01 REJECTED http://pl.wikipedia.org/w/index.php?title=Broniszewo_(powiat_wrzesi%C5%84ski)&veaction=edit&section=1 - post url not allowed
I 2014/01/27 22:50:01 REJECTED http://pl.wikipedia.org/w/index.php?title=Dyskusja_szablonu:Gmina_Wrze%C5%9Bnia&action=edit&redlink=1 - post url not allowed
I 2014/01/27 22:50:01 REJECTED http://www.wikidata.org/wiki/Q4973933 - url does not match must-match filter \Qhttp://pl.wikipedia.org/\E.*
I 2014/01/27 22:50:01 REJECTED https://wikimediafoundation.org/wiki/Warunki_korzystania - url does not match must-match filter \Qhttp://pl.wikipedia.org/\E.*
I 2014/01/27 22:50:01 REJECTED http://upload.wikimedia.org/wikipedia/commons/thumb/1/12/Flag_of_Poland.svg/22px-Flag_of_Poland.svg.png - url does not match must-match filter \Qhttp://pl.wikipedia.org/\E.*
I 2014/01/27 22:50:01 REJECTED http://pl.wikipedia.org/w/index.php?title=Broniszewo_(powiat_wrzesi%C5%84ski)&action=edit - post url not allowed
I 2014/01/27 22:50:01 REJECTED http://pl.wikipedia.org/w/api.php?action=rsd - post url not allowed
I 2014/01/27 22:50:01 REJECTED http://pl.wikipedia.org/w/index.php?title=Specjalna:Zaloguj&returnto=Broniszewo%20(powiat%20wrzesi%C5%84ski)&type=signup - post url not allowed
I 2014/01/27 22:50:01 REJECTED http://meta.wikimedia.org/ - url does not match must-match filter \Qhttp://pl.wikipedia.org/\E.*
I 2014/01/27 22:50:01 REJECTED http://pl.m.wikipedia.org/wiki/Broniszewo_(powiat_wrzesi%C5%84ski) - url does not match must-match filter \Qhttp://pl.wikipedia.org/\E.*
I 2014/01/27 22:50:01 REJECTED http://pl.wikipedia.org/w/index.php?title=Broniszewo_(powiat_wrzesi%C5%84ski)&action=history - post url not allowed
I 2014/01/27 22:50:01 LOADER Forcing sleep of 496 ms for host pl.wikipedia.org
I 2014/01/27 22:50:01 REJECTED http://pl.wikipedia.org/w/index.php?title=Skip_O'Pot2Mus&action=edit&redlink=1 - post url not allowed
I 2014/01/27 22:50:01 REJECTED http://pl.wikipedia.org/w/index.php?title=Half_Baked&action=edit&redlink=1 - post url not allowed
I 2014/01/27 22:50:01 REJECTED http://pl.wikipedia.org/w/index.php?title=Your_Only_Friends_Are_Make_Believe&action=edit&redlink=1 - post url not allowed
I 2014/01/27 22:50:01 REJECTED http://pl.wikipedia.org/w/index.php?title=Specjalna:Ostatnie_zmiany&feed=atom - post url not allowed
I 2014/01/27 22:50:01 REJECTED https://www.mediawiki.org/wiki/Special:MyLanguage/How_to_contribute - url does not match must-match filter \Qhttp://pl.wikipedia.org/\E.*
I 2014/01/27 22:50:01 BALANCER forcing crawl-delay of 485 milliseconds for pl.wikipedia.org: minimumDelta = 500, flux = 0, host.average = 528, robots.delay = 0, ((waitig = 500) - (timeSinceLastAccess = 17)) = 483, domainStacks.size() = 1, domainStacksInitSize = 1
I 2014/01/27 22:50:01 REJECTED http://pl.wikipedia.org/w/index.php?title=Specjalna:Cytuj&page=L%C3%BCp%C3%BCs_Th%C3%BCnder&id=35917897 - post url not allowed
I 2014/01/27 22:50:01 REJECTED http://pl.m.wikipedia.org/wiki/L%C3%BCp%C3%BCs_Th%C3%BCnder - url does not match must-match filter \Qhttp://pl.wikipedia.org/\E.*
I 2014/01/27 22:50:01 REJECTED http://upload.wikimedia.org/wikipedia/commons/thumb/4/4d/The_Bloodhound_Gang.jpg/200px-The_Bloodhound_Gang.jpg - url does not match must-match filter \Qhttp://pl.wikipedia.org/\E.*
I 2014/01/27 22:50:01 REJECTED https://creativecommons.org/licenses/by-sa/3.0/deed.pl - url does not match must-match filter \Qhttp://pl.wikipedia.org/\E.*
I 2014/01/27 22:50:01 REJECTED http://pl.wikipedia.org/w/index.php?title=Dyskusja:L%C3%BCp%C3%BCs_Th%C3%BCnder&action=edit&redlink=1 - post url not allowed
I 2014/01/27 22:50:01 REJECTED http://pl.wikipedia.org/w/index.php?title=Specjalna:Ksi%C4%85%C5%BCka&bookcmd=render_article&arttitle=L%C3%BCp%C3%BCs%20Th%C3%BCnder&oldid=35917897&writer=rl - post url not allowed
I 2014/01/27 22:50:01 REJECTED http://www.mediawiki.org/ - url does not match must-match filter \Qhttp://pl.wikipedia.org/\E.*
I 2014/01/27 22:50:01 REJECTED http://www.riaa.com/goldandplatinumdata.php?table=SEARCH - url does not match must-match filter \Qhttp://pl.wikipedia.org/\E.*
I 2014/01/27 22:50:01 REJECTED http://www.ultimate-guitar.com/interviews/interviews/lupus_thunder_everything_is_nasty_in_bloodhound_gang.html - url does not match must-match filter \Qhttp://pl.wikipedia.org/\E.*
I 2014/01/27 22:50:01 REJECTED http://wikimediafoundation.org/wiki/Zasady_zachowania_poufno%C5%9Bci - url does not match must-match filter \Qhttp://pl.wikipedia.org/\E.*
I 2014/01/27 22:50:01 REJECTED http://creativecommons.org/licenses/by-sa/3.0/ - url does not match must-match filter \Qhttp://pl.wikipedia.org/\E.*
I 2014/01/27 22:50:01 REJECTED http://pl.wikipedia.org/wiki/Special:CentralAutoLogin/start?type=1x1 - post url not allowed
I 2014/01/27 22:50:01 REJECTED http://donate.wikimedia.org/wiki/Special:FundraiserRedirector?utm_source=donate&utm_medium=sidebar&utm_campaign=C13_pl.wikipedia.org&uselang=pl - url does not match must-match filter \Qhttp://pl.wikipedia.org/\E.*
I 2014/01/27 22:50:01 REJECTED http://da.wikipedia.org/wiki/Lupus_Thunder - url does not match must-match filter \Qhttp://pl.wikipedia.org/\E.*
I 2014/01/27 22:50:01 REJECTED http://pl.wikipedia.org/w/index.php?title=Specjalna:Ksi%C4%85%C5%BCka&bookcmd=book_creator&referer=L%C3%BCp%C3%BCs%20Th%C3%BCnder - post url not allowed
I 2014/01/27 22:50:01 REJECTED http://bg.wikipedia.org/wiki/%D0%9B%D1%8E%D0%BF%D1%83%D1%81_%D0%A2%D1%8A%D0%BD%D0%B4%D1%8A%D1%80 - url does not match must-match filter \Qhttp://pl.wikipedia.org/\E.*
I 2014/01/27 22:50:01 REJECTED http://pl.wikipedia.org/w/index.php?title=L%C3%BCp%C3%BCs_Th%C3%BCnder&veaction=edit - post url not allowed
I 2014/01/27 22:50:01 REJECTED http://www.wikidata.org/wiki/Q738401 - url does not match must-match filter \Qhttp://pl.wikipedia.org/\E.*
D 2014/01/27 22:50:01 SWITCHBOARD processResourceStack processCase=LOCAL_CRAWLING, depth=3, maxDepth=8, must-match=\Qhttp://pl.wikipedia.org/\E.*, must-not-match=(.*Recentchangeslinked.*)|(.*Whatlinkshere.*)|(.*MediaWiki.*), initiatorHash=DiKwX6E3hLcG, url=http://pl.wikipedia.org/wiki/Plik:NFL_teams_locations.PNG
I 2014/01/27 22:50:01 REJECTED http://pl.wikipedia.org/w/index.php?title=Kiss_Me_Where_It_Smells_Funny&action=edit&redlink=1 - post url not allowed
I 2014/01/27 22:50:01 REJECTED http://bits.wikimedia.org/static-1.23wmf10/skins/vector/images/search-ltr.png?303-4 - url does not match must-match filter \Qhttp://pl.wikipedia.org/\E.*
I 2014/01/27 22:50:01 REJECTED http://pl.wikipedia.org/w/index.php?title=L%C3%BCp%C3%BCs_Th%C3%BCnder&printable=yes - post url not allowed
I 2014/01/27 22:50:01 REJECTED http://pl.wikipedia.org/w/index.php?title=Specjalna:Zaloguj&returnto=L%C3%BCp%C3%BCs%20Th%C3%BCnder - post url not allowed
I 2014/01/27 22:50:01 REJECTED http://pl.wikipedia.org/w/index.php?title=Szablon:Bloodhound_Gang&action=edit - post url not allowed
I 2014/01/27 22:50:03 MEMORY performed explicit GC, freed 63070 KB (requested/available/average: 344385 / 362279 / 7681 KB)
I 2014/01/27 22:50:03 HeapReader close HeapFile citation.index.20140126075150800.blob; trace: net.yacy.kelondro.blob.HeapModifier.close(HeapModifier.java:82) -> net.yacy.kelondro.blob.ArrayStack.unmountBLOB(ArrayStack.java:275) -> net.yacy.kelondro.blob.ArrayStack.unmountBestMatch(ArrayStack.java:322) -> net.yacy.kelondro.rwi.ReferenceContainerArray.shrinkUpToMaxSizeFiles(ReferenceContainerArray.java:415) -> net.yacy.kelondro.rwi.IndexCell.shrink(IndexCell.java:199) -> net.yacy.kelondro.rwi.IndexCell.access$1000(IndexCell.java:62) -> net.yacy.kelondro.rwi.IndexCell$FlushThread.flushBuffer(IndexCell.java:166) -> net.yacy.kelondro.rwi.IndexCell$FlushThread.run(IndexCell.java:116)
I 2014/01/27 22:50:03 HeapReader close HeapFile citation.index.20140123174156145.blob; trace: net.yacy.kelondro.blob.HeapModifier.close(HeapModifier.java:82) -> net.yacy.kelondro.blob.ArrayStack.unmountBLOB(ArrayStack.java:275) -> net.yacy.kelondro.blob.ArrayStack.unmountBestMatch(ArrayStack.java:323) -> net.yacy.kelondro.rwi.ReferenceContainerArray.shrinkUpToMaxSizeFiles(ReferenceContainerArray.java:415) -> net.yacy.kelondro.rwi.IndexCell.shrink(IndexCell.java:199) -> net.yacy.kelondro.rwi.IndexCell.access$1000(IndexCell.java:62) -> net.yacy.kelondro.rwi.IndexCell$FlushThread.flushBuffer(IndexCell.java:166) -> net.yacy.kelondro.rwi.IndexCell$FlushThread.run(IndexCell.java:116)
Zuletzt geändert von krzyszp am Di Jan 28, 2014 12:05 am, insgesamt 1-mal geändert.
krzyszp
 
Beiträge: 21
Registriert: So Jan 12, 2014 2:40 pm

Re: Minimum requirements for YaCy

Beitragvon gTSj » Di Jan 28, 2014 12:03 am

I don't see any log files. Am I missing something?
gTSj
 
Beiträge: 21
Registriert: Mo Jan 27, 2014 10:49 pm

Re: Minimum requirements for YaCy

Beitragvon krzyszp » Di Jan 28, 2014 10:01 pm

gTSj hat geschrieben:I don't see any log files. Am I missing something?

Attached above - can't attach whole log file...
krzyszp
 
Beiträge: 21
Registriert: So Jan 12, 2014 2:40 pm

Re: Minimum requirements for YaCy

Beitragvon davidm » So Feb 02, 2014 6:41 am

I noticed the same problems a year or two back when I tested YaCy. As it approached the upper memory limit it suddenly crashed or became very unstable. I was hoping the bug would be fixed. :( I am about to do an install on a 32 bit 3 GB Ram P4 3.0 Ghz Linux install and will test and see if I run into the same thing.
davidm
 
Beiträge: 6
Registriert: So Feb 02, 2014 6:27 am
Wohnort: US

Re: Minimum requirements for YaCy

Beitragvon Orbiter » Fr Feb 14, 2014 11:55 am

I recently added resource observer actions which starts to delete index elements if the space on disk is low (< 1GB). I will continue to use the system monitoring to adopt the behaviour of YaCy, so one answer to the 'minimum requirements' is: there is no, the architecture will contain elements for auto-scaling. But however, I cannot ensure that this works already.
Orbiter
 
Beiträge: 5793
Registriert: Di Jun 26, 2007 10:58 pm
Wohnort: Frankfurt am Main

Re: Minimum requirements for YaCy

Beitragvon biolizard89 » Sa Feb 15, 2014 5:49 am

davidm hat geschrieben:I noticed the same problems a year or two back when I tested YaCy. As it approached the upper memory limit it suddenly crashed or became very unstable. I was hoping the bug would be fixed. :( I am about to do an install on a 32 bit 3 GB Ram P4 3.0 Ghz Linux install and will test and see if I run into the same thing.


I encountered the same bug; it was fixed for me as of a few months ago. The YaCy devs have been doing an outstanding job.

Orbiter hat geschrieben:I recently added resource observer actions which starts to delete index elements if the space on disk is low (< 1GB). I will continue to use the system monitoring to adopt the behaviour of YaCy, so one answer to the 'minimum requirements' is: there is no, the architecture will contain elements for auto-scaling. But however, I cannot ensure that this works already.


Excellent work, it's great that this is being implemented. A friend asked me a few months ago if it was feasible to install YaCy on a small VPS; at the time I had to tell him no... I guess when the next release is stable he'll finally be able to install it and support the YaCy network.

@Orbiter, do you prefer that English-language technical threads be posted in the English forum or in the " YaCy Coding & Architektur" forum? I posted a technical inquiry here: viewtopic.php?f=8&t=5114 , but have received no reply after 11 days. (I'm a university student interested in doing YaCy development for a grad-level course, so I'm curious if you have suggestions on how to get quick answers... I'm not expecting instant replies, but 11 days and counting makes my course project kind of difficult.) Thanks!
biolizard89
 
Beiträge: 61
Registriert: Do Jan 03, 2013 12:42 am


Zurück zu English

Wer ist online?

Mitglieder in diesem Forum: 0 Mitglieder und 2 Gäste