SVN ab 6200 crawl-verhalten

Hier finden YaCy User Hilfe wenn was nicht funktioniert oder anders funktioniert als man dachte. Bei offensichtlichen Fehlern diese bitte gleich in die Bugs (http://bugs.yacy.net) eintragen.
Forumsregeln
In diesem Forum geht es um Benutzungsprobleme und Anfragen für Hilfe. Wird dabei ein Bug identifiziert, wird der thread zur Bearbeitung in die Bug-Sektion verschoben. Wer hier also einen Thread eingestellt hat und ihn vermisst, wird ihn sicherlich in der Bug-Sektion wiederfinden.

SVN ab 6200 crawl-verhalten

Beitragvon dulcedo » Mi Jul 15, 2009 6:48 pm

Nach Update auf die aktuelle Version 6217/19 sehe ich diese Statusgrafik, er crawlt auch nicht lokal, auch nach Neustart nicht. Betrifft alle von 6200 upgedateten Peers.
Dateianhänge
PerformanceGraph.png
PerformanceGraph.png (6.97 KiB) 2773-mal betrachtet
PerformanceGraph1.png
PerformanceGraph1.png (8.98 KiB) 2772-mal betrachtet
dulcedo
 
Beiträge: 1006
Registriert: Do Okt 16, 2008 6:36 pm
Wohnort: Bei Karlsruhe

Re: SVN ab 6200 crawl-verhalten

Beitragvon Quix0r » Mi Jul 15, 2009 7:35 pm

So sieht es bei mir aus:
PerformanceGraph.png
Neuste Revision
PerformanceGraph.png (8.77 KiB) 2754-mal betrachtet
Quix0r
 
Beiträge: 1345
Registriert: Di Jul 31, 2007 9:22 am
Wohnort: Krefeld

Re: SVN ab 6200 crawl-verhalten

Beitragvon dulcedo » Mi Jul 15, 2009 7:50 pm

Er hat dabei ca. 10 von den url-tables lesend im Zugriff, Prozessorlast 70%, normal ist 30%.
dulcedo
 
Beiträge: 1006
Registriert: Do Okt 16, 2008 6:36 pm
Wohnort: Bei Karlsruhe

Re: SVN ab 6200 crawl-verhalten

Beitragvon dulcedo » Do Jul 16, 2009 1:52 am

Weiterer Test: Version 6207 funktioniert noch richtig.
dulcedo
 
Beiträge: 1006
Registriert: Do Okt 16, 2008 6:36 pm
Wohnort: Bei Karlsruhe

Re: SVN ab 6200 crawl-verhalten

Beitragvon Orbiter » Do Jul 16, 2009 8:52 am

hier bin ich bei weitreichenden Umbauten. Der Crawler hat zwar durch den neuen Balancer aus Release 0.9 gewonnen, aber die Queues für die Organisation sind einfach nur eine Qual. Da werden so viele Altlasten mitgeschleppt, dass ich eine Generalrenovierung angegangen bin. Es wird bestimmt noch einige Tage lang holprig bleiben.
In SVN 6216 habe ich einen Bug eingebaut, der einen wide crawl verhindert hat. War aber nur eine klitzekleine Sache, an der ich gestern stundenlang gesucht habe. Fix ist nun in SVN 6222 .. zusammen mit wieder vielen anderen Umbauten.
Orbiter
 
Beiträge: 5792
Registriert: Di Jun 26, 2007 10:58 pm
Wohnort: Frankfurt am Main

Re: SVN ab 6200 crawl-verhalten

Beitragvon dulcedo » Do Jul 16, 2009 10:01 am

Der Fehler ist wohl behoben, zumindest sieht die Systemkurve wieder normal aus.
Die DHT-Sache im anderen Thread ist nicht vorrangig, PCA42 hat ja geschrieben es sind nur Warnmeldungen. Wichtiger ist dass man den Peer auch (langsam) crawlen lassen kann, unbeaufsichtigt, um eine Portalsuche aktuell indexieren zu können. Nach wie vor als Ergänzung zum Austausch über XML oder als Alternative. Wenn wir da jeden Tag drüberschauen müssen ist das nicht machbar, auch nicht wenns der Betreiber selbst macht. Und selbst mit einer Art Selbstreperatur ist ein grosser Peer bei Neustart 15 Minuten offline. Der Mechanismus zum switchen auf einen andrern Peer bei Nichtverfügbarkeit funktioniert übrigens Client-seitig (JS), wird noch im Wiki dokumentiert.
dulcedo
 
Beiträge: 1006
Registriert: Do Okt 16, 2008 6:36 pm
Wohnort: Bei Karlsruhe

Re: SVN ab 6200 crawl-verhalten

Beitragvon Quix0r » Do Jul 16, 2009 10:50 am

Bei mir leider nicht. :( Hier der Verlauf:
PerformanceGraph.png
Verlauf von free-search2
PerformanceGraph.png (5.37 KiB) 2653-mal betrachtet

Es klappt nur folgendes:
- INDEX-TRANSFER-DISPATCHER
- PLASMA dhtTransferJob
- YACY yacyClient.publishMySeed
- YACY PeerPing
- YACY yacyClient.transferRWI

Revision 6222. Mittlerweile hat die Node die Queues entweder gelöscht, oder abgearbeitet. Ich tippe aber eher auf ersteres, da er > 6 Mio. Einträge hatte.
Zuletzt geändert von Quix0r am Do Jul 16, 2009 10:59 am, insgesamt 2-mal geändert.
Quix0r
 
Beiträge: 1345
Registriert: Di Jul 31, 2007 9:22 am
Wohnort: Krefeld

Re: SVN ab 6200 crawl-verhalten

Beitragvon Quix0r » Do Jul 16, 2009 10:55 am

Nun ist ein Thread blockiert:
Thread= Session_0:0:0:0:0:0:0:1:58133#0 id=227 BLOCKED
Thread= Session_0:0:0:0:0:0:0:1:34391#0 id=237 BLOCKED
Thread= Session_0:0:0:0:0:0:0:1:34396#0 id=242 BLOCKED
at de.anomic.crawler.Balancer.top(Balancer.java:444) [for (String n: top) {]
at de.anomic.crawler.NoticedURL.top(NoticedURL.java:249)
at de.anomic.crawler.NoticedURL.top(NoticedURL.java:188)
at queues_p.respond(queues_p.java:117)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at de.anomic.http.httpdFileHandler.invokeServlet(httpdFileHandler.java:1173)
at de.anomic.http.httpdFileHandler.doResponse(httpdFileHandler.java:753)
at de.anomic.http.httpdFileHandler.doGet(httpdFileHandler.java:242)
at de.anomic.http.httpd.GET(httpd.java:490)
at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at de.anomic.server.serverCore$Session.listen(serverCore.java:728)
at de.anomic.server.serverCore$Session.run(serverCore.java:619)

Revision ist 6222. Auch diese Abfrage haengt:
http://localpeer/api/queues_p.xml?html=

Anbei der komplette Thread-Dump waehren keine Crawls, auch keine Remotes mehr laufen nach 1000 Durchlaeufen.
Dateianhänge
dump.zip
Thread-Dump SVN 6222 nach 1000 Durchlaeufen
(4.77 KiB) 45-mal heruntergeladen
Quix0r
 
Beiträge: 1345
Registriert: Di Jul 31, 2007 9:22 am
Wohnort: Krefeld

Re: SVN ab 6200 crawl-verhalten

Beitragvon dulcedo » Do Jul 16, 2009 1:34 pm

Das mit den verworfenen Crawls ist mir auch aufgefallen.
Versuche mal konservative crawl-Einstellungen, oder einfach über die ppm begrenzen, zwischen 200 und 500 crawlt er auf einem Quad sehr konstant und flüssig. Es wird ja grade gebastelt am Crawler.
dulcedo
 
Beiträge: 1006
Registriert: Do Okt 16, 2008 6:36 pm
Wohnort: Bei Karlsruhe

Re: SVN ab 6200 crawl-verhalten

Beitragvon Orbiter » Mo Jul 20, 2009 9:13 am

dhtTransferJob hat nichts mit dem crawler zu tun.
Orbiter
 
Beiträge: 5792
Registriert: Di Jun 26, 2007 10:58 pm
Wohnort: Frankfurt am Main

Re: SVN ab 6200 crawl-verhalten

Beitragvon dulcedo » Mo Jul 20, 2009 9:57 am

Ich habe 6233 auf 2 mittelgrossen Peers im Einsatz voher auch schon 6222, dort war das crawling noch ein bischen holprig aber ohne richtige Hänger. Jetzt in 6233 ohne diese Hänger, wenn dann wohl absichtlich wegen Wartezeiten.
Geschwindigkeit begrenze ich auf 600ppm und damit kann ich auf einem dual-Atom mit 4GB unter Windows7 auch noch gut arbeiten (2GB für YaCy die auch gebraucht werden), ich glaube das ist auf dem richtigen Weg. Die Erläuterungen im SVN übrigens prima.
dulcedo
 
Beiträge: 1006
Registriert: Do Okt 16, 2008 6:36 pm
Wohnort: Bei Karlsruhe

Re: SVN ab 6200 crawl-verhalten

Beitragvon Quix0r » Mo Jul 20, 2009 5:43 pm

Orbiter hat geschrieben:dhtTransferJob hat nichts mit dem crawler zu tun.

Okay, sorry. :roll: Wo passt das dann besser hin, damit die Uebersicht bleibt.
Quix0r
 
Beiträge: 1345
Registriert: Di Jul 31, 2007 9:22 am
Wohnort: Krefeld

Re: SVN ab 6200 crawl-verhalten

Beitragvon dulcedo » Mo Jul 20, 2009 9:20 pm

Das hat mich zu anfang auch verwirrt weils es oft gleichzeitig passiert. Habe das Teilen aber unfallfrei hinbekommen.
Wenn dieser Fehler dort gefixt ist dürften viele echte Hänger wegfallen da das sehr oft die Ursache ist.
dulcedo
 
Beiträge: 1006
Registriert: Do Okt 16, 2008 6:36 pm
Wohnort: Bei Karlsruhe

Re: SVN ab 6200 crawl-verhalten

Beitragvon Quix0r » Mo Jul 20, 2009 9:28 pm

Danke, meine Liste wird zu lang. :(
Quix0r
 
Beiträge: 1345
Registriert: Di Jul 31, 2007 9:22 am
Wohnort: Krefeld

Re: SVN ab 6200 crawl-verhalten

Beitragvon dulcedo » Sa Jul 25, 2009 7:23 am

Das ist SVN6242 beim Versuch einen Crawl zu starten, eigentlich müssten 500MB Speicher frei sein.
EDIT: Die selben Fehler auch im normalen DHT-Betrieb.
Nach downgrade auf 6236 wieder normaler Betrieb.

Auf einem anderen, etwa gleich grossen peer tritt das Problem allerdings nicht auf, dort läuft 6242 einwandfrei (ohne crawling bisher).


Code: Alles auswählen
************* Start Thread Dump Sat Jul 25 08:20:22 CEST 2009 *******************

YaCy Version: 0.91/6242
Total Memory = 2155479040
Used  Memory = 1673580224
Free  Memory = 481898816

THREADS WITH STATES: BLOCKED
THREADS WITH STATES: RUNNABLE
...

Sorry die LF gehen unter Win bei CRTL+C aus dem Log irgendwie verloren.
Code: Alles auswählen
I 2009/07/25 08:10:36 INDEX-TRANSFER-DISPATCHER enqueueContainers: selected 9 targets for primary target key j1tNkulJ0U__/8 with 1 index containers.I 2009/07/25 08:10:36 INDEX-TRANSFER-DISPATCHER enqueueContainers: selected 9 targets for primary target key n1tNkulJ0U__/9 with 1 index containers.I 2009/07/25 08:10:36 INDEX-TRANSFER-DISPATCHER enqueueContainers: selected 9 targets for primary target key r1tNkulJ0U__/10 with 1 index containers.I 2009/07/25 08:10:36 INDEX-TRANSFER-DISPATCHER enqueueContainers: selected 9 targets for primary target key v1tNkulJ0U__/11 with 1 index containers.I 2009/07/25 08:10:36 INDEX-TRANSFER-DISPATCHER enqueueContainers: selected 9 targets for primary target key z1tNkulJ0U__/12 with 1 index containers.I 2009/07/25 08:10:36 INDEX-TRANSFER-DISPATCHER enqueueContainers: selected 9 targets for primary target key 31tNkulJ0U__/13 with 1 index containers.I 2009/07/25 08:10:36 INDEX-TRANSFER-DISPATCHER enqueueContainers: selected 9 targets for primary target key 71tNkulJ0U__/14 with 1 index containers.I 2009/07/25 08:10:36 INDEX-TRANSFER-DISPATCHER enqueueContainers: selected 9 targets for primary target key _1tNkulJ0U__/15 with 1 index containers.E 2009/07/25 08:10:37 BUSYTHREAD Runtime Error in serverInstantThread.job, thread 'de.anomic.search.Switchboard.dhtTransferJob': null; target exception: Java heap space
java.lang.OutOfMemoryError: Java heap spaceI 2009/07/25 08:10:44 PLASMA Received 79 Entries 1 Words [VTwMoSH3VNEf .. VTwMoSH3VNEf]/2611163816405809264 from oqJ9r8sRv_v0:Hasenjagd/0.91006243, processed in 16 milliseconds, requesting 65/79 URLs, blocked 0 RWIsE 2009/07/25 08:10:49 FILEHANDLER INTERNAL ERROR: java.lang.reflect.InvocationTargetException:null target exception at D:\b4-fam\YaCy-dulcedo2\htroot\yacy\transferURL.class: java.lang.OutOfMemoryError: Java heap space:Java heap space
java.lang.reflect.InvocationTargetException
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
   at java.lang.reflect.Method.invoke(Unknown Source)
   at de.anomic.http.server.HTTPDFileHandler.invokeServlet(HTTPDFileHandler.java:1177)
   at de.anomic.http.server.HTTPDFileHandler.doResponse(HTTPDFileHandler.java:757)
   at de.anomic.http.server.HTTPDFileHandler.doPost(HTTPDFileHandler.java:254)
   at de.anomic.http.server.HTTPDemon.POST(HTTPDemon.java:630)
   at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
   at java.lang.reflect.Method.invoke(Unknown Source)
   at de.anomic.server.serverCore$Session.listen(serverCore.java:728)
   at de.anomic.server.serverCore$Session.run(serverCore.java:619)
Caused by: java.lang.OutOfMemoryError: Java heap spaceW 2009/07/25 08:10:49 FILEHANDLER Unexpected error while processing query.
Session: Session_84.138.93.134:43666#9
Query:   /yacy/transferURL.html
Client:  84.138.93.134
Reason:  java.lang.reflect.InvocationTargetException
java.lang.reflect.InvocationTargetException
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
   at java.lang.reflect.Method.invoke(Unknown Source)
   at de.anomic.http.server.HTTPDFileHandler.invokeServlet(HTTPDFileHandler.java:1177)
   at de.anomic.http.server.HTTPDFileHandler.doResponse(HTTPDFileHandler.java:757)
   at de.anomic.http.server.HTTPDFileHandler.doPost(HTTPDFileHandler.java:254)
   at de.anomic.http.server.HTTPDemon.POST(HTTPDemon.java:630)
   at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
   at java.lang.reflect.Method.invoke(Unknown Source)
   at de.anomic.server.serverCore$Session.listen(serverCore.java:728)
   at de.anomic.server.serverCore$Session.run(serverCore.java:619)
Caused by: java.lang.OutOfMemoryError: Java heap spaceE 2009/07/25 08:10:54 YACY yacyClient.queryUrlCount error asking peer 'Green':org.apache.commons.httpclient.NoHttpResponseException: The server 194.204.33.13 failed to respondI 2009/07/25 08:10:54 YACY hello: responded remote junior peer 'Green' from 194.204.33.13I 2009/07/25 08:10:54 YACY hello: responded remote peer 'Green' [194.204.33.13] in 249 millisecondsI 2009/07/25 08:11:00 PLASMA Received 49 Entries 1 Words [VTwMoSH3VNEf .. VTwMoSH3VNEf]/2611163816405809264 from oqJ9r8sRv_v0:Hasenjagd/0.91006243, processed in 16 milliseconds, requesting 48/49 URLs, blocked 0 RWIsE 2009/07/25 08:11:02 FILEHANDLER INTERNAL ERROR: java.lang.reflect.InvocationTargetException:null target exception at D:\b4-fam\YaCy-dulcedo2\htroot\yacy\transferURL.class: java.lang.OutOfMemoryError: Java heap space:Java heap space
java.lang.reflect.InvocationTargetException
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
   at java.lang.reflect.Method.invoke(Unknown Source)
   at de.anomic.http.server.HTTPDFileHandler.invokeServlet(HTTPDFileHandler.java:1177)
   at de.anomic.http.server.HTTPDFileHandler.doResponse(HTTPDFileHandler.java:757)
   at de.anomic.http.server.HTTPDFileHandler.doPost(HTTPDFileHandler.java:254)
   at de.anomic.http.server.HTTPDemon.POST(HTTPDemon.java:630)
   at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
   at java.lang.reflect.Method.invoke(Unknown Source)
   at de.anomic.server.serverCore$Session.listen(serverCore.java:728)
   at de.anomic.server.serverCore$Session.run(serverCore.java:619)
Caused by: java.lang.OutOfMemoryError: Java heap spaceW 2009/07/25 08:11:02 FILEHANDLER Unexpected error while processing query.
Session: Session_84.138.93.134:43666#11
Query:   /yacy/transferURL.html
Client:  84.138.93.134
Reason:  java.lang.reflect.InvocationTargetException
java.lang.reflect.InvocationTargetException
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
   at java.lang.reflect.Method.invoke(Unknown Source)
   at de.anomic.http.server.HTTPDFileHandler.invokeServlet(HTTPDFileHandler.java:1177)
   at de.anomic.http.server.HTTPDFileHandler.doResponse(HTTPDFileHandler.java:757)
   at de.anomic.http.server.HTTPDFileHandler.doPost(HTTPDFileHandler.java:254)
   at de.anomic.http.server.HTTPDemon.POST(HTTPDemon.java:630)
   at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
   at java.lang.reflect.Method.invoke(Unknown Source)
   at de.anomic.server.serverCore$Session.listen(serverCore.java:728)
   at de.anomic.server.serverCore$Session.run(serverCore.java:619)
Caused by: java.lang.OutOfMemoryError: Java heap spaceI 2009/07/25 08:11:07 PLASMA dhtTransferJob: selected mpU9y3ucwNec as start hashI 2009/07/25 08:11:07 PLASMA dhtTransferJob: selected nItvx0OBcbZU as limit hashI 2009/07/25 08:11:08 INDEX-TRANSFER-DISPATCHER selectContainersToCache: selectedContainerCache was filled with 61 entriesI 2009/07/25 08:11:08 INDEX-TRANSFER-DISPATCHER splitContainersFromCache: splittedContainerCache filled with 16 partitions, deleting selectedContainerCacheI 2009/07/25 08:11:08 INDEX-TRANSFER-DISPATCHER enqueueContainers: selected 9 targets for primary target key CpV6HAVvQn__/0 with 61 index containers.E 2009/07/25 08:11:09 BUSYTHREAD Runtime Error in serverInstantThread.job, thread 'de.anomic.search.Switchboard.dhtTransferJob': null; target exception: Java heap space
java.lang.OutOfMemoryError: Java heap spaceI 2009/07/25 08:11:17 YACY hello: responded remote peer 'KIT01-04f' [141.52.175.15] in 562 millisecondsI 2009/07/25 08:11:34 YACY hello: responded remote peer 'lasatonline' [84.56.123.241] in 468 millisecondsI 2009/07/25 08:11:39 PLASMA dhtTransferJob: selected lZTlBnfZ9-oY as start hashI 2009/07/25 08:11:39 PLASMA dhtTransferJob: selected mZlxEMR_z-xC as limit hashI 2009/07/25 08:11:39 INDEX-TRANSFER-DISPATCHER selectContainersToCache: selectedContainerCache was filled with 5 entriesI 2009/07/25 08:11:39 INDEX-TRANSFER-DISPATCHER splitContainersFromCache: splittedContainerCache filled with 16 partitions, deleting selectedContainerCacheI 2009/07/25 08:11:39 INDEX-TRANSFER-DISPATCHER enqueueContainers: selected 9 targets for primary target key BZTojmW9-2__/0 with 5 index containers.E 2009/07/25 08:11:41 BUSYTHREAD Runtime Error in serverInstantThread.job, thread 'de.anomic.search.Switchboard.dhtTransferJob': null; target exception: Java heap space
java.lang.OutOfMemoryError: Java heap spaceI 2009/07/25 08:11:42 PLASMA Received 38 Entries 13 Words [SGWsrPMYPM86 .. SGXWw0B5sNrq]/3073668385503962996 from nItvx0OBcbZU:KIT01-04f/0.9100617, processed in 0 milliseconds, requesting 2/38 URLs, blocked 2 RWIsE 2009/07/25 08:11:43 FILEHANDLER INTERNAL ERROR: java.lang.reflect.InvocationTargetException:null target exception at D:\b4-fam\YaCy-dulcedo2\htroot\yacy\transferURL.class: java.lang.OutOfMemoryError: Java heap space:Java heap space
java.lang.reflect.InvocationTargetException
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
   at java.lang.reflect.Method.invoke(Unknown Source)
   at de.anomic.http.server.HTTPDFileHandler.invokeServlet(HTTPDFileHandler.java:1177)
   at de.anomic.http.server.HTTPDFileHandler.doResponse(HTTPDFileHandler.java:757)
   at de.anomic.http.server.HTTPDFileHandler.doPost(HTTPDFileHandler.java:254)
   at de.anomic.http.server.HTTPDemon.POST(HTTPDemon.java:630)
   at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
   at java.lang.reflect.Method.invoke(Unknown Source)
   at de.anomic.server.serverCore$Session.listen(serverCore.java:728)
   at de.anomic.server.serverCore$Session.run(serverCore.java:619)
Caused by: java.lang.OutOfMemoryError: Java heap spaceW 2009/07/25 08:11:43 FILEHANDLER Unexpected error while processing query.
Session: Session_141.52.175.15:59153#2
Query:   /yacy/transferURL.html
Client:  141.52.175.15
Reason:  java.lang.reflect.InvocationTargetException
java.lang.reflect.InvocationTargetException
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
   at java.lang.reflect.Method.invoke(Unknown Source)
   at de.anomic.http.server.HTTPDFileHandler.invokeServlet(HTTPDFileHandler.java:1177)
   at de.anomic.http.server.HTTPDFileHandler.doResponse(HTTPDFileHandler.java:757)
   at de.anomic.http.server.HTTPDFileHandler.doPost(HTTPDFileHandler.java:254)
   at de.anomic.http.server.HTTPDemon.POST(HTTPDemon.java:630)
   at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
   at java.lang.reflect.Method.invoke(Unknown Source)
   at de.anomic.server.serverCore$Session.listen(serverCore.java:728)
   at de.anomic.server.serverCore$Session.run(serverCore.java:619)
Caused by: java.lang.OutOfMemoryError: Java heap spaceI 2009/07/25 08:11:46 YACY yacyClient.publishMySeed thread 'PublishSeed_Hermes' contacted peer at 84.60.136.202:8088, received 9737 bytes, time = 11186 millisecondsI 2009/07/25 08:11:46 YACY connect: SELF reference 92.205.106.227:8081I 2009/07/25 08:11:46 YACY publish: handshaked principal peer 'Hermes' at 84.60.136.202:8088I 2009/07/25 08:11:46 YACY PeerPing: I am accessible for 7 peer(s), not accessible for 1 peer(s).I 2009/07/25 08:11:46 YACY PeerPing: myType is seniorE 2009/07/25 08:11:56 SERVER receive interrupted - exception 2 = Connection resetE 2009/07/25 08:11:56 SERVER receive interrupted - exception 2 = Connection resetE 2009/07/25 08:11:56 SERVER receive interrupted - exception 2 = Connection resetE 2009/07/25 08:11:56 SERVER receive interrupted - exception 2 = Connection resetE 2009/07/25 08:11:56 SERVER receive interrupted - exception 2 = Connection resetE 2009/07/25 08:11:56 SERVER receive interrupted - exception 2 = Connection resetI 2009/07/25 08:12:04 PLASMA Received 18 Entries 5 Words [dvr15PCPK7Fo .. dvs1LnGV0TNP]/1395327757040952224 from KuTo8XxqOoAA:Knallfrosch/0.91006243, processed in 0 milliseconds, requesting 7/18 URLs, blocked 0 RWIsE 2009/07/25 08:12:05 FILEHANDLER INTERNAL ERROR: java.lang.reflect.InvocationTargetException:null target exception at D:\b4-fam\YaCy-dulcedo2\htroot\yacy\transferURL.class: java.lang.OutOfMemoryError: Java heap space:Java heap space
java.lang.reflect.InvocationTargetException
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
   at java.lang.reflect.Method.invoke(Unknown Source)
   at de.anomic.http.server.HTTPDFileHandler.invokeServlet(HTTPDFileHandler.java:1177)
   at de.anomic.http.server.HTTPDFileHandler.doResponse(HTTPDFileHandler.java:757)
   at de.anomic.http.server.HTTPDFileHandler.doPost(HTTPDFileHandler.java:254)
   at de.anomic.http.server.HTTPDemon.POST(HTTPDemon.java:630)
   at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
   at java.lang.reflect.Method.invoke(Unknown Source)
   at de.anomic.server.serverCore$Session.listen(serverCore.java:728)
   at de.anomic.server.serverCore$Session.run(serverCore.java:619)
Caused by: java.lang.OutOfMemoryError: Java heap spaceW 2009/07/25 08:12:05 FILEHANDLER Unexpected error while processing query.
Session: Session_94.220.117.207:1397#1
Query:   /yacy/transferURL.html
Client:  94.220.117.207
Reason:  java.lang.reflect.InvocationTargetException
java.lang.reflect.InvocationTargetException
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
   at java.lang.reflect.Method.invoke(Unknown Source)
   at de.anomic.http.server.HTTPDFileHandler.invokeServlet(HTTPDFileHandler.java:1177)
   at de.anomic.http.server.HTTPDFileHandler.doResponse(HTTPDFileHandler.java:757)
   at de.anomic.http.server.HTTPDFileHandler.doPost(HTTPDFileHandler.java:254)
   at de.anomic.http.server.HTTPDemon.POST(HTTPDemon.java:630)
   at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
   at java.lang.reflect.Method.invoke(Unknown Source)
   at de.anomic.server.serverCore$Session.listen(serverCore.java:728)
   at de.anomic.server.serverCore$Session.run(serverCore.java:619)
Caused by: java.lang.OutOfMemoryError: Java heap spaceI 2009/07/25 08:12:05 PLASMA Received 7 Entries 5 Words [dvr15PCPK7Fo .. dvs1LnGV0TNP]/1395327757040952224 from KuTo8XxqOoAA:Knallfrosch/0.91006243, processed in 0 milliseconds, requesting 1/7 URLs, blocked 0 RWIsI 2009/07/25 08:12:06 PLASMA Received 0 URLs from peer KuTo8XxqOoAA:Knallfrosch/0.91006243 in 0 ms, blocked 1 URLsI 2009/07/25 08:12:09 PLASMA Received 10 Entries 4 Words [dvr15PCPK7Fo .. dvs1LnGV0TNP]/1395327757040952224 from KuTo8XxqOoAA:Knallfrosch/0.91006243, processed in 0 milliseconds, requesting 3/10 URLs, blocked 0 RWIsE 2009/07/25 08:12:10 FILEHANDLER INTERNAL ERROR: java.lang.reflect.InvocationTargetException:null target exception at D:\b4-fam\YaCy-dulcedo2\htroot\yacy\transferURL.class: java.lang.OutOfMemoryError: Java heap space:Java heap space
java.lang.reflect.InvocationTargetException
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
   at java.lang.reflect.Method.invoke(Unknown Source)
   at de.anomic.http.server.HTTPDFileHandler.invokeServlet(HTTPDFileHandler.java:1177)
   at de.anomic.http.server.HTTPDFileHandler.doResponse(HTTPDFileHandler.java:757)
   at de.anomic.http.server.HTTPDFileHandler.doPost(HTTPDFileHandler.java:254)
   at de.anomic.http.server.HTTPDemon.POST(HTTPDemon.java:630)
   at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
   at java.lang.reflect.Method.invoke(Unknown Source)
   at de.anomic.server.serverCore$Session.listen(serverCore.java:728)
   at de.anomic.server.serverCore$Session.run(serverCore.java:619)
Caused by: java.lang.OutOfMemoryError: Java heap spaceW 2009/07/25 08:12:10 FILEHANDLER Unexpected error while processing query.
Session: Session_94.220.117.207:1401#3
Query:   /yacy/transferURL.html
Client:  94.220.117.207
Reason:  java.lang.reflect.InvocationTargetException
java.lang.reflect.InvocationTargetException
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
   at java.lang.reflect.Method.invoke(Unknown Source)
   at de.anomic.http.server.HTTPDFileHandler.invokeServlet(HTTPDFileHandler.java:1177)
   at de.anomic.http.server.HTTPDFileHandler.doResponse(HTTPDFileHandler.java:757)
   at de.anomic.http.server.HTTPDFileHandler.doPost(HTTPDFileHandler.java:254)
   at de.anomic.http.server.HTTPDemon.POST(HTTPDemon.java:630)
   at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
   at java.lang.reflect.Method.invoke(Unknown Source)
   at de.anomic.server.serverCore$Session.listen(serverCore.java:728)
   at de.anomic.server.serverCore$Session.run(serverCore.java:619)
Caused by: java.lang.OutOfMemoryError: Java heap spaceI 2009/07/25 08:12:11 PLASMA dhtTransferJob: selected EbG312LoAlal as start hashI 2009/07/25 08:12:11 PLASMA dhtTransferJob: selected E2E_-NKZgOQQ as limit hashI 2009/07/25 08:12:11 INDEX-TRANSFER-DISPATCHER selectContainersToCache: selectedContainerCache was filled with 62 entriesI 2009/07/25 08:12:11 INDEX-TRANSFER-DISPATCHER splitContainersFromCache: splittedContainerCache filled with 16 partitions, deleting selectedContainerCacheI 2009/07/25 08:12:11 INDEX-TRANSFER-DISPATCHER enqueueContainers: selected 9 targets for primary target key AbHsU02IZP__/0 with 62 index containers.E 2009/07/25 08:12:13 BUSYTHREAD Runtime Error in serverInstantThread.job, thread 'de.anomic.search.Switchboard.dhtTransferJob': null; target exception: Java heap space
java.lang.OutOfMemoryError: Java heap spaceI 2009/07/25 08:12:15 PLASMA Received 32 Entries 3 Words [dvr15PCPK7Fo .. dvs1LnGV0TNP]/1395327757040952224 from KuTo8XxqOoAA:Knallfrosch/0.91006243, processed in 0 milliseconds, requesting 2/32 URLs, blocked 0 RWIsE 2009/07/25 08:12:16 FILEHANDLER INTERNAL ERROR: java.lang.reflect.InvocationTargetException:null target exception at D:\b4-fam\YaCy-dulcedo2\htroot\yacy\transferURL.class: java.lang.OutOfMemoryError: Java heap space:Java heap space
java.lang.reflect.InvocationTargetException
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
   at java.lang.reflect.Method.invoke(Unknown Source)
   at de.anomic.http.server.HTTPDFileHandler.invokeServlet(HTTPDFileHandler.java:1177)
   at de.anomic.http.server.HTTPDFileHandler.doResponse(HTTPDFileHandler.java:757)
   at de.anomic.http.server.HTTPDFileHandler.doPost(HTTPDFileHandler.java:254)
   at de.anomic.http.server.HTTPDemon.POST(HTTPDemon.java:630)
   at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
   at java.lang.reflect.Method.invoke(Unknown Source)
   at de.anomic.server.serverCore$Session.listen(serverCore.java:728)
   at de.anomic.server.serverCore$Session.run(serverCore.java:619)
Caused by: java.lang.OutOfMemoryError: Java heap spaceW 2009/07/25 08:12:16 FILEHANDLER Unexpected error while processing query.
Session: Session_94.220.117.207:1401#5
Query:   /yacy/transferURL.html
Client:  94.220.117.207
Reason:  java.lang.reflect.InvocationTargetException
java.lang.reflect.InvocationTargetException
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
   at java.lang.reflect.Method.invoke(Unknown Source)
   at de.anomic.http.server.HTTPDFileHandler.invokeServlet(HTTPDFileHandler.java:1177)
   at de.anomic.http.server.HTTPDFileHandler.doResponse(HTTPDFileHandler.java:757)
   at de.anomic.http.server.HTTPDFileHandler.doPost(HTTPDFileHandler.java:254)
   at de.anomic.http.server.HTTPDemon.POST(HTTPDemon.java:630)
   at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
   at java.lang.reflect.Method.invoke(Unknown Source)
   at de.anomic.server.serverCore$Session.listen(serverCore.java:728)
   at de.anomic.server.serverCore$Session.run(serverCore.java:619)
Caused by: java.lang.OutOfMemoryError: Java heap spaceI 2009/07/25 08:12:25 PLASMA Received 28 Entries 12 Words [qRmUHtNldXRr .. qRougaqrqxGJ]/-410446245380744520 from seejciQmifSs:KIT01-01F/0.9100617, processed in 0 milliseconds, requesting 4/28 URLs, blocked 0 RWIsI 2009/07/25 08:12:25 PLASMA Received 0 URLs from peer seejciQmifSs:KIT01-01F/0.9100617 in 0 ms, blocked 4 URLsI 2009/07/25 08:12:26 PLASMA Received 21 Entries 8 Words [qRmiXAm14OUz .. qRougaqrqxGJ]/-410450159397503012 from seejciQmifSs:KIT01-01F/0.9100617, processed in 0 milliseconds, requesting 3/21 URLs, blocked 1 RWIsE 2009/07/25 08:12:28 FILEHANDLER INTERNAL ERROR: java.lang.reflect.InvocationTargetException:null target exception at D:\b4-fam\YaCy-dulcedo2\htroot\yacy\transferURL.class: java.lang.OutOfMemoryError: Java heap space:Java heap space
java.lang.reflect.InvocationTargetException
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
   at java.lang.reflect.Method.invoke(Unknown Source)
   at de.anomic.http.server.HTTPDFileHandler.invokeServlet(HTTPDFileHandler.java:1177)
   at de.anomic.http.server.HTTPDFileHandler.doResponse(HTTPDFileHandler.java:757)
   at de.anomic.http.server.HTTPDFileHandler.doPost(HTTPDFileHandler.java:254)
   at de.anomic.http.server.HTTPDemon.POST(HTTPDemon.java:630)
   at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
   at java.lang.reflect.Method.invoke(Unknown Source)
   at de.anomic.server.serverCore$Session.listen(serverCore.java:728)
   at de.anomic.server.serverCore$Session.run(serverCore.java:619)
Caused by: java.lang.OutOfMemoryError: Java heap spaceW 2009/07/25 08:12:28 FILEHANDLER Unexpected error while processing query.
Session: Session_141.52.175.31:33944#3
Query:   /yacy/transferURL.html
Client:  141.52.175.31
Reason:  java.lang.reflect.InvocationTargetException
java.lang.reflect.InvocationTargetException
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
   at java.lang.reflect.Method.invoke(Unknown Source)
   at de.anomic.http.server.HTTPDFileHandler.invokeServlet(HTTPDFileHandler.java:1177)
   at de.anomic.http.server.HTTPDFileHandler.doResponse(HTTPDFileHandler.java:757)
   at de.anomic.http.server.HTTPDFileHandler.doPost(HTTPDFileHandler.java:254)
   at de.anomic.http.server.HTTPDemon.POST(HTTPDemon.java:630)
   at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
   at java.lang.reflect.Method.invoke(Unknown Source)
   at de.anomic.server.serverCore$Session.listen(serverCore.java:728)
   at de.anomic.server.serverCore$Session.run(serverCore.java:619)
Caused by: java.lang.OutOfMemoryError: Java heap spaceI 2009/07/25 08:12:30 PLASMA Received 16 Entries 7 Words [qRmI4bx-_TXJ .. qRougaqrqxGJ]/-410443156129466168 from seejciQmifSs:KIT01-01F/0.9100617, processed in 0 milliseconds, requesting 5/16 URLs, blocked 0 RWIsE 2009/07/25 08:12:32 FILEHANDLER INTERNAL ERROR: java.lang.reflect.InvocationTargetException:null target exception at D:\b4-fam\YaCy-dulcedo2\htroot\yacy\transferURL.class: java.lang.OutOfMemoryError: Java heap space:Java heap space
java.lang.reflect.InvocationTargetException
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
   at java.lang.reflect.Method.invoke(Unknown Source)
   at de.anomic.http.server.HTTPDFileHandler.invokeServlet(HTTPDFileHandler.java:1177)
   at de.anomic.http.server.HTTPDFileHandler.doResponse(HTTPDFileHandler.java:757)
   at de.anomic.http.server.HTTPDFileHandler.doPost(HTTPDFileHandler.java:254)
   at de.anomic.http.server.HTTPDemon.POST(HTTPDemon.java:630)
   at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
   at java.lang.reflect.Method.invoke(Unknown Source)
   at de.anomic.server.serverCore$Session.listen(serverCore.java:728)
   at de.anomic.server.serverCore$Session.run(serverCore.java:619)
Caused by: java.lang.OutOfMemoryError: Java heap spaceW 2009/07/25 08:12:32 FILEHANDLER Unexpected error while processing query.
Session: Session_141.52.175.31:33944#5
Query:   /yacy/transferURL.html
Client:  141.52.175.31
Reason:  java.lang.reflect.InvocationTargetException
java.lang.reflect.InvocationTargetException
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
   at java.lang.reflect.Method.invoke(Unknown Source)
   at de.anomic.http.server.HTTPDFileHandler.invokeServlet(HTTPDFileHandler.java:1177)
   at de.anomic.http.server.HTTPDFileHandler.doResponse(HTTPDFileHandler.java:757)
   at de.anomic.http.server.HTTPDFileHandler.doPost(HTTPDFileHandler.java:254)
   at de.anomic.http.server.HTTPDemon.POST(HTTPDemon.java:630)
   at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
   at java.lang.reflect.Method.invoke(Unknown Source)
   at de.anomic.server.serverCore$Session.listen(serverCore.java:728)
   at de.anomic.server.serverCore$Session.run(serverCore.java:619)
Caused by: java.lang.OutOfMemoryError: Java heap spaceI 2009/07/25 08:12:33 PLASMA Received 2 Entries 1 Words [TIEwJ_JGzZq3 .. TIEwJ_JGzZq3]/2925692571916237840 from 8TB5J3xHaIVX:Hermes/0.9100624, processed in 0 milliseconds, requesting 0/2 URLs, blocked 0 RWIsI 2009/07/25 08:12:33 PLASMA Received 1 Entries 1 Words [TIEwJ_JGzZq3 .. TIEwJ_JGzZq3]/2925692571916237840 from 8TB5J3xHaIVX:Hermes/0.9100624, processed in 0 milliseconds, requesting 0/1 URLs, blocked 0 RWIsI 2009/07/25 08:12:43 PLASMA dhtTransferJob: selected Qo9QjmTqAXUD as start hashI 2009/07/25 08:12:43 PLASMA dhtTransferJob: selected RrPkRuW9m7mM as limit hashI 2009/07/25 08:12:43 INDEX-TRANSFER-DISPATCHER selectContainersToCache: selectedContainerCache was filled with 8 entriesI 2009/07/25 08:12:43 INDEX-TRANSFER-DISPATCHER splitContainersFromCache: splittedContainerCache filled with 16 partitions, deleting selectedContainerCacheI 2009/07/25 08:12:43 INDEX-TRANSFER-DISPATCHER enqueueContainers: selected 9 targets for primary target key Ao9a7kKpDG__/0 with 8 index containers.E 2009/07/25 08:12:44 BUSYTHREAD Runtime Error in serverInstantThread.job, thread 'de.anomic.search.Switchboard.dhtTransferJob': null; target exception: Java heap space
java.lang.OutOfMemoryError: Java heap spaceI 2009/07/25 08:13:05 YACY hello: responded remote peer 'ICSY' [131.246.103.128] in 358 millisecondsI 2009/07/25 08:13:06 PLASMA Received 3 Entries 3 Words [Qo26bn9URU0y .. Qo3kGQVEmrYz]/3284204221344131804 from 8TB5J3xHaIVX:Hermes/0.9100624, processed in 0 milliseconds, requesting 0/3 URLs, blocked 0 RWIsI 2009/07/25 08:13:14 PLASMA dhtTransferJob: selected UNQ1NBEPNXhQ as start hashI 2009/07/25 08:13:14 PLASMA dhtTransferJob: selected UticSIaYLHkd as limit hashI 2009/07/25 08:13:15 INDEX-TRANSFER-DISPATCHER selectContainersToCache: selectedContainerCache was filled with 12 entriesI 2009/07/25 08:13:15 INDEX-TRANSFER-DISPATCHER splitContainersFromCache: splittedContainerCache filled with 16 partitions, deleting selectedContainerCacheI 2009/07/25 08:13:15 INDEX-TRANSFER-DISPATCHER enqueueContainers: selected 9 targets for primary target key ANQ8C9RaOm__/0 with 12 index containers.E 2009/07/25 08:13:16 BUSYTHREAD Runtime Error in serverInstantThread.job, thread 'de.anomic.search.Switchboard.dhtTransferJob': null; target exception: Java heap space
java.lang.OutOfMemoryError: Java heap spaceD 2009/07/25 08:13:31 CRAWLER LOCALCRAWL[1, 0, 0, 0]: URL=http://www.kochrezepte.at/index_63_63_5_285_1_0__.html, initiator=nbV78UBoxbFi, crawlOrder=false, depth=0, crawlDepth=5, must-match=.*, must-not-match=, permission=trueD 2009/07/25 08:13:31 ROBOTS Trying to download the robots.txt file from URL 'http://www.kochrezepte.at/robots.txt'.D 2009/07/25 08:13:31 ROBOTS Robots.txt successfully loaded from URL 'http://www.kochrezepte.at/robots.txt' in 109 ms.E 2009/07/25 08:13:31 SERVER receive interrupted - exception 2 = Connection resetI 2009/07/25 08:13:31 PLASMA CRAWL: ADDED 40 LINKS FROM http://www.kochrezepte.at/index_63_63_5_285_1_0__.html, NEW CRAWL STACK SIZE IS 0, STACKING TIME = 0, PARSING TIME = 125D 2009/07/25 08:13:33 CRAWLER CoreCrawl: online caution for localsearch, omitting processingE 2009/07/25 08:13:33 BLOCKINGTHREAD Runtime Error in serverInstantThread.job, thread 'java.lang.reflect.Method.job.12': null; target exception: Java heap space
java.lang.OutOfMemoryError: Java heap spaceE 2009/07/25 08:13:35 BLOCKINGTHREAD Runtime Error in serverInstantThread.job, thread 'java.lang.reflect.Method.job.15': null; target exception: Java heap space
java.lang.OutOfMemoryError: Java heap spaceD 2009/07/25 08:13:36 CRAWLER CoreCrawl: online caution for localsearch, omitting processingE 2009/07/25 08:13:36 BLOCKINGTHREAD Runtime Error in serverInstantThread.job, thread 'java.lang.reflect.Method.job.16': null; target exception: Java heap space
java.lang.OutOfMemoryError: Java heap spaceE 2009/07/25 08:13:38 BLOCKINGTHREAD Runtime Error in serverInstantThread.job, thread 'java.lang.reflect.Method.job.16': null; target exception: Java heap space
java.lang.OutOfMemoryError: Java heap spaceD 2009/07/25 08:13:39 CRAWLER CoreCrawl: online caution for localsearch, omitting processingE 2009/07/25 08:13:39 BLOCKINGTHREAD Runtime Error in serverInstantThread.job, thread 'java.lang.reflect.Method.job.16': null; target exception: Java heap space
java.lang.OutOfMemoryError: Java heap spaceE 2009/07/25 08:13:41 BLOCKINGTHREAD Runtime Error in serverInstantThread.job, thread 'java.lang.reflect.Method.job.16': null; target exception: Java heap space
java.lang.OutOfMemoryError: Java heap spaceD 2009/07/25 08:13:42 CRAWLER LOCALCRAWL[9, 0, 0, 0]: URL=http://www.suf.at/webomi/, initiator=nbV78UBoxbFi, crawlOrder=false, depth=1, crawlDepth=5, must-match=.*, must-not-match=, permission=trueD 2009/07/25 08:13:42 ROBOTS Trying to download the robots.txt file from URL 'http://www.suf.at/robots.txt'.E 2009/07/25 08:13:42 BLOCKINGTHREAD Runtime Error in serverInstantThread.job, thread 'java.lang.reflect.Method.job.16': null; target exception: Java heap space
java.lang.OutOfMemoryError: Java heap spaceD 2009/07/25 08:13:42 CRAWLER LOCALCRAWL[8, 0, 0, 0]: URL=http://www.kawo2.rwth-aachen.de/, initiator=nbV78UBoxbFi, crawlOrder=false, depth=1, crawlDepth=5, must-match=.*, must-not-match=, permission=trueE 2009/07/25 08:13:43 YACY yacyClient.queryUrlCount error asking peer '192-168-0-231-1874dpnufe0':java.net.ConnectException: Connection refused: connectI 2009/07/25 08:13:43 YACY hello: responded remote junior peer '192-168-0-231-1874dpnufe0' from 93.209.230.251D 2009/07/25 08:13:43 CRAWLER LOCALCRAWL[7, 0, 0, 0]: URL=http://www.musikschule-spandau.de/rezepte.htm, initiator=nbV78UBoxbFi, crawlOrder=false, depth=1, crawlDepth=5, must-match=.*, must-not-match=, permission=trueD 2009/07/25 08:13:43 ROBOTS Trying to download the robots.txt file from URL 'http://www.musikschule-spandau.de/robots.txt'.E 2009/07/25 08:13:43 BLOCKINGTHREAD Runtime Error in serverInstantThread.job, thread 'java.lang.reflect.Method.job.16': null; target exception: Java heap space
java.lang.OutOfMemoryError: Java heap spaceD 2009/07/25 08:13:43 ROBOTS Trying to download the robots.txt file from URL 'http://www.kawo2.rwth-aachen.de/robots.txt'.D 2009/07/25 08:13:43 CRAWLER LOCALCRAWL[6, 0, 0, 0]: URL=http://www.kochwelt.de/, initiator=nbV78UBoxbFi, crawlOrder=false, depth=1, crawlDepth=5, must-match=.*, must-not-match=, permission=trueD 2009/07/25 08:13:43 ROBOTS Trying to download the robots.txt file from URL 'http://www.kochwelt.de/robots.txt'.I 2009/07/25 08:13:43 YACY hello: responded remote peer '192-168-0-231-1874dpnufe0' [93.209.230.251] in 2932 millisecondsD 2009/07/25 08:13:45 CRAWLER LOCALCRAWL[5, 0, 0, 0]: URL=http://www.kochatelier.de/kochreze.htm, initiator=nbV78UBoxbFi, crawlOrder=false, depth=1, crawlDepth=5, must-match=.*, must-not-match=, permission=trueD 2009/07/25 08:13:45 ROBOTS Trying to download the robots.txt file from URL 'http://www.kochatelier.de/robots.txt'.E 2009/07/25 08:13:45 BLOCKINGTHREAD Runtime Error in serverInstantThread.job, thread 'java.lang.reflect.Method.job.14': null; target exception: Java heap space
java.lang.OutOfMemoryError: Java heap spaceD 2009/07/25 08:13:45 CRAWLER LOCALCRAWL[4, 0, 0, 0]: URL=http://www.bayernservice.de/Kochrezepte.html, initiator=nbV78UBoxbFi, crawlOrder=false, depth=1, crawlDepth=5, must-match=.*, must-not-match=, permission=trueD 2009/07/25 08:13:45 ROBOTS Trying to download the robots.txt file from URL 'http://www.bayernservice.de/robots.txt'.I 2009/07/25 08:13:45 PLASMA Excluded 0 words in URL http://www.kochrezepte.at/index_63_63_5_285_1_0__.htmlI 2009/07/25 08:13:46 PLASMA dhtTransferJob: selected -z2aThL66kGp as start hashI 2009/07/25 08:13:46 PLASMA dhtTransferJob: selected AmP-WNn_Cc__ as limit hashD 2009/07/25 08:13:46 CRAWLER LOCALCRAWL[3, 0, 0, 0]: URL=http://www.bionetz.ch/konsum/kochrezepte/, initiator=nbV78UBoxbFi, crawlOrder=false, depth=1, crawlDepth=5, must-match=.*, must-not-match=, permission=trueD 2009/07/25 08:13:46 ROBOTS Trying to download the robots.txt file from URL 'http://www.bionetz.ch/robots.txt'.E 2009/07/25 08:13:46 BLOCKINGTHREAD Runtime Error in serverInstantThread.job, thread 'java.lang.reflect.Method.job.14': null; target exception: Java heap space
java.lang.OutOfMemoryError: Java heap spaceD 2009/07/25 08:13:46 CRAWLER LOCALCRAWL[6, 0, 0, 0]: URL=http://www.dccv.de/rezepte/, initiator=nbV78UBoxbFi, crawlOrder=false, depth=1, crawlDepth=5, must-match=.*, must-not-match=, permission=trueD 2009/07/25 08:13:46 ROBOTS Trying to download the robots.txt file from URL 'http://www.dccv.de/robots.txt'.D 2009/07/25 08:13:48 ROBOTS Robots.txt successfully loaded from URL 'http://www.kochwelt.de/robots.txt' in 4368 ms.E 2009/07/25 08:13:48 BLOCKINGTHREAD Runtime Error in serverInstantThread.job, thread 'java.lang.reflect.Method.storeDocumentIndex.17': null; target exception: Java heap space
java.lang.OutOfMemoryError: Java heap spaceD 2009/07/25 08:13:48 CRAWLER LOCALCRAWL[6, 0, 0, 0]: URL=http://www.maggi-kochrezepte.de/, initiator=nbV78UBoxbFi, crawlOrder=false, depth=1, crawlDepth=5, must-match=.*, must-not-match=, permission=trueD 2009/07/25 08:13:48 ROBOTS Robots.txt successfully loaded from URL 'http://www.kawo2.rwth-aachen.de/robots.txt' in 4399 ms.W 2009/07/25 08:13:48 ReferenceContainerArray timout in index retrieval (1): 1 tables searched. timeout = 1000D 2009/07/25 08:13:48 ROBOTS Trying to download the robots.txt file from URL 'http://www.maggi-kochrezepte.de/robots.txt'.D 2009/07/25 08:13:48 ROBOTS Robots.txt successfully loaded from URL 'http://www.musikschule-spandau.de/robots.txt' in 4415 ms.D 2009/07/25 08:13:48 CRAWLER LOCALCRAWL[7, 0, 0, 0]: URL=http://www.dein-allgaeu.de/regionen/regionen_.html, initiator=nbV78UBoxbFi, crawlOrder=false, depth=1, crawlDepth=5, must-match=.*, must-not-match=, permission=trueD 2009/07/25 08:13:49 ROBOTS Robots.txt successfully loaded from URL 'http://www.bionetz.ch/robots.txt' in 2948 ms.W 2009/07/25 08:13:49 ReferenceContainerArray timout in index retrieval (1): 1 tables searched. timeout = 1000D 2009/07/25 08:13:49 CRAWLER LOCALCRAWL[6, 0, 0, 0]: URL=http://www.studis-online.de/Life/Rezepte/, initiator=nbV78UBoxbFi, crawlOrder=false, depth=1, crawlDepth=5, must-match=.*, must-not-match=, permission=trueD 2009/07/25 08:13:49 ROBOTS Trying to download the robots.txt file from URL 'http://www.dein-allgaeu.de/robots.txt'.D 2009/07/25 08:13:49 CRAWLER LOCALCRAWL[5, 0, 0, 0]: URL=http://www.toblach.it/de/lifestyle/, initiator=nbV78UBoxbFi, crawlOrder=false, depth=1, crawlDepth=5, must-match=.*, must-not-match=, permission=trueD 2009/07/25 08:13:49 ROBOTS Trying to download the robots.txt file from URL 'http://www.studis-online.de/robots.txt'.D 2009/07/25 08:13:49 ROBOTS robots.txt could not be downloaded from URL 'http://www.kochatelier.de/robots.txt'. [404 Not Found].D 2009/07/25 08:13:49 ROBOTS Trying to download the robots.txt file from URL 'http://www.toblach.it/robots.txt'.D 2009/07/25 08:13:49 ROBOTS Robots.txt from URL 'http://www.bayernservice.de/robots.txt' has wrong mimetype 'text/html'.I 2009/07/25 08:13:49 INDEX-TRANSFER-DISPATCHER selectContainersToCache: selectedContainerCache was filled with 1 entriesI 2009/07/25 08:13:49 INDEX-TRANSFER-DISPATCHER splitContainersFromCache: splittedContainerCache filled with 16 partitions, deleting selectedContainerCacheI 2009/07/25 08:13:49 INDEX-TRANSFER-DISPATCHER enqueueContainers: selected 9 targets for primary target key Cz2ab_hpAq__/0 with 1 index containers.I 2009/07/25 08:13:49 INDEX-TRANSFER-DISPATCHER enqueueContainers: selected 9 targets for primary target key Gz2ab_hpAq__/1 with 1 index containers.I 2009/07/25 08:13:49 INDEX-TRANSFER-DISPATCHER enqueueContainers: selected 9 targets for primary target key Kz2ab_hpAq__/2 with 1 index containers.D 2009/07/25 08:13:49 CRAWLER LOCALCRAWL[4, 0, 0, 0]: URL=http://www.saarbruecken.de/sbnet/11a/11_run.htm, initiator=nbV78UBoxbFi, crawlOrder=false, depth=1, crawlDepth=5, must-match=.*, must-not-match=, permission=trueD 2009/07/25 08:13:49 ROBOTS Trying to download the robots.txt file from URL 'http://www.saarbruecken.de/robots.txt'.D 2009/07/25 08:13:49 CRAWLER LOCALCRAWL[3, 0, 0, 0]: URL=http://www.germanien.net/werner.eichelberg/rezepte.htm, initiator=nbV78UBoxbFi, crawlOrder=false, depth=1, crawlDepth=5, must-match=.*, must-not-match=, permission=trueD 2009/07/25 08:13:49 ROBOTS Trying to download the robots.txt file from URL 'http://www.germanien.net/robots.txt'.D 2009/07/25 08:13:52 ROBOTS robots.txt could not be downloaded from URL 'http://www.dccv.de/robots.txt'. [404 Not Found].D 2009/07/25 08:13:52 ROBOTS Robots.txt from URL 'http://www.suf.at/robots.txt' has wrong mimetype 'text/html'.D 2009/07/25 08:13:52 CRAWLER LOCALCRAWL[2, 0, 0, 0]: URL=http://www.altmuehltal.de/allgemein/, initiator=nbV78UBoxbFi, crawlOrder=false, depth=1, crawlDepth=5, must-match=.*, must-not-match=, permission=trueI 2009/07/25 08:13:52 YACY hello: responded remote peer 'endeavour' [87.155.255.225] in 3838 millisecondsD 2009/07/25 08:13:52 ROBOTS Trying to download the robots.txt file from URL 'http://www.altmuehltal.de/robots.txt'.I 2009/07/25 08:13:54 YACY yacyClient.publishMySeed thread 'PublishSeed_KIT01-09-n66' contacted peer at 141.52.175.66:8080, received 9934 bytes, time = 7581 millisecondsI 2009/07/25 08:13:54 YACY yacyClient.publishMySeed: Peer 'KIT01-09-n66' reported us as junior.D 2009/07/25 08:13:54 CRAWLER LOCALCRAWL[1, 0, 0, 0]: URL=http://www.gesund.co.at/gesund/, initiator=nbV78UBoxbFi, crawlOrder=false, depth=1, crawlDepth=5, must-match=.*, must-not-match=, permission=trueE 2009/07/25 08:13:54 BLOCKINGTHREAD Runtime Error in serverInstantThread.job, thread 'java.lang.reflect.Method.job.14': null; target exception: Java heap space
java.lang.OutOfMemoryError: Java heap spaceD 2009/07/25 08:13:54 ROBOTS Trying to download the robots.txt file from URL 'http://www.gesund.co.at/robots.txt'.E 2009/07/25 08:13:54 BLOCKINGTHREAD Runtime Error in serverInstantThread.job, thread 'java.lang.reflect.Method.job.13': null; target exception: Java heap space
java.lang.OutOfMemoryError: Java heap spaceD 2009/07/25 08:13:55 ROBOTS Robots.txt successfully loaded from URL 'http://www.germanien.net/robots.txt' in 6037 ms.D 2009/07/25 08:13:55 ROBOTS Robots.txt successfully loaded from URL 'http://www.saarbruecken.de/robots.txt' in 6053 ms.D 2009/07/25 08:13:55 ROBOTS Robots.txt successfully loaded from URL 'http://www.dein-allgaeu.de/robots.txt' in 6084 ms.D 2009/07/25 08:13:55 ROBOTS Robots.txt successfully loaded from URL 'http://www.studis-online.de/robots.txt' in 6068 ms.I 2009/07/25 08:13:55 YACY publish: handshaked senior peer 'KIT01-09-n66' at 141.52.175.66:8080I 2009/07/25 08:13:55 YACY PeerPing: I am accessible for 6 peer(s), not accessible for 2 peer(s).I 2009/07/25 08:13:55 YACY PeerPing: myType is seniorE 2009/07/25 08:13:55 BLOCKINGTHREAD Runtime Error in serverInstantThread.job, thread 'java.lang.reflect.Method.job.15': null; target exception: Java heap space
java.lang.OutOfMemoryError: Java heap spaceD 2009/07/25 08:13:57 CRAWLER LOCALCRAWL[2, 0, 0, 0]: URL=http://www.bad-bad.de/restaur/rezepte/inh_rez.htm, initiator=nbV78UBoxbFi, crawlOrder=false, depth=1, crawlDepth=5, must-match=.*, must-not-match=, permission=trueD 2009/07/25 08:13:57 ROBOTS Robots.txt successfully loaded from URL 'http://www.altmuehltal.de/robots.txt' in 5194 ms.D 2009/07/25 08:13:58 ROBOTS robots.txt could not be downloaded from URL 'http://www.gesund.co.at/robots.txt'. [404 Not Found].D 2009/07/25 08:13:58 CRAWLER LOCALCRAWL[1, 0, 0, 0]: URL=http://www.andechs.de/essen/kochrezepte/, initiator=nbV78UBoxbFi, crawlOrder=false, depth=1, crawlDepth=5, must-match=.*, must-not-match=, permission=trueD 2009/07/25 08:14:00 ROBOTS Trying to download the robots.txt file from URL 'http://www.andechs.de/robots.txt'.D 2009/07/25 08:14:00 ROBOTS robots.txt could not be downloaded from URL 'http://www.toblach.it/robots.txt'. [404 Not Found].D 2009/07/25 08:14:03 ROBOTS Robots.txt from URL 'http://www.andechs.de/robots.txt' has wrong mimetype 'text/html'.E 2009/07/25 08:14:03 BLOCKINGTHREAD Runtime Error in serverInstantThread.job, thread 'java.lang.reflect.Method.job.12': null; target exception: Java heap space
java.lang.OutOfMemoryError: Java heap spaceE 2009/07/25 08:14:04 BLOCKINGTHREAD Runtime Error in serverInstantThread.job, thread 'java.lang.reflect.Method.job.13': null; target exception: Java heap space
java.lang.OutOfMemoryError: Java heap spaceI 2009/07/25 08:14:04 PLASMA Received 1 Entries 1 Words [IVV8aOf2xjmg .. IVV8aOf2xjmg]/4481081372277538752 from 8TB5J3xHaIVX:Hermes/0.9100624, processed in 16 milliseconds, requesting 0/1 URLs, blocked 0 RWIsD 2009/07/25 08:14:07 CRAWLER LOCALCRAWL[1, 0, 0, 0]: URL=http://www.lust-auf.ch/1/sidessen.html, initiator=nbV78UBoxbFi, crawlOrder=false, depth=1, crawlDepth=5, must-match=.*, must-not-match=, permission=trueD 2009/07/25 08:14:08 ROBOTS Trying to download the robots.txt file from URL 'http://www.lust-auf.ch/robots.txt'.E 2009/07/25 08:14:08 BLOCKINGTHREAD Runtime Error in serverInstantThread.job, thread 'java.lang.reflect.Method.job.14': null; target exception: Java heap space
java.lang.OutOfMemoryError: Java heap spaceI 2009/07/25 08:14:08 INDEX-TRANSFER-DISPATCHER enqueueContainers: selected 9 targets for primary target key Oz2ab_hpAq__/3 with 1 index containers.D 2009/07/25 08:14:11 CRAWLER LOCALCRAWL[1, 0, 0, 0]: URL=http://www.bayernservice.de/Bayrische%20Rezepte%20Auswahl.htm, initiator=nbV78UBoxbFi, crawlOrder=false, depth=1, crawlDepth=5, must-match=.*, must-not-match=, permission=trueD 2009/07/25 08:14:13 ROBOTS Robots.txt successfully loaded from URL 'http://www.lust-auf.ch/robots.txt' in 5647 ms.E 2009/07/25 08:14:13 BLOCKINGTHREAD Runtime Error in serverInstantThread.job, thread 'java.lang.reflect.Method.job.16': null; target exception: Java heap space
java.lang.OutOfMemoryError: Java heap spaceE 2009/07/25 08:14:22 BUSYTHREAD Runtime Error in serverInstantThread.job, thread 'de.anomic.search.Switchboard.dhtTransferJob': null; target exception: Java heap space
java.lang.OutOfMemoryError: Java heap spaceE 2009/07/25 08:14:22 BLOCKINGTHREAD Runtime Error in serverInstantThread.job, thread 'java.lang.reflect.Method.job.13': null; target exception: Java heap space
java.lang.OutOfMemoryError: Java heap spaceE 2009/07/25 08:14:24 BLOCKINGTHREAD Runtime Error in serverInstantThread.job, thread 'java.lang.reflect.Method.job.12': null; target exception: Java heap space
java.lang.OutOfMemoryError: Java heap spaceE 2009/07/25 08:14:33 BLOCKINGTHREAD Runtime Error in serverInstantThread.job, thread 'java.lang.reflect.Method.job.15': null; target exception: Java heap space
java.lang.OutOfMemoryError: Java heap spaceE 2009/07/25 08:14:39 SERVER receive interrupted - exception 2 = Connection resetI 2009/07/25 08:14:51 PLASMA dhtTransferJob: no selection, too many entries in transmission cloud: 65I 2009/07/25 08:14:51 INDEX-TRANSFER-DISPATCHER starting new index transmission request to Kz2ab_hpAq__I 2009/07/25 08:14:51 PLASMA dhtTransferJob: result from dequeueing: trueI 2009/07/25 08:14:51 INDEX-TRANSFER-DISPATCHER Index transfer of 1 words [-z2ab_hpAq29 .. Kz2ab_hpAq__] and 1 URLs to peer KIT01-02f:Mlk_sHB93XBf in 0 seconds successful (3 words/s)I 2009/07/25 08:14:51 INDEX-TRANSFER-DISPATCHER Transfer finished of chunk to target Mlk_sHB93XBf/KIT01-02fI 2009/07/25 08:14:51 INDEX-TRANSFER-DISPATCHER starting new index transmission request to Kz2ab_hpAq__
dulcedo
 
Beiträge: 1006
Registriert: Do Okt 16, 2008 6:36 pm
Wohnort: Bei Karlsruhe

Re: SVN ab 6200 crawl-verhalten

Beitragvon Orbiter » So Jul 26, 2009 2:27 am

ich kann mir momentan keinen Reim darauf machen machen wie das Crawling den transferURL betrifft. Auch schon deswegen weil der transferRWI eigentlich mehr RAM braucht, glaube ich. Warum bekommen wir eigentlich bei den exceptions innerhalb von code der über reflection ausgelöst wurde keine Angabe über die genaue Codestelle im Servlet, hat da jemand eine Idee wie man das forcieren kann?
Orbiter
 
Beiträge: 5792
Registriert: Di Jun 26, 2007 10:58 pm
Wohnort: Frankfurt am Main

Re: SVN ab 6200 crawl-verhalten

Beitragvon dulcedo » So Jul 26, 2009 5:08 am

Fixiere dich nich zu sehr auf das Crawling, ich denke es war Zufall dass ich gerade dann ins Log geschaut habe.
Es ist auch der selbe Peer wie dieser Bug, eventuell eine Datei kaputt?
dulcedo
 
Beiträge: 1006
Registriert: Do Okt 16, 2008 6:36 pm
Wohnort: Bei Karlsruhe


Zurück zu Fragen und Antworten

Wer ist online?

Mitglieder in diesem Forum: 0 Mitglieder und 4 Gäste

cron