6836 OOMs bei (such)zugriffen

Hier finden YaCy User Hilfe wenn was nicht funktioniert oder anders funktioniert als man dachte. Bei offensichtlichen Fehlern diese bitte gleich in die Bugs (http://bugs.yacy.net) eintragen.
Forumsregeln
In diesem Forum geht es um Benutzungsprobleme und Anfragen für Hilfe. Wird dabei ein Bug identifiziert, wird der thread zur Bearbeitung in die Bug-Sektion verschoben. Wer hier also einen Thread eingestellt hat und ihn vermisst, wird ihn sicherlich in der Bug-Sektion wiederfinden.

6836 OOMs bei (such)zugriffen

Beitragvon dulcedo » Sa Apr 24, 2010 1:24 pm

Diese Fehler kommen reihenweise, 6828 hatte tagelang einwandfrei funktioniert.
Performancegrafik zeigt keine Aktivität, peer aber normal bedienbar.
Code: Alles auswählen
I 2010/04/24 13:53:26 YACY INIT HASH SEARCH (query-auto): [Gmm........., beV.........] - 10 links

I 2010/04/24 13:53:27 YACY INIT HASH SEARCH (query-auto): [Gmm........., beV.........] - 10 links

E 2010/04/24 13:53:36 FILEHANDLER INTERNAL ERROR: java.lang.reflect.InvocationTargetException:null target exception at /home/yacy/yacy-b/htroot/yacy/search.class: java.lang.OutOfMemoryError: Java heap space:Java heap space
java.lang.reflect.InvocationTargetException
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at de.anomic.http.server.HTTPDFileHandler.invokeServlet(HTTPDFileHandler.java:1197)
   at de.anomic.http.server.HTTPDFileHandler.doResponse(HTTPDFileHandler.java:754)
   at de.anomic.http.server.HTTPDFileHandler.doPost(HTTPDFileHandler.java:243)
   at de.anomic.http.server.HTTPDemon.POST(HTTPDemon.java:587)
   at sun.reflect.GeneratedMethodAccessor16.invoke(Unknown Source)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at de.anomic.server.serverCore$Session.listen(serverCore.java:728)
   at de.anomic.server.serverCore$Session.run(serverCore.java:626)
Caused by: java.lang.OutOfMemoryError: Java heap space

W 2010/04/24 13:53:36 FILEHANDLER Unexpected error while processing query.
Session: Session_141.52.175.26:43506#0
Query:   /yacy/search.html
Client:  141.52.175.26
Reason:  java.lang.reflect.InvocationTargetException
java.lang.reflect.InvocationTargetException
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at de.anomic.http.server.HTTPDFileHandler.invokeServlet(HTTPDFileHandler.java:1197)
   at de.anomic.http.server.HTTPDFileHandler.doResponse(HTTPDFileHandler.java:754)
   at de.anomic.http.server.HTTPDFileHandler.doPost(HTTPDFileHandler.java:243)
   at de.anomic.http.server.HTTPDemon.POST(HTTPDemon.java:587)
   at sun.reflect.GeneratedMethodAccessor16.invoke(Unknown Source)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at de.anomic.server.serverCore$Session.listen(serverCore.java:728)
   at de.anomic.server.serverCore$Session.run(serverCore.java:626)
Caused by: java.lang.OutOfMemoryError: Java heap space

I 2010/04/24 13:53:36 YACY INIT HASH SEARCH (query-auto): [Gmm........., beV.........] - 10 links

E 2010/04/24 13:53:45 FILEHANDLER INTERNAL ERROR: java.lang.reflect.InvocationTargetException:null target exception at /home/yacy/yacy-b/htroot/yacy/search.class: java.lang.OutOfMemoryError: Java heap space:Java heap space
java.lang.reflect.InvocationTargetException
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at de.anomic.http.server.HTTPDFileHandler.invokeServlet(HTTPDFileHandler.java:1197)
   at de.anomic.http.server.HTTPDFileHandler.doResponse(HTTPDFileHandler.java:754)
   at de.anomic.http.server.HTTPDFileHandler.doPost(HTTPDFileHandler.java:243)
   at de.anomic.http.server.HTTPDemon.POST(HTTPDemon.java:587)
   at sun.reflect.GeneratedMethodAccessor16.invoke(Unknown Source)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at de.anomic.server.serverCore$Session.listen(serverCore.java:728)
   at de.anomic.server.serverCore$Session.run(serverCore.java:626)
Caused by: java.lang.OutOfMemoryError: Java heap space

W 2010/04/24 13:53:54 FILEHANDLER Unexpected error while processing query.
Session: Session_141.52.175.26:58077#0
Query:   /yacy/search.html
Client:  141.52.175.26
Reason:  java.lang.reflect.InvocationTargetException
java.lang.reflect.InvocationTargetException
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at de.anomic.http.server.HTTPDFileHandler.invokeServlet(HTTPDFileHandler.java:1197)
   at de.anomic.http.server.HTTPDFileHandler.doResponse(HTTPDFileHandler.java:754)
   at de.anomic.http.server.HTTPDFileHandler.doPost(HTTPDFileHandler.java:243)
   at de.anomic.http.server.HTTPDemon.POST(HTTPDemon.java:587)
   at sun.reflect.GeneratedMethodAccessor16.invoke(Unknown Source)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at de.anomic.server.serverCore$Session.listen(serverCore.java:728)
   at de.anomic.server.serverCore$Session.run(serverCore.java:626)
Caused by: java.lang.OutOfMemoryError: Java heap space

I 2010/04/24 13:53:54 YACY yacyClient.publishMySeed thread 'PublishSeed_sixcooler1', peer 85.178.88.140:8000; exception: The host did not accept the connection within timeout of 30000 ms; retry = 1

E 2010/04/24 13:53:55 FILEHANDLER INTERNAL ERROR: java.lang.reflect.InvocationTargetException:null target exception at /home/yacy/yacy-b/htroot/yacy/search.class: java.lang.OutOfMemoryError: Java heap space:Java heap space
java.lang.reflect.InvocationTargetException
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at de.anomic.http.server.HTTPDFileHandler.invokeServlet(HTTPDFileHandler.java:1197)
   at de.anomic.http.server.HTTPDFileHandler.doResponse(HTTPDFileHandler.java:754)
   at de.anomic.http.server.HTTPDFileHandler.doPost(HTTPDFileHandler.java:243)
   at de.anomic.http.server.HTTPDemon.POST(HTTPDemon.java:587)
   at sun.reflect.GeneratedMethodAccessor16.invoke(Unknown Source)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at de.anomic.server.serverCore$Session.listen(serverCore.java:728)
   at de.anomic.server.serverCore$Session.run(serverCore.java:626)
Caused by: java.lang.OutOfMemoryError: Java heap space

W 2010/04/24 13:53:55 FILEHANDLER Unexpected error while processing query.
Session: Session_141.52.175.26:37223#0
Query:   /yacy/search.html
Client:  141.52.175.26
Reason:  java.lang.reflect.InvocationTargetException
java.lang.reflect.InvocationTargetException
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at de.anomic.http.server.HTTPDFileHandler.invokeServlet(HTTPDFileHandler.java:1197)
   at de.anomic.http.server.HTTPDFileHandler.doResponse(HTTPDFileHandler.java:754)
   at de.anomic.http.server.HTTPDFileHandler.doPost(HTTPDFileHandler.java:243)
   at de.anomic.http.server.HTTPDemon.POST(HTTPDemon.java:587)
   at sun.reflect.GeneratedMethodAccessor16.invoke(Unknown Source)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at de.anomic.server.serverCore$Session.listen(serverCore.java:728)
   at de.anomic.server.serverCore$Session.run(serverCore.java:626)
Caused by: java.lang.OutOfMemoryError: Java heap space

I 2010/04/24 13:53:55 YACY INIT HASH SEARCH (query-auto): [Gmm........., beV.........] - 10 links

I 2010/04/24 13:53:55 YACY INIT HASH SEARCH (query-auto): [Gmm........., beV.........] - 10 links

I 2010/04/24 13:53:55 YACY hello: responded remote peer 'matt23' [79.222.106.163] in 105 milliseconds

E 2010/04/24 13:54:06 FILEHANDLER INTERNAL ERROR: java.lang.reflect.InvocationTargetException:null target exception at /home/yacy/yacy-b/htroot/yacy/search.class: java.lang.OutOfMemoryError: Java heap space:Java heap space
java.lang.reflect.InvocationTargetException
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at de.anomic.http.server.HTTPDFileHandler.invokeServlet(HTTPDFileHandler.java:1197)
   at de.anomic.http.server.HTTPDFileHandler.doResponse(HTTPDFileHandler.java:754)
   at de.anomic.http.server.HTTPDFileHandler.doPost(HTTPDFileHandler.java:243)
   at de.anomic.http.server.HTTPDemon.POST(HTTPDemon.java:587)
   at sun.reflect.GeneratedMethodAccessor16.invoke(Unknown Source)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at de.anomic.server.serverCore$Session.listen(serverCore.java:728)
   at de.anomic.server.serverCore$Session.run(serverCore.java:626)
Caused by: java.lang.OutOfMemoryError: Java heap space

W 2010/04/24 13:54:06 FILEHANDLER Unexpected error while processing query.
Session: Session_141.52.175.26:35691#0
Query:   /yacy/search.html
Client:  141.52.175.26
Reason:  java.lang.reflect.InvocationTargetException
java.lang.reflect.InvocationTargetException
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at de.anomic.http.server.HTTPDFileHandler.invokeServlet(HTTPDFileHandler.java:1197)
   at de.anomic.http.server.HTTPDFileHandler.doResponse(HTTPDFileHandler.java:754)
   at de.anomic.http.server.HTTPDFileHandler.doPost(HTTPDFileHandler.java:243)
   at de.anomic.http.server.HTTPDemon.POST(HTTPDemon.java:587)
   at sun.reflect.GeneratedMethodAccessor16.invoke(Unknown Source)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at de.anomic.server.serverCore$Session.listen(serverCore.java:728)
   at de.anomic.server.serverCore$Session.run(serverCore.java:626)
Caused by: java.lang.OutOfMemoryError: Java heap space

E 2010/04/24 13:54:16 FILEHANDLER INTERNAL ERROR: java.lang.reflect.InvocationTargetException:null target exception at /home/yacy/yacy-b/htroot/yacy/search.class: java.lang.OutOfMemoryError: Java heap space:Java heap space
java.lang.reflect.InvocationTargetException
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at de.anomic.http.server.HTTPDFileHandler.invokeServlet(HTTPDFileHandler.java:1197)
   at de.anomic.http.server.HTTPDFileHandler.doResponse(HTTPDFileHandler.java:754)
   at de.anomic.http.server.HTTPDFileHandler.doPost(HTTPDFileHandler.java:243)
   at de.anomic.http.server.HTTPDemon.POST(HTTPDemon.java:587)
   at sun.reflect.GeneratedMethodAccessor16.invoke(Unknown Source)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at de.anomic.server.serverCore$Session.listen(serverCore.java:728)
   at de.anomic.server.serverCore$Session.run(serverCore.java:626)
Caused by: java.lang.OutOfMemoryError: Java heap space

W 2010/04/24 13:54:25 FILEHANDLER Unexpected error while processing query.
Session: Session_141.52.175.26:50826#0
Query:   /yacy/search.html
Client:  141.52.175.26
Reason:  java.lang.reflect.InvocationTargetException
java.lang.reflect.InvocationTargetException
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at de.anomic.http.server.HTTPDFileHandler.invokeServlet(HTTPDFileHandler.java:1197)
   at de.anomic.http.server.HTTPDFileHandler.doResponse(HTTPDFileHandler.java:754)
   at de.anomic.http.server.HTTPDFileHandler.doPost(HTTPDFileHandler.java:243)
   at de.anomic.http.server.HTTPDemon.POST(HTTPDemon.java:587)
   at sun.reflect.GeneratedMethodAccessor16.invoke(Unknown Source)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at de.anomic.server.serverCore$Session.listen(serverCore.java:728)
   at de.anomic.server.serverCore$Session.run(serverCore.java:626)
Caused by: java.lang.OutOfMemoryError: Java heap space

I 2010/04/24 13:54:25 YACY INIT HASH SEARCH (query-auto): [Gmm........., beV.........] - 10 links

I 2010/04/24 13:54:25 YACY yacyClient.publishMySeed thread 'PublishSeed_sixcooler1', peer 85.178.88.140:8000; exception: The host did not accept the connection within timeout of 30000 ms; retry = 2

E 2010/04/24 13:54:26 FILEHANDLER INTERNAL ERROR: java.lang.reflect.InvocationTargetException:null target exception at /home/yacy/yacy-b/htroot/yacy/search.class: java.lang.OutOfMemoryError: Java heap space:Java heap space
java.lang.reflect.InvocationTargetException
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at de.anomic.http.server.HTTPDFileHandler.invokeServlet(HTTPDFileHandler.java:1197)
   at de.anomic.http.server.HTTPDFileHandler.doResponse(HTTPDFileHandler.java:754)
   at de.anomic.http.server.HTTPDFileHandler.doPost(HTTPDFileHandler.java:243)
   at de.anomic.http.server.HTTPDemon.POST(HTTPDemon.java:587)
   at sun.reflect.GeneratedMethodAccessor16.invoke(Unknown Source)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at de.anomic.server.serverCore$Session.listen(serverCore.java:728)
   at de.anomic.server.serverCore$Session.run(serverCore.java:626)
Caused by: java.lang.OutOfMemoryError: Java heap space

W 2010/04/24 13:54:26 FILEHANDLER Unexpected error while processing query.
Session: Session_141.52.175.26:55833#0
Query:   /yacy/search.html
Client:  141.52.175.26
Reason:  java.lang.reflect.InvocationTargetException
java.lang.reflect.InvocationTargetException
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at de.anomic.http.server.HTTPDFileHandler.invokeServlet(HTTPDFileHandler.java:1197)
   at de.anomic.http.server.HTTPDFileHandler.doResponse(HTTPDFileHandler.java:754)
   at de.anomic.http.server.HTTPDFileHandler.doPost(HTTPDFileHandler.java:243)
   at de.anomic.http.server.HTTPDemon.POST(HTTPDemon.java:587)
   at sun.reflect.GeneratedMethodAccessor16.invoke(Unknown Source)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at de.anomic.server.serverCore$Session.listen(serverCore.java:728)
   at de.anomic.server.serverCore$Session.run(serverCore.java:626)
Caused by: java.lang.OutOfMemoryError: Java heap space
dulcedo
 
Beiträge: 1006
Registriert: Do Okt 16, 2008 6:36 pm
Wohnort: Bei Karlsruhe

Re: 6836 OOMs bei (such)zugriffen

Beitragvon Quix0r » So Apr 25, 2010 3:27 am

Lad die Grafik mal bitte mit hoch. :) Und kannst du vielleicht noch etwas Speicher fuer YaCy opfern? Bei mir war dies der letzte Ausweg vor dem Loeschen von URLs/RWIs. Nur ist bei mir langsam Ende mit Speicher, ich swappe bereits schon (ohne Swap geht mittlerweile gar nichts mehr, was ohne YaCy kein Problem war).

Was aber wohl normal ist, mir aber gleiches aufgefallen ist, dass Suchanfragen ganz schoen mehr Speicher brauchen. Vielleicht koennen die Such-Threads etwas reduziert werden?
Quix0r
 
Beiträge: 1347
Registriert: Di Jul 31, 2007 9:22 am
Wohnort: Krefeld

Re: 6836 OOMs bei (such)zugriffen

Beitragvon dulcedo » So Apr 25, 2010 6:22 am

Die Grafik ist dann leer deshalb habe ich sie auch nicht gezeigt und mehr Speicher macht die Maschine viel zu langsam.
Wenn ich den hier 1GB grossen response-header lösche, oder zu 6824 zurückgehe hat er wieder etwas Luft. Kann ich dessen Grösse begrenzen oder läuft das auch unter maxFilesize?
dulcedo
 
Beiträge: 1006
Registriert: Do Okt 16, 2008 6:36 pm
Wohnort: Bei Karlsruhe

Re: 6836 OOMs bei (such)zugriffen

Beitragvon Quix0r » So Apr 25, 2010 10:59 am

Die OOM nun etwas detailierter:
Code: Alles auswählen
Problems with request: GET http://localpeer/yacysearch.html?query=dove&Enter=Search&contentdom=image&former=taube&maximumRecords=64&startRecord=0&verify=true&resource=global&nav=all&urlmaskfilter=.*&prefermaskfilter=&depth=0&cat=href&display=2&constraint=&meanCount=5

500 Internal Server Error

Unexpected error while processing query.
Session: Session_192.168.1.17:40465#0
Query: /yacysearch.html
Client: 192.168.1.17
Reason: java.lang.reflect.InvocationTargetException

Exception occured: java.lang.reflect.InvocationTargetException

TRACE:
      java.lang.reflect.InvocationTargetException
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at de.anomic.http.server.HTTPDFileHandler.invokeServlet(HTTPDFileHandler.java:1197)
   at de.anomic.http.server.HTTPDFileHandler.doResponse(HTTPDFileHandler.java:754)
   at de.anomic.http.server.HTTPDFileHandler.doGet(HTTPDFileHandler.java:235)
   at de.anomic.http.server.HTTPDemon.GET(HTTPDemon.java:454)
   at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at de.anomic.server.serverCore$Session.listen(serverCore.java:728)
   at de.anomic.server.serverCore$Session.run(serverCore.java:626)
Caused by: java.lang.OutOfMemoryError: Java heap space
   at net.yacy.kelondro.index.RowSet.importRowSet(RowSet.java:81)
   at net.yacy.kelondro.rwi.ReferenceContainerArray$BLOBRewriter.rewrite(ReferenceContainerArray.java:266)
   at net.yacy.kelondro.blob.HeapModifier.replace(HeapModifier.java:275)
   at net.yacy.kelondro.blob.ArrayStack.replace(ArrayStack.java:721)
   at net.yacy.kelondro.rwi.ReferenceContainerArray.replace(ReferenceContainerArray.java:251)
   at net.yacy.kelondro.rwi.IndexCell.remove(IndexCell.java:255)
   at de.anomic.search.SearchEvent.cleanup(SearchEvent.java:237)
   at de.anomic.search.SearchEventCache.cleanupEvents(SearchEventCache.java:57)
   at de.anomic.search.SearchEvent.(SearchEvent.java:193)
   at de.anomic.search.SearchEventCache.getEvent(SearchEventCache.java:95)
   at yacysearch.respond(yacysearch.java:499)
   ... 13 more

screenshot-20100425_1204.png
Waehren er hing
screenshot-20100425_1204.png (13.01 KiB) 602-mal betrachtet


Dazu auch ein anderes Posting: viewtopic.php?f=5&t=2720&p=19509&hilit=OutOfMemoryError#p19509

DHT scheint ~400 bis 512 MB RAM zusaetzlich zu verlangen: viewtopic.php?f=5&t=2796&hilit=
Dateianhänge
jstack04.log.bz2
jstack -l
(9.8 KiB) 26-mal heruntergeladen
Zuletzt geändert von Quix0r am So Apr 25, 2010 11:39 am, insgesamt 1-mal geändert.
Quix0r
 
Beiträge: 1347
Registriert: Di Jul 31, 2007 9:22 am
Wohnort: Krefeld

Re: 6836 OOMs bei (such)zugriffen

Beitragvon Quix0r » So Apr 25, 2010 11:24 am

Hmm, mit weniger RWIs (~200.000) hatte ich nicht solch Probleme, da konnte ich suchen wie ich wollte. Derzeit hat mein Peer ~2.9 Mio. RWIs und 30.8 Mio. URLs (zuvor: 30 Mio. URLs und die ~200.000 RWIs, er hatte wohl Blobs geloescht, die zu stark beschaedigt waren wegen Festplattenplatzmangel).
Quix0r
 
Beiträge: 1347
Registriert: Di Jul 31, 2007 9:22 am
Wohnort: Krefeld

Re: 6836 OOMs bei (such)zugriffen

Beitragvon dulcedo » So Apr 25, 2010 4:47 pm

Mit insgesamt 8,5GB läuft er nun problemlos, im Betrieb braucht er aber sehr viel weniger Speicher. Er kann dann trotz Grösse recht konstant 500ppm crawlen und dabei verteilen.
Das ist insofern schade weil es einen recht hohen Kostenfaktor darstellt ob man 8GB Hauptspeicher oder mehr verwendet. Hier wird aber quasi 1GB nur benötigt damit der peer eine bestimmte Aufgabe ausführt.
Tätigkeiten während der vollen Speicherauslastung hier geloggt.
Code: Alles auswählen
I 2010/04/25 17:24:06 SERVER Initializing serverCore ...
I 2010/04/25 17:24:06 SERVER Trying to bind server to port 8092
S 2010/04/25 17:24:06 BusyThread thread 'httpd:8092' deployed, starting loop.
S 2010/04/25 17:24:07 BusyThread thread 'de.anomic.yacy.yacyCore.peerPing' delayed, starting now loop.
S 2010/04/25 17:24:11 BusyThread thread 'de.anomic.search.Switchboard.dhtTransferJob' delayed, starting now loop.
I 2010/04/25 17:24:11 STARTUP Copied the default locales to /home/yacy/yacy-b/DATA/LOCALE/locales
S 2010/04/25 17:24:19 STARTUP Registering Shutdown Hook
S 2010/04/25 17:24:19 BusyThread thread 'de.anomic.crawler.CrawlQueues.coreCrawlJob' delayed, starting now loop.
S 2010/04/25 17:24:19 BusyThread thread 'de.anomic.search.Switchboard.surrogateProcess' delayed, starting now loop.
I 2010/04/25 17:24:21 YACY yacyClient.publishMySeed thread 'PublishSeed_yog-sothoth' contacted peer at 81.169.168.60:8080, received 11197 bytes, time = 406 milliseconds
I 2010/04/25 17:24:21 YACY yacyClient.publishMySeed: Peer 'yog-sothoth' reported us as junior.
I 2010/04/25 17:24:21 YACY yacyClient.publishMySeed thread 'PublishSeed_yacystats-de-01' contacted peer at 62.75.214.113:8080, received 10562 bytes, time = 414 milliseconds
I 2010/04/25 17:24:21 YACY yacyClient.publishMySeed: Peer 'yacystats-de-01' reported us as junior.
I 2010/04/25 17:24:21 YACY yacyClient.publishMySeed thread 'PublishSeed_FireFlyYacy' contacted peer at 78.47.38.188:8080, received 11327 bytes, time = 456 milliseconds
I 2010/04/25 17:24:21 YACY yacyClient.publishMySeed: Peer 'FireFlyYacy' reported us as junior.
I 2010/04/25 17:24:21 YACY yacyClient.publishMySeed thread 'PublishSeed_matt23' contacted peer at 79.222.123.109:8080, received 10345 bytes, time = 451 milliseconds
I 2010/04/25 17:24:21 YACY yacyClient.publishMySeed: Peer 'matt23' reported us as junior.
I 2010/04/25 17:24:22 YACY publish: handshaked senior peer 'yog-sothoth' at 81.169.168.60:8080
I 2010/04/25 17:24:22 YACY publish: handshaked senior peer 'FireFlyYacy' at 78.47.38.188:8080
I 2010/04/25 17:24:22 YACY publish: handshaked senior peer 'matt23' at 79.222.123.109:8080
I 2010/04/25 17:24:22 YACY publish: handshaked principal peer 'yacystats-de-01' at 62.75.214.113:8080
I 2010/04/25 17:24:23 YACY yacyClient.publishMySeed thread 'PublishSeed_Hermes' contacted peer at 84.62.109.52:8088, received 10612 bytes, time = 1745 milliseconds
I 2010/04/25 17:24:23 YACY yacyClient.publishMySeed: Peer 'Hermes' reported us as junior.
I 2010/04/25 17:24:23 YACY publish: handshaked principal peer 'Hermes' at 84.62.109.52:8088
I 2010/04/25 17:24:26 PLASMA dhtTransferJob: selected EAELmVQ3IE4k as start hash
I 2010/04/25 17:24:26 PLASMA dhtTransferJob: selected E2E_-NKZgOQQ as limit hash
I 2010/04/25 17:24:26 YACY yacyClient.publishMySeed thread 'PublishSeed_KIT01-01F-GEOCACHING' contacted peer at 141.52.175.31:8080, received 11322 bytes, time = 5199 milliseconds
I 2010/04/25 17:24:26 YACY yacyClient.publishMySeed: Peer 'KIT01-01F-GEOCACHING' reported us as junior.
I 2010/04/25 17:24:26 YACY publish: handshaked senior peer 'KIT01-01F-GEOCACHING' at 141.52.175.31:8080
I 2010/04/25 17:24:27 YACY yacyClient.publishMySeed thread 'PublishSeed_abmp' contacted peer at 96.39.7.206:8080, received 11049 bytes, time = 5657 milliseconds
I 2010/04/25 17:24:27 YACY yacyClient.publishMySeed: Peer 'abmp' reported us as junior.
I 2010/04/25 17:24:27 YACY publish: handshaked senior peer 'abmp' at 96.39.7.206:8080
I 2010/04/25 17:24:28 RICELL-shrink1 unmountBestMatch(2.0, 268435456)
I 2010/04/25 17:24:28 IODispatcher appended merge job of files text.index.20100425050520086.blob, text.index.20100425052522277.blob to text.index.20100425152428629.blob
I 2010/04/25 17:24:28 RICELL-shrink1 unmountBestMatch(2.0, 268435456)
I 2010/04/25 17:24:28 BLOBArray merging text.index.20100425050520086.blob with text.index.20100425052522277.blob
I 2010/04/25 17:24:28 IODispatcher appended merge job of files text.index.20100425045519225.blob, text.index.20100425053524974.blob to text.index.20100425152428677.blob
I 2010/04/25 17:24:28 RICELL-shrink1 unmountBestMatch(2.0, 268435456)
I 2010/04/25 17:24:28 YACY yacyClient.publishMySeed thread 'PublishSeed_atoemchen' contacted peer at 84.73.145.212:8080, received 12796 bytes, time = 7471 milliseconds
I 2010/04/25 17:24:28 YACY yacyClient.publishMySeed: Peer 'atoemchen' reported us as junior.
I 2010/04/25 17:24:28 YACY publish: handshaked senior peer 'atoemchen' at 84.73.145.212:8080
I 2010/04/25 17:24:30 kelondroBLOBHeapWriter wrote a dump for the 22450 index entries of text.index.20100425152428629.blob in 40 milliseconds.
I 2010/04/25 17:24:30 HeapReader saturation of text.index.20100425152428629.blob.KKnG6oVZi3tn.idx: keylength = 5, vallength = 3, possible saving: 0 MB
I 2010/04/25 17:24:30 HeapReader using a dump of the index of /home/yacy/yacy-b/DATA/INDEX/freeworld/SEGMENTS/default/text.index.20100425152428629.blob.
I 2010/04/25 17:24:30 BLOBArray merged text.index.20100425050520086.blob with text.index.20100425052522277.blob into /home/yacy/yacy-b/DATA/INDEX/freeworld/SEGMENTS/default/text.index.20100425152428629.blob
I 2010/04/25 17:24:30 BLOBArray merging text.index.20100425045519225.blob with text.index.20100425053524974.blob
I 2010/04/25 17:24:30 IODispatcher appended merge job of files text.index.20100424214557922.blob, text.index.20100425044518158.blob to text.index.20100425152428694.blob
I 2010/04/25 17:24:30 RICELL-shrink1 unmountBestMatch(2.0, 268435456)
I 2010/04/25 17:24:30 YACY yacyClient.publishMySeed thread 'PublishSeed_anilam_test' contacted peer at 85.177.170.217:8080, received 11527 bytes, time = 9188 milliseconds
I 2010/04/25 17:24:30 YACY yacyClient.publishMySeed: Peer 'anilam_test' reported us as junior.
I 2010/04/25 17:24:30 YACY publish: handshaked principal peer 'anilam_test' at 85.177.170.217:8080
I 2010/04/25 17:24:30 YACY PeerPing: I am accessible for 0 peer(s), not accessible for 9 peer(s).
I 2010/04/25 17:24:30 YACY PeerPing: myType is junior
I 2010/04/25 17:24:31 kelondroBLOBHeapWriter wrote a dump for the 19184 index entries of text.index.20100425152428677.blob in 71 milliseconds.
I 2010/04/25 17:24:31 HeapReader saturation of text.index.20100425152428677.blob.WaDETjOvwqCh.idx: keylength = 5, vallength = 3, possible saving: 0 MB
I 2010/04/25 17:24:31 HeapReader using a dump of the index of /home/yacy/yacy-b/DATA/INDEX/freeworld/SEGMENTS/default/text.index.20100425152428677.blob.
I 2010/04/25 17:24:31 BLOBArray merged text.index.20100425045519225.blob with text.index.20100425053524974.blob into /home/yacy/yacy-b/DATA/INDEX/freeworld/SEGMENTS/default/text.index.20100425152428677.blob
I 2010/04/25 17:24:31 BLOBArray merging text.index.20100424214557922.blob with text.index.20100425044518158.blob
I 2010/04/25 17:24:31 IODispatcher appended merge job of files text.index.20100425051520250.blob, text.index.20100425152428629.blob to text.index.20100425152430306.blob
I 2010/04/25 17:24:31 RICELL-shrink2 unmountSmallest(268435456)
I 2010/04/25 17:24:31 kelondroBLOBHeapWriter wrote a dump for the 10053 index entries of text.index.20100425152428694.blob in 13 milliseconds.
I 2010/04/25 17:24:31 HeapReader saturation of text.index.20100425152428694.blob.Rz_-4BGwC5aY.idx: keylength = 5, vallength = 3, possible saving: 0 MB
I 2010/04/25 17:24:31 HeapReader using a dump of the index of /home/yacy/yacy-b/DATA/INDEX/freeworld/SEGMENTS/default/text.index.20100425152428694.blob.
I 2010/04/25 17:24:31 BLOBArray merged text.index.20100424214557922.blob with text.index.20100425044518158.blob into /home/yacy/yacy-b/DATA/INDEX/freeworld/SEGMENTS/default/text.index.20100425152428694.blob
I 2010/04/25 17:24:31 BLOBArray merging text.index.20100425051520250.blob with text.index.20100425152428629.blob
I 2010/04/25 17:24:31 IODispatcher appended merge job of files text.index.20100425043517072.blob, text.index.20100425141033947.blob to text.index.20100425152431272.blob
I 2010/04/25 17:24:31 RICELL-shrink2 unmountSmallest(268435456)
I 2010/04/25 17:24:32 PROXY Configuring proxy access logging ...
I 2010/04/25 17:24:32 PROXY Proxy access logging configuration done.
   Filename: DATA/LOG/proxyAccess%u%g.log
   Limit: 1048576
   Count: 20
S 2010/04/25 17:24:32 HTTPDFiles Loading mime mapping file defaults/httpd.mime
I 2010/04/25 17:24:32 kelondroBLOBHeapWriter wrote a dump for the 30589 index entries of text.index.20100425152430306.blob in 57 milliseconds.
I 2010/04/25 17:24:32 HeapReader saturation of text.index.20100425152430306.blob.eL-bcs9UuPTt.idx: keylength = 5, vallength = 3, possible saving: 0 MB
I 2010/04/25 17:24:32 HeapReader using a dump of the index of /home/yacy/yacy-b/DATA/INDEX/freeworld/SEGMENTS/default/text.index.20100425152430306.blob.
I 2010/04/25 17:24:32 BLOBArray merged text.index.20100425051520250.blob with text.index.20100425152428629.blob into /home/yacy/yacy-b/DATA/INDEX/freeworld/SEGMENTS/default/text.index.20100425152430306.blob
I 2010/04/25 17:24:32 BLOBArray merging text.index.20100425043517072.blob with text.index.20100425141033947.blob
I 2010/04/25 17:24:32 IODispatcher appended merge job of files text.index.20100425152428694.blob, text.index.20100425152428677.blob to text.index.20100425152431539.blob
I 2010/04/25 17:24:32 RICELL-shrink2 unmountSmallest(268435456)
I 2010/04/25 17:24:33 kelondroBLOBHeapWriter wrote a dump for the 3345 index entries of text.index.20100425152431272.blob in 8 milliseconds.
I 2010/04/25 17:24:33 HeapReader saturation of text.index.20100425152431272.blob.i3NnwEnELenj.idx: keylength = 5, vallength = 3, possible saving: 0 MB
I 2010/04/25 17:24:33 HeapReader using a dump of the index of /home/yacy/yacy-b/DATA/INDEX/freeworld/SEGMENTS/default/text.index.20100425152431272.blob.
I 2010/04/25 17:24:33 BLOBArray merged text.index.20100425043517072.blob with text.index.20100425141033947.blob into /home/yacy/yacy-b/DATA/INDEX/freeworld/SEGMENTS/default/text.index.20100425152431272.blob
I 2010/04/25 17:24:33 IODispatcher appended merge job of files text.index.20100425152430306.blob, text.index.20100424211300762.blob to text.index.20100425152432929.blob
I 2010/04/25 17:24:33 BLOBArray merging text.index.20100425152428694.blob with text.index.20100425152428677.blob
I 2010/04/25 17:24:35 kelondroBLOBHeapWriter wrote a dump for the 24784 index entries of text.index.20100425152431539.blob in 15 milliseconds.
S 2010/04/25 17:24:35 BusyThread thread 'de.anomic.crawler.CrawlQueues.remoteCrawlLoaderJob' delayed, starting now loop.
S 2010/04/25 17:24:35 BusyThread thread 'de.anomic.crawler.CrawlQueues.remoteTriggeredCrawlJob' delayed, starting now loop.
I 2010/04/25 17:24:36 BALANCER re-fill of domain stacks; fileIndex.size() = 98, domainStacks.size = 16, collection time = 151 ms
I 2010/04/25 17:24:41 HeapReader saturation of text.index.20100425152431539.blob._yxEfK3qD-hI.idx: keylength = 5, vallength = 3, possible saving: 0 MB
I 2010/04/25 17:24:41 HeapReader using a dump of the index of /home/yacy/yacy-b/DATA/INDEX/freeworld/SEGMENTS/default/text.index.20100425152431539.blob.
I 2010/04/25 17:24:41 BLOBArray merged text.index.20100425152428694.blob with text.index.20100425152428677.blob into /home/yacy/yacy-b/DATA/INDEX/freeworld/SEGMENTS/default/text.index.20100425152431539.blob
I 2010/04/25 17:24:41 BLOBArray merging text.index.20100425152430306.blob with text.index.20100424211300762.blob
W 2010/04/25 17:24:41 ReferenceContainerArray timout in index retrieval (2): 4 tables searched. timeout = 3000
I 2010/04/25 17:25:43 RICELL-shrink2 unmountSmallest(268435456)
I 2010/04/25 17:25:43 IODispatcher appended merge job of files text.index.20100425152431272.blob, text.index.20100425152431539.blob to text.index.20100425152543799.blob
W 2010/04/25 17:25:58 ReferenceContainerArray timout in index retrieval (1): 1 tables searched. timeout = 3000
S 2010/04/25 17:26:07 BusyThread thread 'de.anomic.data.bookmarksDB.autoReCrawl' delayed, starting now loop.
I 2010/04/25 17:26:07 BOOKMARKS autoReCrawl - reading schedules from /home/yacy/yacy-b/DATA/SETTINGS/autoReCrawl.conf
I 2010/04/25 17:26:07 BOOKMARKS autoReCrawl - processing: /autoReCrawl/hourly
I 2010/04/25 17:26:07 BOOKMARKS autoReCrawl - processing: /autoReCrawl/daily
I 2010/04/25 17:26:07 BOOKMARKS autoReCrawl - checking schedule for: [2009-11-05T12:23:23Z] http://www.ettlingen.de/
I 2010/04/25 17:26:07 BOOKMARKS autoReCrawl - processing: /autoReCrawl/weekly
I 2010/04/25 17:26:07 BOOKMARKS autoReCrawl - processing: /autoReCrawl/monthly
I 2010/04/25 17:27:09 INDEX-TRANSFER-DISPATCHER selectContainersEnqueueToCloud: selectedContainerCache was filled with 32 entries
I 2010/04/25 17:27:09 INDEX-TRANSFER-DISPATCHER splitContainersFromCache: splittedContainerCache filled with 16 partitions, deleting selectedContainerCache
[
Dateianhänge
yacy08-001k.png
yacy08-001k.png (13.48 KiB) 565-mal betrachtet
dulcedo
 
Beiträge: 1006
Registriert: Do Okt 16, 2008 6:36 pm
Wohnort: Bei Karlsruhe

Re: 6836 OOMs bei (such)zugriffen

Beitragvon Quix0r » So Apr 25, 2010 5:31 pm

Ja, das Mergen, es kostet wirklich mehr Speicher, da ich mal schaetze, dass viel in das byte[]-Array geladen werden muss.
Quix0r
 
Beiträge: 1347
Registriert: Di Jul 31, 2007 9:22 am
Wohnort: Krefeld

Re: 6836 OOMs bei (such)zugriffen

Beitragvon Orbiter » So Apr 25, 2010 11:03 pm

dulcedo hat geschrieben:Die Grafik ist dann leer deshalb habe ich sie auch nicht gezeigt und mehr Speicher macht die Maschine viel zu langsam.
Wenn ich den hier 1GB grossen response-header lösche, oder zu 6824 zurückgehe hat er wieder etwas Luft. Kann ich dessen Grösse begrenzen oder läuft das auch unter maxFilesize?

Ich sehe jetzt nichts was zwischen 6824 und 6828 mehr Speicher benötigen würde. Meine Commits in den letzten 2 Wochen hatten vielmehr zum Ziel weniger Speicher zu belegen, bzw. weniger neue Objekte, die wieder gelöscht werden danach zu erzeugen.
Orbiter
 
Beiträge: 5798
Registriert: Di Jun 26, 2007 10:58 pm
Wohnort: Frankfurt am Main

Re: 6836 OOMs bei (such)zugriffen

Beitragvon dulcedo » Di Apr 27, 2010 7:11 am

Das Problem liegt daran dass im Betrieb manchmal der Anschein erweckt wird der peer käme mit einer bestimmten Menge Speicher aus, in manchen, seltenen Betriebszuständen braucht er aber zwingend mehr.
Meiner Meinung nach steht die jetzt erreichte Stabilität im Vordergrund, man sollte Speichermangel nur deutlicher Kennzeichnen oder abfangen, ein Normalanwender hat grosse Schwierigkeiten die jetzigen Symptome zu deuten. Man könnte im einfachsten Fall Anweisungen geben wie: bei längeren Crawls im DHT-Betrieb regelmässige Neustarts, oder dies auch automatisch auslösen lassen.
dulcedo
 
Beiträge: 1006
Registriert: Do Okt 16, 2008 6:36 pm
Wohnort: Bei Karlsruhe

Re: 6836 OOMs bei (such)zugriffen

Beitragvon Orbiter » Mi Apr 28, 2010 4:13 pm

es gab ja in der letzten RWI Datenstruktur eine Größenbeschränkung, die es nun nicht mehr gibt. Das scheint nun 'zurückzuschlagen'. Bei knapp über 50 millionen Referenzen gibts sogaer einen Integer-Overflow, der verhindert dass man RWI-byte[] allocieren kann. Da muss irgendein Konzept zur Löschung her! Was anderes fällt mir nicht ein.
Orbiter
 
Beiträge: 5798
Registriert: Di Jun 26, 2007 10:58 pm
Wohnort: Frankfurt am Main


Zurück zu Fragen und Antworten

Wer ist online?

Mitglieder in diesem Forum: 0 Mitglieder und 2 Gäste

cron