[closed]Cleanup

Hier finden YaCy User Hilfe wenn was nicht funktioniert oder anders funktioniert als man dachte. Bei offensichtlichen Fehlern diese bitte gleich in die Bugs (http://bugs.yacy.net) eintragen.
Forumsregeln
In diesem Forum geht es um Benutzungsprobleme und Anfragen für Hilfe. Wird dabei ein Bug identifiziert, wird der thread zur Bearbeitung in die Bug-Sektion verschoben. Wer hier also einen Thread eingestellt hat und ihn vermisst, wird ihn sicherlich in der Bug-Sektion wiederfinden.

[closed]Cleanup

Beitragvon dulcedo » Fr Sep 25, 2009 5:40 am

Was macht dieser Prozess? Auf anderen Peers braucht er so gut wie keine res, der peer crawlt auch 30% langsamer.
SVN6343, DebianLenny/ext4, Java 1.6

PerformanceQueues_p.html
Cleanup 1 0 sec 0% 1.800 sec 8% 391 sec 27% 7 0 7 0 257.142 55.974 227.732


Zusammenhang mit diesem sekündlichem error beim crawlen?
Code: Alles auswählen
E 2009/09/25 06:38:45 BLOCKINGTHREAD Runtime Error in serverInstantThread.job, thread 'java.lang.reflect.Method.webStructureAnalysis.19': null; target exception: domain = linksfraktion.de, key = dY7eTA,doener.blogage.de
java.lang.AssertionError: domain = linksfraktion.de, key = dY7eTA,doener.blogage.de
at de.anomic.ymage.WebStructureGraph.outgoingReferences(WebStructureGraph.java:254)
at de.anomic.ymage.WebStructureGraph.learn(WebStructureGraph.java:359)
at de.anomic.ymage.WebStructureGraph.generateCitationReference(WebStructureGraph.java:139)
at de.anomic.document.Document.notifyWebStructure(Document.java:552)
at de.anomic.search.Switchboard.webStructureAnalysis(Switchboard.java:1672)
at sun.reflect.GeneratedMethodAccessor9.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at de.anomic.server.serverInstantBlockingThread.job(serverInstantBlockingThread.java:87)
at de.anomic.server.serverAbstractBlockingThread.run(serverAbstractBlockingThread.java:64)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)



Code: Alles auswählen
YaCy Version: 0.91/6343
Assigned   Memory = 6687424512
Used       Memory = 5928044608
Available  Memory = 759379904


THREADS WITH STATES: BLOCKED

Thread= Thread-45621 id=82533 BLOCKED
at de.anomic.kelondro.blob.MapView.put(MapView.java:146) [String s = map2string(newMap, "W" + DateFormatter.formatShortSecond() + " ");]
at de.anomic.http.client.Cache.store(Cache.java:120)
at de.anomic.crawler.retrieval.LoaderDispatcher.load(LoaderDispatcher.java:219)
at de.anomic.crawler.retrieval.LoaderDispatcher.load(LoaderDispatcher.java:128)
at de.anomic.crawler.CrawlQueues$crawlWorker.run(CrawlQueues.java:565)


Thread= Thread-45629 id=82546 BLOCKED
at java.lang.Thread.join(Thread.java:1151)
at org.apache.commons.httpclient.util.TimeoutController.execute(TimeoutController.java:63)
at org.apache.commons.httpclient.util.TimeoutController.execute(TimeoutController.java:82)
at org.apache.commons.httpclient.protocol.ControllerThreadSocketFactory.createSocket(ControllerThreadSocketFactory.java:95)
at org.apache.commons.httpclient.protocol.DefaultProtocolSocketFactory.createSocket(DefaultProtocolSocketFactory.java:128)
at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707)
at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361)
at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387)
at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323)
at de.anomic.http.client.Client.execute(Client.java:457)
at de.anomic.http.client.Client.GET(Client.java:276)
at de.anomic.crawler.retrieval.HTTPLoader.load(HTTPLoader.java:133)
at de.anomic.crawler.retrieval.HTTPLoader.load(HTTPLoader.java:78)
at de.anomic.crawler.retrieval.LoaderDispatcher.load(LoaderDispatcher.java:213)
at de.anomic.crawler.retrieval.LoaderDispatcher.load(LoaderDispatcher.java:128)
at de.anomic.crawler.CrawlQueues$crawlWorker.run(CrawlQueues.java:565)


Thread= Thread-45609 id=82511 BLOCKED
Thread= Thread-45591 id=82482 BLOCKED
Thread= Thread-45619 id=82529 BLOCKED
Thread= Thread-45622 id=82535 BLOCKED
at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
at org.apache.commons.httpclient.HttpParser.readRawLine(HttpParser.java:78)
at org.apache.commons.httpclient.HttpParser.readLine(HttpParser.java:106)
at org.apache.commons.httpclient.HttpConnection.readLine(HttpConnection.java:1116)
at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.readLine(MultiThreadedHttpConnectionManager.java:1413)
at org.apache.commons.httpclient.HttpMethodBase.readStatusLine(HttpMethodBase.java:1973)
at org.apache.commons.httpclient.HttpMethodBase.readResponse(HttpMethodBase.java:1735)
at org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:1098)
at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:398)
at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323)
at de.anomic.http.client.Client.execute(Client.java:457)
at de.anomic.http.client.Client.GET(Client.java:276)
at de.anomic.crawler.retrieval.HTTPLoader.load(HTTPLoader.java:133)
at de.anomic.crawler.retrieval.HTTPLoader.load(HTTPLoader.java:78)
at de.anomic.crawler.retrieval.LoaderDispatcher.load(LoaderDispatcher.java:213)
at de.anomic.crawler.retrieval.LoaderDispatcher.load(LoaderDispatcher.java:128)
at de.anomic.crawler.CrawlQueues$crawlWorker.run(CrawlQueues.java:565)



THREADS WITH STATES: RUNNABLE

dulcedo
 
Beiträge: 1006
Registriert: Do Okt 16, 2008 6:36 pm
Wohnort: Bei Karlsruhe

Re: Cleanup

Beitragvon Orbiter » Di Sep 29, 2009 9:43 am

der thread dump zeigt 2 verschiedene Blockaden:
- während http-get client threads
- während eines http-cache storage Vorgang

Den ersten Fall muss man so akzeptieren .. http-Zugriffe blockieren nun mal.
Den 2. Fall habe ich genauer angeguckt, da gab es zumindest eine Möglichkeit, die Berechnung an der Position des dumps aus dem Synchronized-Block heraus zu nehmen, was verhindern wird dass es dort wieder blockt.
SVN 6358 sollte daher hier Linderung bringen.
Orbiter
 
Beiträge: 5792
Registriert: Di Jun 26, 2007 10:58 pm
Wohnort: Frankfurt am Main

Re: Cleanup

Beitragvon dulcedo » Di Okt 06, 2009 4:24 pm

Sieht in 6375 jetzt besser aus:
Cleanup 0 0 sec 0% 0 sec 0% 67 sec 5% 1 0 1 0 0 67.865 379.074
dulcedo
 
Beiträge: 1006
Registriert: Do Okt 16, 2008 6:36 pm
Wohnort: Bei Karlsruhe


Zurück zu Fragen und Antworten

Wer ist online?

Mitglieder in diesem Forum: 0 Mitglieder und 1 Gast

cron