SVN 6033: URL-Stack-Threads blocked

Hier finden YaCy User Hilfe wenn was nicht funktioniert oder anders funktioniert als man dachte. Bei offensichtlichen Fehlern diese bitte gleich in die Bugs (http://bugs.yacy.net) eintragen.
Forumsregeln
In diesem Forum geht es um Benutzungsprobleme und Anfragen für Hilfe. Wird dabei ein Bug identifiziert, wird der thread zur Bearbeitung in die Bug-Sektion verschoben. Wer hier also einen Thread eingestellt hat und ihn vermisst, wird ihn sicherlich in der Bug-Sektion wiederfinden.

SVN 6033: URL-Stack-Threads blocked

Beitragvon PCA42 » So Jun 07, 2009 5:25 pm

Der überarbeitete Crawl läuft sehr performant. Bis er größer wird. Dann scheinen die Threads sich öfter blocken und die Geschwindigkeit sinkt stark ab. Vielleicht läßt sich da noch etwas verfeinern.

Beispiel für 1000 Durchläufe:
Code: Alles auswählen
THREADS WITH STATES: BLOCKED

Occurrences: 1287
at de.anomic.kelondro.table.EcoTable.put(EcoTable.java:420) [assert file == null || file.size() == index.size() + fail : "file.size() = " + file.size() + ", index.size() = " + index.size();]
at de.anomic.crawler.ZURL$Entry.store(ZURL.java:212)
at de.anomic.crawler.CrawlStacker.job(CrawlStacker.java:138)
at sun.reflect.GeneratedMethodAccessor20.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at de.anomic.server.serverInstantBlockingThread.job(serverInstantBlockingThread.java:87)
at de.anomic.server.serverAbstractBlockingThread.run(serverAbstractBlockingThread.java:64)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)

Occurrences: 757
at de.anomic.crawler.ZURL.newEntry(ZURL.java:104) [assert executor != null;]
at de.anomic.crawler.CrawlStacker.job(CrawlStacker.java:137)
at sun.reflect.GeneratedMethodAccessor20.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at de.anomic.server.serverInstantBlockingThread.job(serverInstantBlockingThread.java:87)
at de.anomic.server.serverAbstractBlockingThread.run(serverAbstractBlockingThread.java:64)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)

Occurrences: 647
at java.io.RandomAccessFile.write(RandomAccessFile.java:466)
at de.anomic.kelondro.io.EcoFS.put(EcoFS.java:325)
at de.anomic.kelondro.io.BufferedEcoFS.flushBuffer(BufferedEcoFS.java:58)
at de.anomic.kelondro.io.BufferedEcoFS.put(BufferedEcoFS.java:106)
at de.anomic.kelondro.table.EcoTable.put(EcoTable.java:433)
at de.anomic.crawler.ZURL$Entry.store(ZURL.java:212)
at de.anomic.crawler.CrawlStacker.job(CrawlStacker.java:138)
at sun.reflect.GeneratedMethodAccessor20.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at de.anomic.server.serverInstantBlockingThread.job(serverInstantBlockingThread.java:87)
at de.anomic.server.serverAbstractBlockingThread.run(serverAbstractBlockingThread.java:64)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)

Occurrences: 474
at de.anomic.crawler.Balancer.push(Balancer.java:224) [synchronized (this) {]
at de.anomic.crawler.NoticedURL.push(NoticedURL.java:142)
at de.anomic.crawler.CrawlStacker.stackCrawl(CrawlStacker.java:301)
at de.anomic.crawler.CrawlStacker.job(CrawlStacker.java:133)
at sun.reflect.GeneratedMethodAccessor20.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at de.anomic.server.serverInstantBlockingThread.job(serverInstantBlockingThread.java:87)
at de.anomic.server.serverAbstractBlockingThread.run(serverAbstractBlockingThread.java:64)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)

Occurrences: 441
at de.anomic.kelondro.table.EcoTable.get(EcoTable.java:347) [if ((file == null) || (index == null)) return null;]
at de.anomic.crawler.ZURL.getEntry(ZURL.java:140)
at de.anomic.crawler.CrawlQueues.getURL(CrawlQueues.java:114)
at de.anomic.crawler.CrawlStacker.stackCrawl(CrawlStacker.java:234)
at de.anomic.crawler.CrawlStacker.job(CrawlStacker.java:133)
at sun.reflect.GeneratedMethodAccessor20.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at de.anomic.server.serverInstantBlockingThread.job(serverInstantBlockingThread.java:87)
at de.anomic.server.serverAbstractBlockingThread.run(serverAbstractBlockingThread.java:64)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)

Occurrences: 312
at de.anomic.kelondro.text.MetadataRepository.load(MetadataRepository.java:105) [if (urlHash == null) return null;]
at de.anomic.crawler.CrawlStacker.stackCrawl(CrawlStacker.java:258)
at de.anomic.crawler.CrawlStacker.job(CrawlStacker.java:133)
at sun.reflect.GeneratedMethodAccessor20.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at de.anomic.server.serverInstantBlockingThread.job(serverInstantBlockingThread.java:87)
at de.anomic.server.serverAbstractBlockingThread.run(serverAbstractBlockingThread.java:64)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)

Occurrences: 208
at de.anomic.kelondro.table.EcoTable.get(EcoTable.java:347) [if ((file == null) || (index == null)) return null;]
at de.anomic.crawler.Balancer.get(Balancer.java:117)
at de.anomic.crawler.NoticedURL.get(NoticedURL.java:155)
at de.anomic.crawler.CrawlQueues.getURL(CrawlQueues.java:110)
at de.anomic.crawler.CrawlStacker.stackCrawl(CrawlStacker.java:234)
at de.anomic.crawler.CrawlStacker.job(CrawlStacker.java:133)
at sun.reflect.GeneratedMethodAccessor20.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at de.anomic.server.serverInstantBlockingThread.job(serverInstantBlockingThread.java:87)
at de.anomic.server.serverAbstractBlockingThread.run(serverAbstractBlockingThread.java:64)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)


Da läßt sich doch sicher noch etwas machen oder? ;)
PCA42
 
Beiträge: 621
Registriert: Mi Jan 23, 2008 4:19 pm
Wohnort: @Home

Re: SVN 6033: URL-Stack-Threads blocked

Beitragvon Quix0r » Di Jun 16, 2009 2:32 pm

Genau die habe ich jetzt mit Revison von gestern Abend auch. Und Max-PPM ist auf 80 festgesetzt damit das Admin-Interface samt Suche und Proxy noch zugänglich bleibt. :( Ich würde gerne mit 200-300 PPM crawlen und gleichzeitig den Proxy usw. verfügbar haben. Das nur so am Rande. :)
Quix0r
 
Beiträge: 1345
Registriert: Di Jul 31, 2007 9:22 am
Wohnort: Krefeld


Zurück zu Fragen und Antworten

Wer ist online?

Mitglieder in diesem Forum: 0 Mitglieder und 4 Gäste

cron