[closed] massiv paralleles indexieren?

Hier finden YaCy User Hilfe wenn was nicht funktioniert oder anders funktioniert als man dachte. Bei offensichtlichen Fehlern diese bitte gleich in die Bugs (http://bugs.yacy.net) eintragen.
Forumsregeln
In diesem Forum geht es um Benutzungsprobleme und Anfragen für Hilfe. Wird dabei ein Bug identifiziert, wird der thread zur Bearbeitung in die Bug-Sektion verschoben. Wer hier also einen Thread eingestellt hat und ihn vermisst, wird ihn sicherlich in der Bug-Sektion wiederfinden.

[closed] massiv paralleles indexieren?

Beitragvon miTreD » Di Apr 08, 2008 2:12 pm

Rote Einträge in der Indexing Queue bedeuten doch, das diese gerade abgearbeitet werden, oder? Bis jetzt war auch immer nur einer rot, was ja auch passen würde, bei einer CPU. Gerade sind es allerdings 16 Einträge ... Evtl. zeigt der Threaddump eine etwas niedrigere Anzahl, da ich mit der Erstellung kurz gewartet habe.

0.578/04665
Code: Alles auswählen
************* Start Thread Dump Tue Apr 08 15:13:57 CEST 2008 *******************

YaCy Version: 0.578/04665
Total Memory = 665976832
Used  Memory = 543213568
Free  Memory = 122763264
--- --- --- ---

THREADS WITH STATES: BLOCKED

Thread= pool-2-thread-1 id=15 BLOCKED
at de.anomic.index.indexCollectionRI.addEntries(indexCollectionRI.java:172) [ collectionIndex.merge(newEntries);]
at de.anomic.plasma.plasmaWordIndex.flushCache(plasmaWordIndex.java:309)
at de.anomic.plasma.plasmaWordIndex.dhtFlushControl(plasmaWordIndex.java:220)
at de.anomic.plasma.plasmaWordIndex.addEntry(plasmaWordIndex.java:244)
at de.anomic.plasma.plasmaWordIndex.addPageIndex(plasmaWordIndex.java:379)
at de.anomic.plasma.plasmaWordIndex.storeDocument(plasmaWordIndex.java:643)
at de.anomic.plasma.plasmaSwitchboard.storeDocumentIndex(plasmaSwitchboard.java:2290)
at de.anomic.plasma.plasmaSwitchboard.storeDocumentIndex(plasmaSwitchboard.java:2268)
at sun.reflect.GeneratedMethodAccessor22.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at de.anomic.server.serverInstantBlockingThread.job(serverInstantBlockingThread.java:85)
at de.anomic.server.serverAbstractBlockingThread.run(serverAbstractBlockingThread.java:71)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)


THREADS WITH STATES: RUNNABLE

Thread= Session_84.180.237.50:33519#0 id=1760 RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at java.net.SocketInputStream.read(SocketInputStream.java:182)
at java.io.FilterInputStream.read(FilterInputStream.java:66)
at java.io.PushbackInputStream.read(PushbackInputStream.java:122)
at de.anomic.server.serverCore.receive(serverCore.java:931)
at de.anomic.server.serverCore$Session.readLine(serverCore.java:655)
at de.anomic.server.serverCore$Session.listen(serverCore.java:760)
at de.anomic.server.serverCore$Session.run(serverCore.java:709)

Thread= pool-2-thread-2 id=17 RUNNABLE
at de.anomic.kelondro.kelondroCollectionIndex.merge(kelondroCollectionIndex.java:497) [ if ((container == null) || (container.size() == 0)) return;]
at de.anomic.index.indexCollectionRI.addEntries(indexCollectionRI.java:172)
at de.anomic.plasma.plasmaWordIndex.flushCache(plasmaWordIndex.java:309)
at de.anomic.plasma.plasmaWordIndex.dhtFlushControl(plasmaWordIndex.java:220)
at de.anomic.plasma.plasmaWordIndex.addEntry(plasmaWordIndex.java:244)
at de.anomic.plasma.plasmaWordIndex.addPageIndex(plasmaWordIndex.java:379)
at de.anomic.plasma.plasmaWordIndex.storeDocument(plasmaWordIndex.java:643)
at de.anomic.plasma.plasmaSwitchboard.storeDocumentIndex(plasmaSwitchboard.java:2290)
at de.anomic.plasma.plasmaSwitchboard.storeDocumentIndex(plasmaSwitchboard.java:2268)
at sun.reflect.GeneratedMethodAccessor22.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at de.anomic.server.serverInstantBlockingThread.job(serverInstantBlockingThread.java:85)
at de.anomic.server.serverAbstractBlockingThread.run(serverAbstractBlockingThread.java:71)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)

Thread= Session_91.17.197.77:61842#3 id=1530 RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at java.net.SocketInputStream.read(SocketInputStream.java:182)
at java.io.FilterInputStream.read(FilterInputStream.java:66)
at java.io.PushbackInputStream.read(PushbackInputStream.java:122)
at de.anomic.server.serverCore.receive(serverCore.java:931)
at de.anomic.server.serverCore$Session.readLine(serverCore.java:655)
at de.anomic.server.serverCore$Session.listen(serverCore.java:760)
at de.anomic.server.serverCore$Session.run(serverCore.java:709)

Thread= Session_91.63.117.93:3598#0 id=695 RUNNABLE
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
at de.anomic.server.serverFileUtils.copy(serverFileUtils.java:240)
at de.anomic.http.httpdFileHandler.doResponse(httpdFileHandler.java:777)
at de.anomic.http.httpdFileHandler.doPost(httpdFileHandler.java:271)
at de.anomic.http.httpd.POST(httpd.java:613)
at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at de.anomic.server.serverCore$Session.listen(serverCore.java:828)
at de.anomic.server.serverCore$Session.run(serverCore.java:709)

Thread= Session_88.198.20.109:41962#0 id=1841 RUNNABLE
at java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method)
at java.net.InetAddress$1.lookupAllHostAddr(InetAddress.java:849)
at java.net.InetAddress.getAddressFromNameService(InetAddress.java:1200)
at java.net.InetAddress.getAllByName0(InetAddress.java:1153)
at java.net.InetAddress.getAllByName0(InetAddress.java:1128)
at java.net.InetAddress.getHostFromNameService(InetAddress.java:550)
at java.net.InetAddress.getHostName(InetAddress.java:476)
at java.net.InetAddress.getHostName(InetAddress.java:448)
at de.anomic.http.httpd.parseRequestLine(httpd.java:708)
at de.anomic.http.httpd.POST(httpd.java:594)
at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at de.anomic.server.serverCore$Session.listen(serverCore.java:828)
at de.anomic.server.serverCore$Session.run(serverCore.java:709)

Thread= httpd:3963 id=41 RUNNABLE
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
at java.net.ServerSocket.implAccept(ServerSocket.java:453)
at java.net.ServerSocket.accept(ServerSocket.java:421)
at de.anomic.server.serverCore.job(serverCore.java:410)
at de.anomic.server.serverAbstractBusyThread.run(serverAbstractBusyThread.java:142)

Thread= Session_192.168.178.10:53064#0 id=1838 RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at java.net.SocketInputStream.read(SocketInputStream.java:182)
at java.io.FilterInputStream.read(FilterInputStream.java:66)
at java.io.PushbackInputStream.read(PushbackInputStream.java:122)
at de.anomic.server.serverCore.receive(serverCore.java:931)
at de.anomic.server.serverCore$Session.readLine(serverCore.java:655)
at de.anomic.server.serverCore$Session.listen(serverCore.java:760)
at de.anomic.server.serverCore$Session.run(serverCore.java:709)

Thread= Session_192.168.178.10:53065#0 id=1839 RUNNABLE
at java.lang.Thread.dumpThreads(Native Method)
at java.lang.Thread.getAllStackTraces(Thread.java:1477)
at Threaddump_p.respond(Threaddump_p.java:80)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at de.anomic.http.httpdFileHandler.invokeServlet(httpdFileHandler.java:1027)
at de.anomic.http.httpdFileHandler.doResponse(httpdFileHandler.java:629)
at de.anomic.http.httpdFileHandler.doGet(httpdFileHandler.java:263)
at de.anomic.http.httpd.GET(httpd.java:492)
at sun.reflect.GeneratedMethodAccessor13.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at de.anomic.server.serverCore$Session.listen(serverCore.java:828)
at de.anomic.server.serverCore$Session.run(serverCore.java:709)

Thread= PublishSeed_Mailerdemon-1 id=706 RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
at org.apache.commons.httpclient.HttpParser.readRawLine(HttpParser.java:78)
at org.apache.commons.httpclient.HttpParser.readLine(HttpParser.java:106)
at org.apache.commons.httpclient.HttpConnection.readLine(HttpConnection.java:1116)
at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.readLine(MultiThreadedHttpConnectionManager.java:1413)
at org.apache.commons.httpclient.HttpMethodBase.readStatusLine(HttpMethodBase.java:1973)
at org.apache.commons.httpclient.HttpMethodBase.readResponse(HttpMethodBase.java:1735)
at org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:1098)
at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:398)
at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323)
at de.anomic.http.JakartaCommonsHttpClient.execute(JakartaCommonsHttpClient.java:365)
at de.anomic.http.JakartaCommonsHttpClient.POST(JakartaCommonsHttpClient.java:254)
at de.anomic.yacy.yacyClient.wput(yacyClient.java:263)
at de.anomic.yacy.yacyClient.publishMySeed(yacyClient.java:118)
at de.anomic.yacy.yacyCore$publishThread.run(yacyCore.java:307)

Thread= Session_85.177.166.101:46689#0 id=1794 RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
at org.apache.commons.httpclient.HttpParser.readRawLine(HttpParser.java:78)
at org.apache.commons.httpclient.HttpParser.readLine(HttpParser.java:106)
at org.apache.commons.httpclient.HttpConnection.readLine(HttpConnection.java:1116)
at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.readLine(MultiThreadedHttpConnectionManager.java:1413)
at org.apache.commons.httpclient.HttpMethodBase.readStatusLine(HttpMethodBase.java:1973)
at org.apache.commons.httpclient.HttpMethodBase.readResponse(HttpMethodBase.java:1735)
at org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:1098)
at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:398)
at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323)
at de.anomic.http.JakartaCommonsHttpClient.execute(JakartaCommonsHttpClient.java:365)
at de.anomic.http.JakartaCommonsHttpClient.POST(JakartaCommonsHttpClient.java:254)
at de.anomic.yacy.yacyClient.wput(yacyClient.java:263)
at de.anomic.yacy.yacyClient.postToFile(yacyClient.java:283)
at de.anomic.yacy.yacyClient.queryUrlCount(yacyClient.java:341)
at hello.respond(hello.java:154)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at de.anomic.http.httpdFileHandler.invokeServlet(httpdFileHandler.java:1027)
at de.anomic.http.httpdFileHandler.doResponse(httpdFileHandler.java:629)
at de.anomic.http.httpdFileHandler.doPost(httpdFileHandler.java:271)
at de.anomic.http.httpd.POST(httpd.java:613)
at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at de.anomic.server.serverCore$Session.listen(serverCore.java:828)
at de.anomic.server.serverCore$Session.run(serverCore.java:709)

Thread= Signal Dispatcher daemon id=4 RUNNABLE

Thread= de.anomic.plasma.plasmaSwitchboard.cleanupJob id=31 RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
at org.apache.commons.httpclient.HttpParser.readRawLine(HttpParser.java:78)
at org.apache.commons.httpclient.HttpParser.readLine(HttpParser.java:106)
at org.apache.commons.httpclient.HttpConnection.readLine(HttpConnection.java:1116)
at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.readLine(MultiThreadedHttpConnectionManager.java:1413)
at org.apache.commons.httpclient.HttpMethodBase.readStatusLine(HttpMethodBase.java:1973)
at org.apache.commons.httpclient.HttpMethodBase.readResponse(HttpMethodBase.java:1735)
at org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:1098)
at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:398)
at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323)
at de.anomic.http.JakartaCommonsHttpClient.execute(JakartaCommonsHttpClient.java:365)
at de.anomic.http.JakartaCommonsHttpClient.POST(JakartaCommonsHttpClient.java:254)
at de.anomic.yacy.yacyClient.wput(yacyClient.java:263)
at de.anomic.yacy.yacyClient.transferPermission(yacyClient.java:705)
at de.anomic.yacy.yacyClient.transfer(yacyClient.java:742)
at de.anomic.plasma.plasmaRankingDistribution.transferRankingAddress(plasmaRankingDistribution.java:194)
at de.anomic.plasma.plasmaRankingDistribution.transferRankingAnySeed(plasmaRankingDistribution.java:171)
at de.anomic.plasma.plasmaRankingDistribution.transferRanking(plasmaRankingDistribution.java:142)
at de.anomic.plasma.plasmaSwitchboard.cleanupJob(plasmaSwitchboard.java:1928)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at de.anomic.server.serverInstantBusyThread.job(serverInstantBusyThread.java:115)
at de.anomic.server.serverAbstractBusyThread.run(serverAbstractBusyThread.java:142)


THREADS WITH STATES: TIMED_WAITING

Thread= de.anomic.plasma.plasmaSwitchboard.htEntryStoreJob id=34 TIMED_WAITING
at java.lang.Thread.sleep(Native Method)
at de.anomic.server.serverAbstractBusyThread.ratz(serverAbstractBusyThread.java:198)
at de.anomic.server.serverAbstractBusyThread.run(serverAbstractBusyThread.java:163)

Thread= de.anomic.yacy.yacyCore.publishSeedList id=38 TIMED_WAITING
at java.lang.Thread.sleep(Native Method)
at de.anomic.server.serverAbstractBusyThread.ratz(serverAbstractBusyThread.java:198)
at de.anomic.server.serverAbstractBusyThread.run(serverAbstractBusyThread.java:163)

Thread= Timer-0 id=30 TIMED_WAITING
at java.lang.Object.wait(Native Method)
at java.util.TimerThread.mainLoop(Timer.java:509)
at java.util.TimerThread.run(Timer.java:462)

Thread= de.anomic.plasma.crawler.plasmaCrawlQueues.remoteCrawlLoaderJob id=36 TIMED_WAITING
at java.lang.Thread.sleep(Native Method)
at de.anomic.server.serverAbstractBusyThread.ratz(serverAbstractBusyThread.java:198)
at de.anomic.server.serverAbstractBusyThread.run(serverAbstractBusyThread.java:163)

Thread= Session_88.74.245.248:53487#0 id=1840 TIMED_WAITING
at java.io.RandomAccessFile.readBytes(Native Method)
at java.io.RandomAccessFile.read(RandomAccessFile.java:322)
at de.anomic.kelondro.kelondroFileRA.read(kelondroFileRA.java:82)
at de.anomic.kelondro.kelondroBufferedIOChunks.read(kelondroBufferedIOChunks.java:98)
at de.anomic.kelondro.kelondroAbstractIOChunks.readFully(kelondroAbstractIOChunks.java:73)
at de.anomic.kelondro.kelondroFullRecords$EcoNode.<init>(kelondroFullRecords.java:166)
at de.anomic.kelondro.kelondroFixedWidthArray.getIfValid(kelondroFixedWidthArray.java:128)
at de.anomic.kelondro.kelondroFlexWidthArray.get(kelondroFlexWidthArray.java:310)
at de.anomic.kelondro.kelondroFlexTable.put(kelondroFlexTable.java:282)
at de.anomic.kelondro.kelondroCache.put(kelondroCache.java:273)
at de.anomic.kelondro.kelondroSplitTable.put(kelondroSplitTable.java:227)
at de.anomic.index.indexRepositoryReference.store(indexRepositoryReference.java:135)
at de.anomic.plasma.plasmaWordIndex.putURL(plasmaWordIndex.java:116)
at crawlReceipt.respond(crawlReceipt.java:158)
at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at de.anomic.http.httpdFileHandler.invokeServlet(httpdFileHandler.java:1027)
at de.anomic.http.httpdFileHandler.doResponse(httpdFileHandler.java:629)
at de.anomic.http.httpdFileHandler.doPost(httpdFileHandler.java:271)
at de.anomic.http.httpd.POST(httpd.java:613)
at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at de.anomic.server.serverCore$Session.listen(serverCore.java:828)
at de.anomic.server.serverCore$Session.run(serverCore.java:709)

Thread= de.anomic.plasma.crawler.plasmaCrawlQueues.remoteTriggeredCrawlJob id=35 TIMED_WAITING
at java.lang.Thread.sleep(Native Method)
at de.anomic.server.serverAbstractBusyThread.ratz(serverAbstractBusyThread.java:198)
at de.anomic.server.serverAbstractBusyThread.run(serverAbstractBusyThread.java:163)

Thread= Session_192.168.178.10:53063#0 id=1837 TIMED_WAITING
at java.lang.Thread.sleep(Native Method)
at de.anomic.server.serverCore$Session.run(serverCore.java:730)

Thread= de.anomic.plasma.plasmaCrawlStacker.job id=32 TIMED_WAITING
at java.lang.Thread.sleep(Native Method)
at de.anomic.server.serverAbstractBusyThread.ratz(serverAbstractBusyThread.java:198)
at de.anomic.server.serverAbstractBusyThread.run(serverAbstractBusyThread.java:163)

Thread= de.anomic.plasma.plasmaSwitchboard.dhtTransferJob id=40 TIMED_WAITING
at java.lang.Thread.sleep(Native Method)
at de.anomic.server.serverAbstractBusyThread.ratz(serverAbstractBusyThread.java:198)
at de.anomic.server.serverAbstractBusyThread.run(serverAbstractBusyThread.java:163)

Thread= Thread-1 id=8 TIMED_WAITING
at java.lang.Thread.sleep(Native Method)
at de.anomic.server.serverProfiling.run(serverProfiling.java:69)


THREADS WITH STATES: WAITING

Thread= pool-1-thread-1 id=70 WAITING
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:906)
at java.lang.Thread.run(Thread.java:619)

Thread= Finalizer daemon id=3 WAITING
at java.lang.Object.wait(Native Method)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)

Thread= pool-3-thread-2 id=21 WAITING
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
at java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:254)
at de.anomic.server.serverProcessor.enQueue(serverProcessor.java:88)
at de.anomic.server.serverAbstractBlockingThread.run(serverAbstractBlockingThread.java:72)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)

Thread= de.anomic.yacy.yacyCore.peerPing id=39 WAITING
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at de.anomic.server.serverSemaphore.P(serverSemaphore.java:82)
at de.anomic.yacy.yacyCore.publishMySeed(yacyCore.java:465)
at de.anomic.yacy.yacyCore.peerPing(yacyCore.java:267)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at de.anomic.server.serverInstantBusyThread.job(serverInstantBusyThread.java:115)
at de.anomic.server.serverAbstractBusyThread.run(serverAbstractBusyThread.java:142)

Thread= pool-3-thread-1 id=19 WAITING
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
at java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:254)
at de.anomic.server.serverProcessor.enQueue(serverProcessor.java:88)
at de.anomic.server.serverAbstractBlockingThread.run(serverAbstractBlockingThread.java:72)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)

Thread= Reference Handler daemon id=2 WAITING
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)

Thread= pool-5-thread-2 id=29 WAITING
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
at java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:254)
at de.anomic.server.serverProcessor.enQueue(serverProcessor.java:88)
at de.anomic.server.serverAbstractBlockingThread.run(serverAbstractBlockingThread.java:72)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)

Thread= Java2D Disposer daemon id=262 WAITING
at java.lang.Object.wait(Native Method)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
at sun.java2d.Disposer.run(Disposer.java:125)
at java.lang.Thread.run(Thread.java:619)

Thread= pool-4-thread-1 id=23 WAITING
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
at java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:254)
at de.anomic.server.serverProcessor.enQueue(serverProcessor.java:88)
at de.anomic.server.serverAbstractBlockingThread.run(serverAbstractBlockingThread.java:72)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)

Thread= MultiThreadedHttpConnectionManager cleanup daemon id=10 WAITING
at java.lang.Object.wait(Native Method)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$ReferenceQueueThread.run(MultiThreadedHttpConnectionManager.java:1122)

Thread= pool-4-thread-2 id=25 WAITING
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
at java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:254)
at de.anomic.server.serverProcessor.enQueue(serverProcessor.java:88)
at de.anomic.server.serverAbstractBlockingThread.run(serverAbstractBlockingThread.java:72)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)

Thread= main id=1 WAITING
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at de.anomic.server.serverSemaphore.P(serverSemaphore.java:82)
at de.anomic.plasma.plasmaSwitchboard.waitForShutdown(plasmaSwitchboard.java:2732)
at yacy.startup(yacy.java:420)
at yacy.main(yacy.java:991)

Thread= de.anomic.plasma.plasmaSwitchboard.deQueueProcess id=33 WAITING
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
at java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:254)
at de.anomic.server.serverProcessor.enQueue(serverProcessor.java:88)
at de.anomic.plasma.plasmaSwitchboard.deQueueProcess(plasmaSwitchboard.java:1867)
at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at de.anomic.server.serverInstantBusyThread.job(serverInstantBusyThread.java:115)
at de.anomic.server.serverAbstractBusyThread.run(serverAbstractBusyThread.java:142)

Thread= de.anomic.plasma.crawler.plasmaCrawlQueues.coreCrawlJob id=37 WAITING
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at de.anomic.plasma.crawler.plasmaCrawlQueues.coreCrawlJob(plasmaCrawlQueues.java:188)
at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at de.anomic.server.serverInstantBusyThread.job(serverInstantBusyThread.java:115)
at de.anomic.server.serverAbstractBusyThread.run(serverAbstractBusyThread.java:142)

Thread= pool-1-thread-2 id=71 WAITING
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:906)
at java.lang.Thread.run(Thread.java:619)

Thread= pool-5-thread-1 id=27 WAITING
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
at java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:254)
at de.anomic.server.serverProcessor.enQueue(serverProcessor.java:88)
at de.anomic.server.serverAbstractBlockingThread.run(serverAbstractBlockingThread.java:72)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)

Thread= Thread-5 id=13 WAITING
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at de.anomic.plasma.plasmaCrawlStacker.run(plasmaCrawlStacker.java:150)


THREADS WITH STATES: NEW


THREADS WITH STATES: TERMINATED


************* End Thread Dump Tue Apr 08 15:13:57 CEST 2008 *******************
miTreD
 
Beiträge: 1241
Registriert: Mi Jun 27, 2007 11:35 am
Wohnort: /home

Re: massiv paralleles indexieren?

Beitragvon Orbiter » Di Apr 08, 2008 2:22 pm

yep, der Indexierungsvorgang wurde nicht nur parallelisiert, sondern in mehrere separate+parallelisierte Abschnitte eingeteilt. Diese sind:
- parsen
- analysieren (condenser)
- webanalyse (Update der Webstruktur)
- Speichern
..wobei das Speichern nicht parallelisiert wurde, sondern die anderen Prozesse dort wieder in einen Synchronen Prozess einmünden.

Jeder der ersten drei Vorgänge hat CPU+1 Prozesse, die das abarbeiten, diese Queue wird gefüttert solange die Queue für das Speichern leer ist. Nun ist es möglich, das eine Datei erst durch den parser durch geht, dann aber in der Analyse 'hängen bleibt', und nachgeschobene Dateien diesen in der Queue quasi 'überholen'. Alles was hier rumschwirrt sieht man im Monitor als roten Eintrag.
Orbiter
 
Beiträge: 5792
Registriert: Di Jun 26, 2007 10:58 pm
Wohnort: Frankfurt am Main

Re: massiv paralleles indexieren?

Beitragvon miTreD » Di Apr 08, 2008 2:38 pm

OK, danke für die Erklärung. Dann ist ja alles ok.

[closed]
miTreD
 
Beiträge: 1241
Registriert: Mi Jun 27, 2007 11:35 am
Wohnort: /home


Zurück zu Fragen und Antworten

Wer ist online?

Mitglieder in diesem Forum: 0 Mitglieder und 3 Gäste