Plötzliche hohe Systemlast

Hier finden YaCy User Hilfe wenn was nicht funktioniert oder anders funktioniert als man dachte. Bei offensichtlichen Fehlern diese bitte gleich in die Bugs (http://bugs.yacy.net) eintragen.
Forumsregeln
In diesem Forum geht es um Benutzungsprobleme und Anfragen für Hilfe. Wird dabei ein Bug identifiziert, wird der thread zur Bearbeitung in die Bug-Sektion verschoben. Wer hier also einen Thread eingestellt hat und ihn vermisst, wird ihn sicherlich in der Bug-Sektion wiederfinden.

Plötzliche hohe Systemlast

Beitragvon dulcedo » Mo Mai 10, 2010 2:54 pm

Kann das jemand erklären, der peer läuft seit 13 Tagen und auch weiterhin problemlos, allerdings lastet java plötzlich und andauernd 380% aus (bei 4 cores).

Code: Alles auswählen
************* Start Thread Dump Mon May 10 15:49:17 CEST 2010 *******************

YaCy Version: 0.94/6839
Assigned   Memory = 3781033984
Used       Memory = 3404575608
Available  Memory = 376458376


THREADS WITH STATES: BLOCKED


THREADS WITH STATES: RUNNABLE

Thread= Thread-693954 id=2330567 RUNNABLE
at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
at org.apache.commons.httpclient.HttpParser.readRawLine(HttpParser.java:78)
at org.apache.commons.httpclient.HttpParser.readLine(HttpParser.java:106)
at org.apache.commons.httpclient.HttpConnection.readLine(HttpConnection.java:1116)
at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.readLine(MultiThreadedHttpConnectionManager.java:1413)
at org.apache.commons.httpclient.HttpMethodBase.readStatusLine(HttpMethodBase.java:1973)
at org.apache.commons.httpclient.HttpMethodBase.readResponse(HttpMethodBase.java:1735)
at org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:1098)
at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:398)
at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323)
at de.anomic.http.client.Client.execute(Client.java:472)
at de.anomic.http.client.Client.GET(Client.java:281)
at de.anomic.crawler.retrieval.HTTPLoader.load(HTTPLoader.java:134)
at de.anomic.crawler.retrieval.HTTPLoader.load(HTTPLoader.java:79)
at net.yacy.repository.LoaderDispatcher.load(LoaderDispatcher.java:262)
at net.yacy.repository.LoaderDispatcher.load(LoaderDispatcher.java:171)
at de.anomic.crawler.CrawlQueues$crawlWorker.run(CrawlQueues.java:562)


Thread= Session_89.12.79.236:34527#0 id=2330150 RUNNABLE
Thread= Session_89.12.79.236:36507#0 id=2330365 RUNNABLE
Thread= Session_89.12.79.236:39554#0 id=2330260 RUNNABLE
Thread= Session_89.12.79.236:50771#0 id=2330257 RUNNABLE
Thread= Session_89.12.79.236:53605#0 id=2330140 RUNNABLE
Thread= Session_89.12.79.236:57397#0 id=2330307 RUNNABLE
Thread= Session_89.12.79.236:58839#0 id=2330313 RUNNABLE
Thread= Session_89.12.79.236:60958#0 id=2330396 RUNNABLE
at java.util.TreeSet.size(TreeSet.java:190)
at search.respond(search.java:150)
at sun.reflect.GeneratedMethodAccessor926.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at de.anomic.http.server.HTTPDFileHandler.invokeServlet(HTTPDFileHandler.java:1197)
at de.anomic.http.server.HTTPDFileHandler.doResponse(HTTPDFileHandler.java:754)
at de.anomic.http.server.HTTPDFileHandler.doPost(HTTPDFileHandler.java:243)
at de.anomic.http.server.HTTPDemon.POST(HTTPDemon.java:587)
at sun.reflect.GeneratedMethodAccessor13.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at de.anomic.server.serverCore$Session.listen(serverCore.java:728)
at de.anomic.server.serverCore$Session.run(serverCore.java:626)


Thread= Session_127.0.0.1:52515#0 id=2330572 RUNNABLE
at java.lang.Thread.getAllStackTraces(Thread.java:1487)
at Threaddump_p.respond(Threaddump_p.java:99)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at de.anomic.http.server.HTTPDFileHandler.invokeServlet(HTTPDFileHandler.java:1197)
at de.anomic.http.server.HTTPDFileHandler.doResponse(HTTPDFileHandler.java:754)
at de.anomic.http.server.HTTPDFileHandler.doGet(HTTPDFileHandler.java:235)
at de.anomic.http.server.HTTPDemon.GET(HTTPDemon.java:454)
at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at de.anomic.server.serverCore$Session.listen(serverCore.java:728)
at de.anomic.server.serverCore$Session.run(serverCore.java:626)


Thread= httpd:8081 id=134 RUNNABLE
at java.lang.Thread.sleep(Native Method)
at net.yacy.kelondro.workflow.AbstractBusyThread.ratz(AbstractBusyThread.java:218)
at net.yacy.kelondro.workflow.AbstractBusyThread.run(AbstractBusyThread.java:182)



THREADS WITH STATES: TIMED_WAITING

Thread= Thread-1 id=10 TIMED_WAITING
at java.lang.Thread.sleep(Native Method)
at net.yacy.kelondro.util.MemoryTracker.run(MemoryTracker.java:59)


Thread= de.anomic.crawler.CrawlQueues.remoteCrawlLoaderJob id=120 TIMED_WAITING
Thread= de.anomic.data.bookmarksDB.autoReCrawl id=66 TIMED_WAITING
Thread= de.anomic.search.Switchboard.cleanupJob id=117 TIMED_WAITING
Thread= de.anomic.search.Switchboard.dhtTransferJob id=124 TIMED_WAITING
Thread= de.anomic.search.Switchboard.surrogateProcess id=118 TIMED_WAITING
Thread= de.anomic.yacy.yacyCore.peerPing id=123 TIMED_WAITING
Thread= de.anomic.yacy.yacyCore.publishSeedList id=122 TIMED_WAITING
at java.lang.Thread.sleep(Native Method)
at net.yacy.kelondro.workflow.AbstractBusyThread.ratz(AbstractBusyThread.java:218)
at net.yacy.kelondro.workflow.AbstractBusyThread.run(AbstractBusyThread.java:182)


Thread= de.anomic.crawler.CrawlQueues.coreCrawlJob id=121 TIMED_WAITING
at java.lang.Object.wait(Native Method)
at de.anomic.crawler.Balancer.pop(Balancer.java:434)
at de.anomic.crawler.NoticedURL.pop(NoticedURL.java:251)
at de.anomic.crawler.NoticedURL.pop(NoticedURL.java:218)
at de.anomic.crawler.CrawlQueues.coreCrawlJob(CrawlQueues.java:226)
at sun.reflect.GeneratedMethodAccessor16.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at net.yacy.kelondro.workflow.InstantBusyThread.job(InstantBusyThread.java:108)
at net.yacy.kelondro.workflow.AbstractBusyThread.run(AbstractBusyThread.java:161)



THREADS WITH STATES: WAITING

Thread= Thread-16 id=48 WAITING
Thread= Thread-17 id=49 WAITING
Thread= Thread-18 id=50 WAITING
Thread= Thread-19 id=51 WAITING
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
at net.yacy.kelondro.blob.Compressor$Worker.run(Compressor.java:103)


Thread= Log Runner id=9 WAITING
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
at net.yacy.kelondro.logging.Log$logRunner.run(Log.java:320)


Thread= MultiThreadedHttpConnectionManager cleanup daemon id=19 WAITING
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$ReferenceQueueThread.run(MultiThreadedHttpConnectionManager.java:1122)


Thread= main id=1 WAITING
at java.util.concurrent.Semaphore.acquire(Semaphore.java:286)
at de.anomic.search.Switchboard.waitForShutdown(Switchboard.java:2224)
at net.yacy.yacy.startup(yacy.java:395)
at net.yacy.yacy.main(yacy.java:1029)


Thread= Thread-4 id=18 WAITING
at java.util.concurrent.Semaphore.acquire(Semaphore.java:286)
at net.yacy.kelondro.rwi.IODispatcher.run(IODispatcher.java:149)


Thread= de.anomic.crawler.CrawlQueues.remoteTriggeredCrawlJob id=119 WAITING
at java.lang.Object.wait(Object.java:485)
at de.anomic.crawler.CrawlQueues.isPaused(CrawlQueues.java:299)
at de.anomic.crawler.CrawlQueues.remoteTriggeredCrawlJob(CrawlQueues.java:489)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at net.yacy.kelondro.workflow.InstantBusyThread.job(InstantBusyThread.java:108)
at net.yacy.kelondro.workflow.AbstractBusyThread.run(AbstractBusyThread.java:161)


Thread= Reference Handler daemon id=2 WAITING
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)


Thread= Finalizer daemon id=3 WAITING
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)


Thread= Java2D Disposer daemon id=146 WAITING
Thread= partition_pool-1-thread-1 id=20 WAITING
Thread= partition_pool-1-thread-2 id=21 WAITING
Thread= partition_pool-1-thread-7 id=26 WAITING
Thread= partition_pool-1-thread-8 id=27 WAITING
Thread= sorting_pool-1-thread-118 id=2086781 WAITING
Thread= sorting_pool-1-thread-120 id=2086783 WAITING
Thread= sorting_pool-1-thread-123 id=2306890 WAITING
Thread= sorting_pool-1-thread-124 id=2306891 WAITING
at java.lang.Thread.run(Thread.java:619)


Thread= condenseDocument_pool-1-thread-33 id=98 WAITING
Thread= condenseDocument_pool-1-thread-34 id=100 WAITING
Thread= condenseDocument_pool-1-thread-35 id=102 WAITING
Thread= condenseDocument_pool-1-thread-36 id=104 WAITING
Thread= condenseDocument_pool-1-thread-37 id=106 WAITING
Thread= job_pool-1-thread-19 id=70 WAITING
Thread= job_pool-1-thread-20 id=72 WAITING
Thread= job_pool-1-thread-21 id=74 WAITING
Thread= job_pool-1-thread-22 id=76 WAITING
Thread= job_pool-1-thread-23 id=78 WAITING
Thread= job_pool-1-thread-24 id=80 WAITING
Thread= job_pool-1-thread-25 id=82 WAITING
Thread= parseDocument_pool-1-thread-38 id=108 WAITING
Thread= parseDocument_pool-1-thread-39 id=110 WAITING
Thread= parseDocument_pool-1-thread-40 id=112 WAITING
Thread= parseDocument_pool-1-thread-41 id=114 WAITING
Thread= parseDocument_pool-1-thread-42 id=116 WAITING
Thread= storeDocumentIndex_pool-1-thread-10 id=31 WAITING
Thread= storeDocumentIndex_pool-1-thread-11 id=33 WAITING
Thread= storeDocumentIndex_pool-1-thread-12 id=35 WAITING
Thread= storeDocumentIndex_pool-1-thread-13 id=37 WAITING
Thread= storeDocumentIndex_pool-1-thread-14 id=39 WAITING
Thread= storeDocumentIndex_pool-1-thread-15 id=41 WAITING
Thread= storeDocumentIndex_pool-1-thread-16 id=43 WAITING
Thread= storeDocumentIndex_pool-1-thread-17 id=45 WAITING
Thread= storeDocumentIndex_pool-1-thread-18 id=47 WAITING
Thread= storeDocumentIndex_pool-1-thread-26 id=84 WAITING
Thread= storeDocumentIndex_pool-1-thread-27 id=86 WAITING
Thread= storeDocumentIndex_pool-1-thread-9 id=29 WAITING
Thread= webStructureAnalysis_pool-1-thread-28 id=88 WAITING
Thread= webStructureAnalysis_pool-1-thread-29 id=90 WAITING
Thread= webStructureAnalysis_pool-1-thread-30 id=92 WAITING
Thread= webStructureAnalysis_pool-1-thread-31 id=94 WAITING
Thread= webStructureAnalysis_pool-1-thread-32 id=96 WAITING
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
at net.yacy.kelondro.workflow.WorkflowProcessor.take(WorkflowProcessor.java:104)
at net.yacy.kelondro.workflow.AbstractBlockingThread.run(AbstractBlockingThread.java:57)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)



THREADS WITH STATES: NEW


THREADS WITH STATES: TERMINATED


************* End Thread Dump Mon May 10 15:49:17 CEST 2010 *******************
dulcedo
 
Beiträge: 1006
Registriert: Do Okt 16, 2008 6:36 pm
Wohnort: Bei Karlsruhe

Re: Plötzliche hohe Systemlast

Beitragvon Orbiter » Di Mai 11, 2010 11:28 pm

das ist sehr merkwürdig. Du hast da eine Last an einer Stelle die nicht wirklich viel Last erzeugt und ausserdem nur vorkommen kann wenn (jemand) deinen Peer extrem oft von aussen abgefragt hat. Die betreffende Codestelle ist dort wo ein brute force von requests getestet wird.
Damit eine Ursache für die Last weg kommt habe ich hier eine Begrenzung der Zugriffe auf den brute-force cache eingebaut ohne die Funktion zu schmälern. SVN 6866. Mal schauen ob das hilft.
Orbiter
 
Beiträge: 5792
Registriert: Di Jun 26, 2007 10:58 pm
Wohnort: Frankfurt am Main


Zurück zu Fragen und Antworten

Wer ist online?

Mitglieder in diesem Forum: 0 Mitglieder und 1 Gast