v0.593 / yacy hängt - solved

Hier finden YaCy User Hilfe wenn was nicht funktioniert oder anders funktioniert als man dachte. Bei offensichtlichen Fehlern diese bitte gleich in die Bugs (http://bugs.yacy.net) eintragen.
Forumsregeln
In diesem Forum geht es um Benutzungsprobleme und Anfragen für Hilfe. Wird dabei ein Bug identifiziert, wird der thread zur Bearbeitung in die Bug-Sektion verschoben. Wer hier also einen Thread eingestellt hat und ihn vermisst, wird ihn sicherlich in der Bug-Sektion wiederfinden.

v0.593 / yacy hängt - solved

Beitragvon Halborinda » Fr Jul 18, 2008 6:54 am

Es gab bei mir vor kurzen ein Systemabsturz.

Seitdem hängt YaCy. Die "DATA/HTCACHE/responseHeader.heap" habe ich schon gelöscht, was aber leider keine Wirkung zeigte.

YaCy lastet einen meiner beiden CPUs zu 100% aus. Das kann über 40 Minuten gehen, ohne das sich was ändert.

Die letzten aus dem Log (sie sind immer gleich):
Code: Alles auswählen
S 2008/07/18 07:38:09 CRAWLER Starting Crawling Management
I 2008/07/18 07:38:09 Balancer re-creating the urlNoticeCoreStack balancer stack, size = 22 (the old stack size was wrong)
D 2008/07/18 07:38:09 KELONDRO file '/yacy/DATA/PLASMADB/urlNoticeCoreStack9.stack' closed.
W 2008/07/18 07:38:09 KELONDRO USEDC inconsistency at startup: calculated_used = 684262, USEDC = 0, FREEC = 0, recordsize = 20, file = /server/yacy/yacy-8080/DATA/PLASMADB/urlNoticeLimitStack9.stack
I 2008/07/18 07:38:09 YACY BOOTSTRAP: 31 seeds from seed-list URL http://low.audioattack.de/yacy/seed.txt, AGE=0h
I 2008/07/18 07:38:10 YACY BOOTSTRAP: 49 seeds from seed-list URL http://www.lulabad.de/seed.txt, AGE=0h
I 2008/07/18 07:38:10 YACY BOOTSTRAP: 6 new seeds while bootstraping.
I 2008/07/18 07:38:24 Balancer re-creating the urlNoticeLimitStack balancer stack, size = 1344292 (the old stack size was wrong)
D 2008/07/18 07:38:24 KELONDRO file '/yacy/DATA/PLASMADB/urlNoticeLimitStack9.stack' closed.


Auch der Thread-Dump ist immer identisch:
Code: Alles auswählen
Thread 10003 "MultiThreadedHttpConnectionManager cleanup": (state = BLOCKED)
   at java.lang.Object.wait(Native Method)
   - waiting on <0x43b66e30> (a java.lang.ref.ReferenceQueue$Lock)
   at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
   - locked <0x43b66e30> (a java.lang.ref.ReferenceQueue$Lock)
   at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
   at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$ReferenceQueueThread.run(MultiThreadedHttpConnectionManager.java:1122)

Thread 9974 "Thread-1": (state = BLOCKED)
   at java.lang.Thread.sleep(Native Method)
   at de.anomic.server.serverProfiling.run(serverProfiling.java:67)

VM Thread 9971 "Low Memory Detector": (state = BLOCKED)

VM Thread 9970 "CompilerThread1": (state = BLOCKED)

VM Thread 9969 "CompilerThread0": (state = BLOCKED)

Thread 9968 "Signal Dispatcher": (state = BLOCKED)

Thread 9967 "Surrogate Locker Thread (CMS)": (state = BLOCKED)

Thread 9966 "Finalizer": (state = BLOCKED)
   at java.lang.Object.wait(Native Method)
   - waiting on <0x12b00ab0> (a java.lang.ref.ReferenceQueue$Lock)
   at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
   - locked <0x12b00ab0> (a java.lang.ref.ReferenceQueue$Lock)
   at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
   at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)

Thread 9965 "Reference Handler": (state = BLOCKED)
   at java.lang.Object.wait(Native Method)
   - waiting on <0x12b009a8> (a java.lang.ref.Reference$Lock)
   at java.lang.Object.wait(Object.java:485)
   at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
   - locked <0x12b009a8> (a java.lang.ref.Reference$Lock)

Thread 9960 "main": (state = IN_JAVA)
   at java.util.HashMap$EntrySet.iterator(HashMap.java:950)
   at de.anomic.crawler.Balancer.flushOnceDomStacks(Balancer.java:311)
   - locked <0x442a3780> (a java.util.HashMap)
   at de.anomic.crawler.Balancer.pushHash(Balancer.java:373)
   at de.anomic.crawler.Balancer.<init>(Balancer.java:127)
   at de.anomic.crawler.NoticedURL.<init>(NoticedURL.java:81)
   at de.anomic.crawler.CrawlQueues.<init>(CrawlQueues.java:71)
   at de.anomic.plasma.plasmaSwitchboard.<init>(plasmaSwitchboard.java:1037)
   at yacy.startup(yacy.java:236)
   at yacy.main(yacy.java:1008)


Sonstige Informationen:
Code: Alles auswählen
JVM: Java HotSpot(TM) Server VM (10.0-b22, mixed mode)
Java Home: /usr/lib/jvm/java-6-sun-1.6.0.06/jre

JVM arguements:
-Xms128m
-Xmx2560m
-XX:+UseConcMarkSweepGC
-XX:+CMSIncrementalMode
-XX:+UseAdaptiveSizePolicy
-XX:+UseLargePages
-Djava.awt.headless=true

releaseVersion=0.593
releaseNr=$Revision: 5003 $


Grüße

P.S.: Ich habe für den Heap-Dump VisualVM genutzt: https://visualvm.dev.java.net/ (Das ist auch im neuesten Sun Java 1.6.0 Update dabei)
Zuletzt geändert von Halborinda am Fr Jul 25, 2008 6:39 pm, insgesamt 1-mal geändert.
Halborinda
 
Beiträge: 217
Registriert: Sa Jul 07, 2007 7:58 am

Re: v0.593 / yacy hängt

Beitragvon Halborinda » Sa Jul 19, 2008 7:55 am

Ich habe heute ein Update auf Revision 5008 gemacht, was leider nichts geändert hat.

Um 8:14 habe ich den Server gestartet. Um 8:17 wurde der letzte Log-Eintrag geschrieben. Auch um 8:47 ist nur CPU-Last zu verspüren.

Die letzten Log-Einträge:
Code: Alles auswählen
I 2008/07/19 08:17:15 PARSER Found functional parser for mimeType 'application/x-zip'.
        Name:    Compressed Archive File Parser
        Version: 0.1
        Class:   de.anomic.plasma.parser.zip.zipParser
I 2008/07/19 08:17:15 PARSER Found functional parser for mimeType 'application/java-archive'.
        Name:    Compressed Archive File Parser
        Version: 0.1
        Class:   de.anomic.plasma.parser.zip.zipParser
I 2008/07/19 08:17:15 PARSER Found functional parser for mimeType 'application/zip'.
        Name:    Compressed Archive File Parser
        Version: 0.1
        Class:   de.anomic.plasma.parser.zip.zipParser
I 2008/07/19 08:17:15 PARSER Found functional parser for mimeType 'application/x-zip-compressed'.
        Name:    Compressed Archive File Parser
        Version: 0.1
        Class:   de.anomic.plasma.parser.zip.zipParser
S 2008/07/19 08:17:15 CRAWLER Starting Crawling Management
I 2008/07/19 08:17:15 Balancer re-creating the urlNoticeCoreStack balancer stack, size = 22 (the old stack size was wrong)
D 2008/07/19 08:17:15 KELONDRO file '/yacy/DATA/PLASMADB/urlNoticeCoreStack9.stack' closed.
W 2008/07/19 08:17:15 KELONDRO USEDC inconsistency at startup: calculated_used = 136115, USEDC = 0, FREEC = 0, recordsize = 20, file = /yacy/DATA/PLASMADB/urlNoticeLimitStack9.stack
I 2008/07/19 08:17:16 YACY BOOTSTRAP: 50 seeds from seed-list URL http://low.audioattack.de/yacy/seed.txt, AGE=0h
I 2008/07/19 08:17:16 YACY BOOTSTRAP: 52 seeds from seed-list URL http://www.lulabad.de/seed.txt, AGE=0h
I 2008/07/19 08:17:16 YACY BOOTSTRAP: 3 new seeds while bootstraping.
I 2008/07/19 08:17:31 Balancer re-creating the urlNoticeLimitStack balancer stack, size = 1344292 (the old stack size was wrong)
D 2008/07/19 08:17:31 KELONDRO file '/yacy/DATA/PLASMADB/urlNoticeLimitStack9.stack' closed.


Ich habe zwei Screenshots des Profilers noch angehangen, da kann man die Thread- und Heap-Entwicklung gut sehen. Was sich da abzeichnet, ist ein Sägezahnmuster im Heap.

Sonst noch ein paar Threaddumps:
8:18:58 Uhr
Code: Alles auswählen
Thread 8764 "de.anomic.plasma.plasmaHTCache.cacheScan": (state = BLOCKED)
   at java.lang.Thread.sleep(Native Method)
   at de.anomic.server.serverAbstractBusyThread.ratz(serverAbstractBusyThread.java:198)
   at de.anomic.server.serverAbstractBusyThread.run(serverAbstractBusyThread.java:103)

Thread 8763 "MultiThreadedHttpConnectionManager cleanup": (state = BLOCKED)
   at java.lang.Object.wait(Native Method)
   - waiting on <0x43931cd0> (a java.lang.ref.ReferenceQueue$Lock)
   at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
   - locked <0x43931cd0> (a java.lang.ref.ReferenceQueue$Lock)
   at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
   at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$ReferenceQueueThread.run(MultiThreadedHttpConnectionManager.java:1122)

Thread 8555 "Thread-1": (state = BLOCKED)
   at java.lang.Thread.sleep(Native Method)
   at de.anomic.server.serverProfiling.run(serverProfiling.java:67)

VM Thread 8553 "Low Memory Detector": (state = BLOCKED)

VM Thread 8552 "CompilerThread1": (state = BLOCKED)

VM Thread 8551 "CompilerThread0": (state = BLOCKED)

Thread 8550 "Signal Dispatcher": (state = BLOCKED)

Thread 8549 "Surrogate Locker Thread (CMS)": (state = BLOCKED)

Thread 8548 "Finalizer": (state = BLOCKED)
   at java.lang.Object.wait(Native Method)
   - waiting on <0x12b60ab0> (a java.lang.ref.ReferenceQueue$Lock)
   at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
   - locked <0x12b60ab0> (a java.lang.ref.ReferenceQueue$Lock)
   at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
   at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)

Thread 8547 "Reference Handler": (state = BLOCKED)
   at java.lang.Object.wait(Native Method)
   - waiting on <0x12b609a8> (a java.lang.ref.Reference$Lock)
   at java.lang.Object.wait(Object.java:485)
   at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
   - locked <0x12b609a8> (a java.lang.ref.Reference$Lock)

Thread 8542 "main": (state = IN_JAVA)
   at java.util.HashMap$EntrySet.iterator(HashMap.java:950)
   at de.anomic.crawler.Balancer.flushOnceDomStacks(Balancer.java:311)
   - locked <0x4354a500> (a java.util.HashMap)
   at de.anomic.crawler.Balancer.pushHash(Balancer.java:373)
   at de.anomic.crawler.Balancer.<init>(Balancer.java:127)
   at de.anomic.crawler.NoticedURL.<init>(NoticedURL.java:81)
   at de.anomic.crawler.CrawlQueues.<init>(CrawlQueues.java:71)
   at de.anomic.plasma.plasmaSwitchboard.<init>(plasmaSwitchboard.java:1037)
   at yacy.startup(yacy.java:237)
   at yacy.main(yacy.java:1012)


8:20:37 Uhr:
Code: Alles auswählen
Thread 8763 "MultiThreadedHttpConnectionManager cleanup": (state = BLOCKED)
   at java.lang.Object.wait(Native Method)
   - waiting on <0x43931cd0> (a java.lang.ref.ReferenceQueue$Lock)
   at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
   - locked <0x43931cd0> (a java.lang.ref.ReferenceQueue$Lock)
   at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
   at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$ReferenceQueueThread.run(MultiThreadedHttpConnectionManager.java:1122)

Thread 8555 "Thread-1": (state = BLOCKED)
   at java.lang.Thread.sleep(Native Method)
   at de.anomic.server.serverProfiling.run(serverProfiling.java:67)

VM Thread 8553 "Low Memory Detector": (state = BLOCKED)

VM Thread 8552 "CompilerThread1": (state = BLOCKED)

VM Thread 8551 "CompilerThread0": (state = BLOCKED)

Thread 8550 "Signal Dispatcher": (state = BLOCKED)

Thread 8549 "Surrogate Locker Thread (CMS)": (state = BLOCKED)

Thread 8548 "Finalizer": (state = BLOCKED)
   at java.lang.Object.wait(Native Method)
   - waiting on <0x12b60ab0> (a java.lang.ref.ReferenceQueue$Lock)
   at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
   - locked <0x12b60ab0> (a java.lang.ref.ReferenceQueue$Lock)
   at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
   at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)

Thread 8547 "Reference Handler": (state = BLOCKED)
   at java.lang.Object.wait(Native Method)
   - waiting on <0x12b609a8> (a java.lang.ref.Reference$Lock)
   at java.lang.Object.wait(Object.java:485)
   at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
   - locked <0x12b609a8> (a java.lang.ref.Reference$Lock)

Thread 8542 "main": (state = IN_JAVA)
   at java.util.HashMap$EntrySet.iterator(HashMap.java:950)
   at de.anomic.crawler.Balancer.flushOnceDomStacks(Balancer.java:311)
   - locked <0x4354a500> (a java.util.HashMap)
   at de.anomic.crawler.Balancer.pushHash(Balancer.java:373)
   at de.anomic.crawler.Balancer.<init>(Balancer.java:127)
   at de.anomic.crawler.NoticedURL.<init>(NoticedURL.java:81)
   at de.anomic.crawler.CrawlQueues.<init>(CrawlQueues.java:71)
   at de.anomic.plasma.plasmaSwitchboard.<init>(plasmaSwitchboard.java:1037)
   at yacy.startup(yacy.java:237)
   at yacy.main(yacy.java:1012)


Seit dem ist der Threaddump absolut identisch. Auch um 8:45:11 Uhr.
Dateianhänge
yacy-monitor-02.png
Heap, Threads, Loaded Classes, PermGen
ca 8:47 Uhr
yacy-monitor-02.png (46.74 KiB) 1538-mal betrachtet
yacy-monitor-01.png
Heap, Threads, Loaded Classes, PermGen
ca 8:21 Uhr
yacy-monitor-01.png (47.52 KiB) 1537-mal betrachtet
Halborinda
 
Beiträge: 217
Registriert: Sa Jul 07, 2007 7:58 am

Re: v0.593 / yacy hängt

Beitragvon DanielR » Sa Jul 19, 2008 8:33 am

da kann dir wohl nur Orbiter helfen. Der weiß sicherlich was da gemacht wird und was das Problem sein kann...
DanielR
 
Beiträge: 395
Registriert: Di Feb 12, 2008 2:22 pm

Re: v0.593 / yacy hängt

Beitragvon Halborinda » Sa Jul 19, 2008 8:40 am

Okay...

Ich würd meinen Server gerne wieder zum Laufen bringen. 11 Millionen Links und 19 Millionen RWIs würde ich ungerne wegschmeißen. Da haben meine Rechner ganz schon geknechtet dafür...

Jetzt um 9:35 hat sich der Threaddump immer noch nicht geändert. Der Heap wächst dafür wohl ganz gemächlich... Und die Zähne scheinen länger zu werden. (Siehe Screenshot)

Werd Mal YaCy killen.

Grüße
Dateianhänge
yacy-monitor-03.png
Heap, Threads, Loaded Classes, PermGen
ca 9:36 Uhr
yacy-monitor-03.png (46.12 KiB) 1524-mal betrachtet
Halborinda
 
Beiträge: 217
Registriert: Sa Jul 07, 2007 7:58 am

Re: v0.593 / yacy hängt

Beitragvon Halborinda » Sa Jul 19, 2008 9:00 am

Kaum wollte ich den kill Befehl absetzen, hat sich alles geändert. YaCy scheint wieder zu laufen, nach fast 1 1/2 Stunden reine CPU Last.

Nun dachte ich mir, fahre ich ihn Mal geregelt herunten und schau, ob auch dann das sich reproduzieren lässt...

Aber seit gut zwanzig Minuten arbeitet YaCy trotzdem fröhlich weiter:
Code: Alles auswählen
D 2008/07/19 09:59:15 ROBOTS Robots.txt successfully loaded from URL 'http://xxx.xxx.com/robots.txt' in 478 ms.
D 2008/07/19 09:59:15 ROBOTS Trying to download the robots.txt file from URL 'http://www.xxx.com/robots.txt'.
D 2008/07/19 09:59:16 ROBOTS robots.txt could not be downloaded from URL 'http://www.xxx.com/robots.txt'. [404 Not Found].
D 2008/07/19 09:59:17 ROBOTS Trying to download the robots.txt file from URL 'http://www.xxx.org/robots.txt'.
I 2008/07/19 09:59:17 PLASMA DEQUEUE: Shutdown detected.
D 2008/07/19 09:59:18 ROBOTS Robots.txt successfully loaded from URL 'http://www.xxx.org/robots.txt' in 970 ms.
D 2008/07/19 09:59:18 ROBOTS Trying to download the robots.txt file from URL 'http://xxx.org/robots.txt'.
D 2008/07/19 09:59:19 ROBOTS Trying to download the robots.txt file from URL 'http://www.xxx.com/robots.txt'.
Halborinda
 
Beiträge: 217
Registriert: Sa Jul 07, 2007 7:58 am

Re: v0.593 / yacy hängt

Beitragvon Halborinda » Sa Jul 19, 2008 9:17 am

Anscheinend irritiert YaCy der Shutdown-Wunsch gar nicht...

Thread-Dump 10:01:05: http://nopaste.debianforum.de/9692
Thread-Dump 10:01:46: http://nopaste.debianforum.de/9693
Thread-Dump 10:09:00: http://nopaste.debianforum.de/9694
Dateianhänge
yacy-monitor-05.png
Heap, Threads, Loaded Classes, PermGen
ca 10:09 Uhr
yacy-monitor-05.png (52.39 KiB) 1510-mal betrachtet
yacy-monitor-04.png
Heap, Threads, Loaded Classes, PermGen
ca 10:02 Uhr
yacy-monitor-04.png (51.83 KiB) 1510-mal betrachtet
Halborinda
 
Beiträge: 217
Registriert: Sa Jul 07, 2007 7:58 am

Re: v0.593 / yacy hängt

Beitragvon miTreD » Mo Jul 21, 2008 9:54 am

Halborinda hat geschrieben:Anscheinend irritiert YaCy der Shutdown-Wunsch gar nicht...
Jepp, hier auch nicht.
Das Log ist nur voll von "I 2008/07/21 10:53:36 PLASMA DEQUEUE: Shutdown detected."
0.593/05012
thread dump:
Code: Alles auswählen
Full thread dump Java HotSpot(TM) Server VM (10.0-b19 mixed mode):

"Session_127.0.0.1:34732#0" prio=10 tid=0x0897e000 nid=0x1810 runnable [0x8807e000..0x8807ef20]
   java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(SocketInputStream.java:129)
        at java.net.SocketInputStream.read(SocketInputStream.java:182)
        at java.io.FilterInputStream.read(FilterInputStream.java:66)
        at java.io.PushbackInputStream.read(PushbackInputStream.java:122)
        at de.anomic.server.serverCore.receive(serverCore.java:838)
        at de.anomic.server.serverCore$Session.readLine(serverCore.java:564)
        at de.anomic.server.serverCore$Session.listen(serverCore.java:669)
        at de.anomic.server.serverCore$Session.run(serverCore.java:618)

"Session_127.0.0.1:34731#1" prio=10 tid=0x083cdc00 nid=0x180a runnable [0x8846a000..0x8846afa0]
   java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(SocketInputStream.java:129)
        at java.net.SocketInputStream.read(SocketInputStream.java:182)
        at java.io.FilterInputStream.read(FilterInputStream.java:66)
        at java.io.PushbackInputStream.read(PushbackInputStream.java:122)
        at de.anomic.server.serverCore.receive(serverCore.java:838)
        at de.anomic.server.serverCore$Session.readLine(serverCore.java:564)
        at de.anomic.server.serverCore$Session.listen(serverCore.java:669)
        at de.anomic.server.serverCore$Session.run(serverCore.java:618)

"Session_127.0.0.1:34730#1" prio=10 tid=0x087b4c00 nid=0x1804 runnable [0x8855d000..0x8855e020]
   java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(SocketInputStream.java:129)
        at java.net.SocketInputStream.read(SocketInputStream.java:182)
        at java.io.FilterInputStream.read(FilterInputStream.java:66)
        at java.io.PushbackInputStream.read(PushbackInputStream.java:122)
        at de.anomic.server.serverCore.receive(serverCore.java:838)
        at de.anomic.server.serverCore$Session.readLine(serverCore.java:564)
        at de.anomic.server.serverCore$Session.listen(serverCore.java:669)
        at de.anomic.server.serverCore$Session.run(serverCore.java:618)

"Session_127.0.0.1:34727#5" prio=10 tid=0x0863dc00 nid=0x17eb runnable [0x8815c000..0x8815ce20]
   java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(SocketInputStream.java:129)
        at java.net.SocketInputStream.read(SocketInputStream.java:182)
        at java.io.FilterInputStream.read(FilterInputStream.java:66)
        at java.io.PushbackInputStream.read(PushbackInputStream.java:122)
        at de.anomic.server.serverCore.receive(serverCore.java:838)
        at de.anomic.server.serverCore$Session.readLine(serverCore.java:564)
        at de.anomic.server.serverCore$Session.listen(serverCore.java:669)
        at de.anomic.server.serverCore$Session.run(serverCore.java:618)

"Session_127.0.0.1:34726#5" prio=10 tid=0x0863cc00 nid=0x17e5 runnable [0x881ad000..0x881adea0]
   java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(SocketInputStream.java:129)
        at java.net.SocketInputStream.read(SocketInputStream.java:182)
        at java.io.FilterInputStream.read(FilterInputStream.java:66)
        at java.io.PushbackInputStream.read(PushbackInputStream.java:122)
        at de.anomic.server.serverCore.receive(serverCore.java:838)
        at de.anomic.server.serverCore$Session.readLine(serverCore.java:564)
        at de.anomic.server.serverCore$Session.listen(serverCore.java:669)
        at de.anomic.server.serverCore$Session.run(serverCore.java:618)

"Session_127.0.0.1:34725#5" prio=10 tid=0x08804c00 nid=0x17df runnable [0x881fe000..0x881fef20]
   java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(SocketInputStream.java:129)
        at java.net.SocketInputStream.read(SocketInputStream.java:182)
        at java.io.FilterInputStream.read(FilterInputStream.java:66)
        at java.io.PushbackInputStream.read(PushbackInputStream.java:122)
        at de.anomic.server.serverCore.receive(serverCore.java:838)
        at de.anomic.server.serverCore$Session.readLine(serverCore.java:564)
        at de.anomic.server.serverCore$Session.listen(serverCore.java:669)
        at de.anomic.server.serverCore$Session.run(serverCore.java:618)

"Java2D Disposer" daemon prio=10 tid=0x08803800 nid=0x17d9 in Object.wait() [0x8824f000..0x8824ffa0]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0xa4ce8130> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
        - locked <0xa4ce8130> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
        at sun.java2d.Disposer.run(Disposer.java:125)
        at java.lang.Thread.run(Thread.java:619)

"Session_127.0.0.1:34724#5" prio=10 tid=0x08266800 nid=0x17d8 runnable [0x882a0000..0x882a1020]
   java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(SocketInputStream.java:129)
        at java.net.SocketInputStream.read(SocketInputStream.java:182)
        at java.io.FilterInputStream.read(FilterInputStream.java:66)
        at java.io.PushbackInputStream.read(PushbackInputStream.java:122)
        at de.anomic.server.serverCore.receive(serverCore.java:838)
        at de.anomic.server.serverCore$Session.readLine(serverCore.java:564)
        at de.anomic.server.serverCore$Session.listen(serverCore.java:669)
        at de.anomic.server.serverCore$Session.run(serverCore.java:618)

"Session_127.0.0.1:34723#4" prio=10 tid=0x0820c400 nid=0x17d2 runnable [0x88377000..0x883780a0]
   java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(SocketInputStream.java:129)
        at java.net.SocketInputStream.read(SocketInputStream.java:182)
        at java.io.FilterInputStream.read(FilterInputStream.java:66)
        at java.io.PushbackInputStream.read(PushbackInputStream.java:122)
        at de.anomic.server.serverCore.receive(serverCore.java:838)
        at de.anomic.server.serverCore$Session.readLine(serverCore.java:564)
        at de.anomic.server.serverCore$Session.listen(serverCore.java:669)
        at de.anomic.server.serverCore$Session.run(serverCore.java:618)

"Session_127.0.0.1:34721#6" prio=10 tid=0x08237400 nid=0x17c8 runnable [0x88419000..0x88419da0]
   java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(SocketInputStream.java:129)
        at java.net.SocketInputStream.read(SocketInputStream.java:182)
        at java.io.FilterInputStream.read(FilterInputStream.java:66)
        at java.io.PushbackInputStream.read(PushbackInputStream.java:122)
        at de.anomic.server.serverCore.receive(serverCore.java:838)
        at de.anomic.server.serverCore$Session.readLine(serverCore.java:564)
        at de.anomic.server.serverCore$Session.listen(serverCore.java:669)
        at de.anomic.server.serverCore$Session.run(serverCore.java:618)

"PublishSeed_yasuna" prio=10 tid=0x085aa400 nid=0x1793 runnable [0x8916f000..0x8916fda0]
   java.lang.Thread.State: RUNNABLE
        at java.net.PlainSocketImpl.socketConnect(Native Method)
        at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
        - locked <0x8c310ee8> (a java.net.SocksSocketImpl)
        at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
        at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
        at java.net.Socket.connect(Socket.java:519)
        at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.commons.httpclient.protocol.ReflectionSocketFactory.createSocket(ReflectionSocketFactory.java:140)
        at org.apache.commons.httpclient.protocol.DefaultProtocolSocketFactory.createSocket(DefaultProtocolSocketFactory.java:125)
        at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707)
        at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361)
        at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387)
        at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
        at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
        at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323)
        at de.anomic.http.JakartaCommonsHttpClient.execute(JakartaCommonsHttpClient.java:413)
        at de.anomic.http.JakartaCommonsHttpClient.POST(JakartaCommonsHttpClient.java:314)
        at de.anomic.yacy.yacyClient.wput(yacyClient.java:277)
        at de.anomic.yacy.yacyClient.publishMySeed(yacyClient.java:123)
        at de.anomic.yacy.yacyCore$publishThread.run(yacyCore.java:263)

"Session_85.179.98.7:2112#19" prio=10 tid=0x081f1000 nid=0x1777 runnable [0x88650000..0x88651020]
   java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(SocketInputStream.java:129)
        at java.net.SocketInputStream.read(SocketInputStream.java:182)
        at java.io.FilterInputStream.read(FilterInputStream.java:66)
        at java.io.PushbackInputStream.read(PushbackInputStream.java:122)
        at de.anomic.server.serverCore.receive(serverCore.java:838)
        at de.anomic.server.serverCore$Session.readLine(serverCore.java:564)
        at de.anomic.server.serverCore$Session.listen(serverCore.java:669)
        at de.anomic.server.serverCore$Session.run(serverCore.java:618)

"Session_130.75.2.39:43782#3" prio=10 tid=0x08a12400 nid=0x1773 runnable [0x885ff000..0x885ffda0]
   java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(SocketInputStream.java:129)
        at java.net.SocketInputStream.read(SocketInputStream.java:182)
        at java.io.FilterInputStream.read(FilterInputStream.java:66)
        at java.io.PushbackInputStream.read(PushbackInputStream.java:122)
        at de.anomic.server.serverCore.receive(serverCore.java:838)
        at de.anomic.server.serverCore$Session.readLine(serverCore.java:564)
        at de.anomic.server.serverCore$Session.listen(serverCore.java:669)
        at de.anomic.server.serverCore$Session.run(serverCore.java:618)

"urls_pool-1-thread-10" prio=10 tid=0x08218c00 nid=0x1762 waiting on condition [0x886a1000..0x886a2120]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x92227158> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:906)
        at java.lang.Thread.run(Thread.java:619)

"urls_pool-1-thread-9" prio=10 tid=0x087b3c00 nid=0x1761 waiting on condition [0x892fb000..0x892fbda0]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x92227158> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:906)
        at java.lang.Thread.run(Thread.java:619)

"Session_87.164.230.44:3533#0" prio=10 tid=0x087b3400 nid=0x175d runnable [0x88743000..0x88743ea0]
   java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(SocketInputStream.java:129)
        at java.net.SocketInputStream.read(SocketInputStream.java:182)
        at java.io.FilterInputStream.read(FilterInputStream.java:66)
        at java.io.PushbackInputStream.read(PushbackInputStream.java:122)
        at de.anomic.server.serverCore.receive(serverCore.java:838)
        at de.anomic.server.serverCore$Session.readLine(serverCore.java:564)
        at de.anomic.server.serverCore$Session.listen(serverCore.java:669)
        at de.anomic.server.serverCore$Session.run(serverCore.java:618)

"Session_91.36.165.220:49451#0" prio=10 tid=0x0880b400 nid=0x175b runnable [0x8850c000..0x8850cfa0]
   java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(SocketInputStream.java:129)
        at java.net.SocketInputStream.read(SocketInputStream.java:182)
        at java.io.FilterInputStream.read(FilterInputStream.java:66)
        at java.io.PushbackInputStream.read(PushbackInputStream.java:122)
        at de.anomic.server.serverCore.receive(serverCore.java:838)
        at de.anomic.server.serverCore$Session.readLine(serverCore.java:564)
        at de.anomic.server.serverCore$Session.listen(serverCore.java:669)
        at de.anomic.server.serverCore$Session.run(serverCore.java:618)

"Session_91.36.165.220:60958#0" prio=10 tid=0x0863fc00 nid=0x1753 runnable [0x88794000..0x88794fa0]
   java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(SocketInputStream.java:129)
        at java.net.SocketInputStream.read(SocketInputStream.java:182)
        at java.io.FilterInputStream.read(FilterInputStream.java:66)
        at java.io.PushbackInputStream.read(PushbackInputStream.java:122)
        at de.anomic.server.serverCore.receive(serverCore.java:838)
        at de.anomic.server.serverCore$Session.readLine(serverCore.java:564)
        at de.anomic.server.serverCore$Session.listen(serverCore.java:669)
        at de.anomic.server.serverCore$Session.run(serverCore.java:618)

"httpd:27001" prio=10 tid=0x0811ec00 nid=0x174c runnable [0x88aca000..0x88acaea0]
   java.lang.Thread.State: RUNNABLE
        at java.net.PlainSocketImpl.socketAccept(Native Method)
        at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
        - locked <0xa4bd8920> (a java.net.SocksSocketImpl)
        at java.net.ServerSocket.implAccept(ServerSocket.java:453)
        at java.net.ServerSocket.accept(ServerSocket.java:421)
        at de.anomic.server.serverCore.job(serverCore.java:330)
        at de.anomic.server.serverAbstractBusyThread.run(serverAbstractBusyThread.java:142)

"de.anomic.plasma.plasmaSwitchboard.dhtTransferJob" prio=10 tid=0x083ffc00 nid=0x174b waiting on condition [0x8902b000..0x8902bf20]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
        at java.lang.Thread.sleep(Native Method)
        at de.anomic.server.serverAbstractBusyThread.ratz(serverAbstractBusyThread.java:198)
        at de.anomic.server.serverAbstractBusyThread.run(serverAbstractBusyThread.java:163)

"de.anomic.yacy.yacyCore.peerPing" prio=10 tid=0x0864d000 nid=0x174a in Object.wait() [0x88b1b000..0x88b1bfa0]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0xa4c25870> (a de.anomic.server.serverSemaphore)
        at java.lang.Object.wait(Object.java:485)
        at de.anomic.server.serverSemaphore.P(serverSemaphore.java:63)
        - locked <0xa4c25870> (a de.anomic.server.serverSemaphore)
        at de.anomic.yacy.yacyCore.publishMySeed(yacyCore.java:421)
        at de.anomic.yacy.yacyCore.peerPing(yacyCore.java:223)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at de.anomic.server.serverInstantBusyThread.job(serverInstantBusyThread.java:96)
        at de.anomic.server.serverAbstractBusyThread.run(serverAbstractBusyThread.java:142)

"de.anomic.yacy.yacyCore.publishSeedList" prio=10 tid=0x0864c000 nid=0x1749 waiting on condition [0x88b6c000..0x88b6d020]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
        at java.lang.Thread.sleep(Native Method)
        at de.anomic.server.serverAbstractBusyThread.ratz(serverAbstractBusyThread.java:198)
        at de.anomic.server.serverAbstractBusyThread.run(serverAbstractBusyThread.java:163)

"de.anomic.crawler.CrawlQueues.coreCrawlJob" prio=10 tid=0x085e4c00 nid=0x1748 waiting on condition [0x88bbd000..0x88bbe0a0]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
        at java.lang.Thread.sleep(Native Method)
        at de.anomic.server.serverAbstractBusyThread.ratz(serverAbstractBusyThread.java:198)
        at de.anomic.server.serverAbstractBusyThread.run(serverAbstractBusyThread.java:163)

"de.anomic.crawler.CrawlQueues.remoteCrawlLoaderJob" prio=10 tid=0x085e4000 nid=0x1747 waiting on condition [0x88c0e000..0x88c0f120]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
        at java.lang.Thread.sleep(Native Method)
        at de.anomic.server.serverAbstractBusyThread.ratz(serverAbstractBusyThread.java:198)
        at de.anomic.server.serverAbstractBusyThread.run(serverAbstractBusyThread.java:163)

"de.anomic.crawler.CrawlQueues.remoteTriggeredCrawlJob" prio=10 tid=0x085e3400 nid=0x1746 waiting on condition [0x88c5f000..0x88c5fda0]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
        at java.lang.Thread.sleep(Native Method)
        at de.anomic.server.serverAbstractBusyThread.ratz(serverAbstractBusyThread.java:198)
        at de.anomic.server.serverAbstractBusyThread.run(serverAbstractBusyThread.java:163)

"de.anomic.plasma.plasmaSwitchboard.htEntryStoreJob" prio=10 tid=0x085e2800 nid=0x1745 waiting on condition [0x88cb0000..0x88cb0e20]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
        at java.lang.Thread.sleep(Native Method)
        at de.anomic.server.serverAbstractBusyThread.ratz(serverAbstractBusyThread.java:198)
        at de.anomic.server.serverAbstractBusyThread.run(serverAbstractBusyThread.java:163)

"de.anomic.plasma.plasmaSwitchboard.deQueueProcess" prio=10 tid=0x084b8400 nid=0x1744 runnable [0x88d01000..0x88d01ea0]
   java.lang.Thread.State: RUNNABLE
        at java.io.RandomAccessFile.readBytes(Native Method)
        at java.io.RandomAccessFile.read(RandomAccessFile.java:322)
        at de.anomic.kelondro.kelondroFileRA.read(kelondroFileRA.java:64)
        at de.anomic.kelondro.kelondroBufferedIOChunks.read(kelondroBufferedIOChunks.java:79)
        - locked <0x92221720> (a de.anomic.kelondro.kelondroFileRA)
        - locked <0x93b8e938> (a java.util.TreeMap)
        - locked <0x921f9e78> (a de.anomic.kelondro.kelondroBufferedIOChunks)
        at de.anomic.kelondro.kelondroAbstractIOChunks.readFully(kelondroAbstractIOChunks.java:54)
        - locked <0x921f9e78> (a de.anomic.kelondro.kelondroBufferedIOChunks)
        at de.anomic.kelondro.kelondroFullRecords$EcoNode.<init>(kelondroFullRecords.java:166)
        at de.anomic.kelondro.kelondroFixedWidthArray.get(kelondroFixedWidthArray.java:116)
        - locked <0x8c4bb0c8> (a de.anomic.kelondro.kelondroFixedWidthArray)
        at de.anomic.kelondro.kelondroCollectionIndex.getwithparams(kelondroCollectionIndex.java:747)
        - locked <0x8c4a6768> (a de.anomic.kelondro.kelondroCollectionIndex)
        at de.anomic.kelondro.kelondroCollectionIndex.getdelete(kelondroCollectionIndex.java:741)
        at de.anomic.kelondro.kelondroCollectionIndex$keycollectionIterator.next(kelondroCollectionIndex.java:817)
        at de.anomic.kelondro.kelondroCollectionIndex$keycollectionIterator.next(kelondroCollectionIndex.java:798)
        at de.anomic.index.indexCollectionRI$wordContainersIterator.next(indexCollectionRI.java:117)
        at de.anomic.index.indexCollectionRI$wordContainersIterator.next(indexCollectionRI.java:98)
        at de.anomic.kelondro.kelondroMergeIterator.nextb(kelondroMergeIterator.java:73)
        at de.anomic.kelondro.kelondroMergeIterator.next(kelondroMergeIterator.java:88)
        at de.anomic.kelondro.kelondroRotateIterator.next(kelondroRotateIterator.java:62)
        at de.anomic.plasma.plasmaWordIndex.indexContainerSet(plasmaWordIndex.java:767)
        - locked <0x8c4bcf68> (a de.anomic.plasma.plasmaWordIndex)
        at de.anomic.plasma.plasmaDHTChunk.selectTransferContainersResource(plasmaDHTChunk.java:193)
        at de.anomic.plasma.plasmaDHTChunk.selectTransferContainers(plasmaDHTChunk.java:180)
        at de.anomic.plasma.plasmaDHTChunk.<init>(plasmaDHTChunk.java:116)
        at de.anomic.plasma.plasmaSwitchboard.deQueueProcess(plasmaSwitchboard.java:1752)
        at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at de.anomic.server.serverInstantBusyThread.job(serverInstantBusyThread.java:96)
        at de.anomic.server.serverAbstractBusyThread.run(serverAbstractBusyThread.java:142)

"de.anomic.crawler.CrawlStacker.job" prio=10 tid=0x084b7800 nid=0x1743 waiting on condition [0x88d52000..0x88d52f20]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
        at java.lang.Thread.sleep(Native Method)
        at de.anomic.server.serverAbstractBusyThread.ratz(serverAbstractBusyThread.java:198)
        at de.anomic.server.serverAbstractBusyThread.run(serverAbstractBusyThread.java:163)

"de.anomic.plasma.plasmaSwitchboard.cleanupJob" prio=10 tid=0x08564800 nid=0x1742 waiting on condition [0x8907c000..0x8907cfa0]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
        at java.lang.Thread.sleep(Native Method)
        at de.anomic.server.serverAbstractBusyThread.ratz(serverAbstractBusyThread.java:198)
        at de.anomic.server.serverAbstractBusyThread.run(serverAbstractBusyThread.java:103)

"Timer-0" prio=10 tid=0x08119000 nid=0x1741 in Object.wait() [0x88da3000..0x88da4020]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0xa4bd7e38> (a java.util.TaskQueue)
        at java.util.TimerThread.mainLoop(Timer.java:509)
        - locked <0xa4bd7e38> (a java.util.TaskQueue)
        at java.util.TimerThread.run(Timer.java:462)

"parseDocument_pool-1-thread-8" prio=10 tid=0x085eb800 nid=0x173f waiting on condition [0x88df4000..0x88df50a0]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0xa4a9b348> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
        at de.anomic.server.serverAbstractBlockingThread.run(serverAbstractBlockingThread.java:63)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
        at java.util.concurrent.FutureTask.run(FutureTask.java:138)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
        at java.lang.Thread.run(Thread.java:619)

"parseDocument_pool-1-thread-7" prio=10 tid=0x085ea400 nid=0x173e waiting on condition [0x88e45000..0x88e46120]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0xa4a9b348> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
        at de.anomic.server.serverAbstractBlockingThread.run(serverAbstractBlockingThread.java:63)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
        at java.util.concurrent.FutureTask.run(FutureTask.java:138)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
        at java.lang.Thread.run(Thread.java:619)

"condenseDocument_pool-1-thread-6" prio=10 tid=0x085e9000 nid=0x173d waiting on condition [0x88e96000..0x88e96da0]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0xa4a9a838> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
        at de.anomic.server.serverAbstractBlockingThread.run(serverAbstractBlockingThread.java:63)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
        at java.util.concurrent.FutureTask.run(FutureTask.java:138)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
        at java.lang.Thread.run(Thread.java:619)

"condenseDocument_pool-1-thread-5" prio=10 tid=0x085e8000 nid=0x173c waiting on condition [0x88ee7000..0x88ee7e20]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0xa4a9a838> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
        at de.anomic.server.serverAbstractBlockingThread.run(serverAbstractBlockingThread.java:63)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
        at java.util.concurrent.FutureTask.run(FutureTask.java:138)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
        at java.lang.Thread.run(Thread.java:619)

"webStructureAnalysis_pool-1-thread-4" prio=10 tid=0x085e7c00 nid=0x173b waiting on condition [0x88f38000..0x88f38ea0]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0xa4a99d08> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
        at de.anomic.server.serverAbstractBlockingThread.run(serverAbstractBlockingThread.java:63)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
        at java.util.concurrent.FutureTask.run(FutureTask.java:138)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
        at java.lang.Thread.run(Thread.java:619)

"webStructureAnalysis_pool-1-thread-3" prio=10 tid=0x085e6800 nid=0x173a waiting on condition [0x88f89000..0x88f89f20]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0xa4a99d08> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
        at de.anomic.server.serverAbstractBlockingThread.run(serverAbstractBlockingThread.java:63)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
        at java.util.concurrent.FutureTask.run(FutureTask.java:138)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
        at java.lang.Thread.run(Thread.java:619)

"storeDocumentIndex_pool-1-thread-2" prio=10 tid=0x085e6000 nid=0x1739 waiting on condition [0x88fda000..0x88fdafa0]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0xa4a98c00> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
        at de.anomic.server.serverAbstractBlockingThread.run(serverAbstractBlockingThread.java:63)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
        at java.util.concurrent.FutureTask.run(FutureTask.java:138)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
        at java.lang.Thread.run(Thread.java:619)

"storeDocumentIndex_pool-1-thread-1" prio=10 tid=0x085e5c00 nid=0x1738 waiting on condition [0x890cd000..0x890ce020]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0xa4a98c00> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
        at de.anomic.server.serverAbstractBlockingThread.run(serverAbstractBlockingThread.java:63)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
        at java.util.concurrent.FutureTask.run(FutureTask.java:138)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
        at java.lang.Thread.run(Thread.java:619)

"Thread-9" prio=10 tid=0x083ff800 nid=0x1737 in Object.wait() [0x8911e000..0x8911f0a0]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0xa4a9beb8> (a de.anomic.crawler.CrawlStacker)
        at java.lang.Object.wait(Object.java:485)
        at de.anomic.crawler.CrawlStacker.run(CrawlStacker.java:131)
        - locked <0xa4a9beb8> (a de.anomic.crawler.CrawlStacker)

"MultiThreadedHttpConnectionManager cleanup" daemon prio=10 tid=0x08583400 nid=0x1727 in Object.wait() [0x891fc000..0x891fd120]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0xa46df470> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
        - locked <0xa46df470> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
        at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$ReferenceQueueThread.run(MultiThreadedHttpConnectionManager.java:1122)

"Thread-1" prio=10 tid=0x08197400 nid=0x16a5 waiting on condition [0x89358000..0x89358e20]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
        at java.lang.Thread.sleep(Native Method)
        at de.anomic.server.serverProfiling.run(serverProfiling.java:67)

"Low Memory Detector" daemon prio=10 tid=0x080de400 nid=0x16a1 runnable [0x00000000..0x00000000]
   java.lang.Thread.State: RUNNABLE

"CompilerThread1" daemon prio=10 tid=0x080dd000 nid=0x16a0 waiting on condition [0x00000000..0x894bf468]
   java.lang.Thread.State: RUNNABLE

"CompilerThread0" daemon prio=10 tid=0x080da800 nid=0x169f waiting on condition [0x00000000..0x895404e8]
   java.lang.Thread.State: RUNNABLE

"Signal Dispatcher" daemon prio=10 tid=0x080d9400 nid=0x169e waiting on condition [0x00000000..0x00000000]
   java.lang.Thread.State: RUNNABLE

"Surrogate Locker Thread (CMS)" daemon prio=10 tid=0x080d8000 nid=0x169d waiting on condition [0x00000000..0x895e330c]
   java.lang.Thread.State: RUNNABLE

"Finalizer" daemon prio=10 tid=0x080bc800 nid=0x169c in Object.wait() [0x89672000..0x89672da0]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x8c4a0ab0> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
        - locked <0x8c4a0ab0> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
        at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)

"Reference Handler" daemon prio=10 tid=0x080bb800 nid=0x169b in Object.wait() [0x896c3000..0x896c3e20]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x8c4a09a8> (a java.lang.ref.Reference$Lock)
        at java.lang.Object.wait(Object.java:485)
        at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
        - locked <0x8c4a09a8> (a java.lang.ref.Reference$Lock)

"main" prio=10 tid=0x08059800 nid=0x1698 in Object.wait() [0xb7e37000..0xb7e391f8]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0xb14c0140> (a java.lang.Class for org.jdesktop.jdic.tray.internal.impl.DisplayThread)
        at java.lang.Object.wait(Object.java:485)
        at org.jdesktop.jdic.tray.internal.impl.WinSystemTrayService.<clinit>(Unknown Source)
        - locked <0xb14c0140> (a java.lang.Class for org.jdesktop.jdic.tray.internal.impl.DisplayThread)
        at org.jdesktop.jdic.tray.internal.impl.ServiceManagerStub.getService(Unknown Source)
        at org.jdesktop.jdic.tray.internal.ServiceManager.getService(Unknown Source)
        at org.jdesktop.jdic.tray.SystemTray.<clinit>(Unknown Source)
        at de.anomic.yacy.yacyTray.<clinit>(yacyTray.java:57)
        at yacy.startup(yacy.java:348)
        at yacy.main(yacy.java:993)

"VM Thread" prio=10 tid=0x080b8400 nid=0x169a runnable

"Concurrent Mark-Sweep GC Thread" prio=10 tid=0x0806c000 nid=0x1699 runnable
"VM Periodic Task Thread" prio=10 tid=0x080dfc00 nid=0x16a2 waiting on condition

JNI global references: 836

Heap
def new generation   total 14784K, used 9987K [0x8b4a0000, 0x8c4a0000, 0x8c4a0000)
  eden space 13184K,  73% used [0x8b4a0000, 0x8be139d8, 0x8c180000)
  from space 1600K,  19% used [0x8c310000, 0x8c35d388, 0x8c4a0000)
  to   space 1600K,   0% used [0x8c180000, 0x8c180000, 0x8c310000)
concurrent mark-sweep generation total 598016K, used 402741K [0x8c4a0000, 0xb0ca0000, 0xb0ca0000)
concurrent-mark-sweep perm gen total 18688K, used 11224K [0xb0ca0000, 0xb1ee0000, 0xb4ca0000)
miTreD
 
Beiträge: 1241
Registriert: Mi Jun 27, 2007 11:35 am
Wohnort: /home

Re: v0.593 / yacy hängt

Beitragvon Burkhard » Mi Jul 23, 2008 6:47 am

Halborinda hat geschrieben:Anscheinend irritiert YaCy der Shutdown-Wunsch gar nicht...

Mein großer Peer, burkhardYACY, und mein kleiner Testpeer, yasuna, melden auch den Schutdown Wunsch, wollen aber nicht herunter fahren. Da hilft nur Killen. Die Schutdown Meldungen sind locker in die üblichen Betriebsmeldungen im Log eingestreut. Seit einigen Tagen/Versionen geht das so. Sonst arbeiten sie normal, ohne Hänger.
Beide zur Zeit v0.593/05012
Burkhard
 
Beiträge: 9
Registriert: Mo Okt 22, 2007 5:38 pm

Re: v0.593 / yacy hängt

Beitragvon DanielR » Mi Jul 23, 2008 5:41 pm

Ja, da ist ein Bug. Beim Start kommt YaCy nicht dazu den Shutdown-Handler zu registrieren. YaCy hat zwar schon das Webinterface und die ganzen Threads gestartet ist aber eigentlich noch in der Startphase. Deshalb wird er Shutdown-Wunsch ignoriert.

Er hängt an dieser Zeile der yacy.java
Code: Alles auswählen
                    // unlock yacyTray browser popup
                    yacyTray.lockBrowserPopup = false;

Das besondere: Hier ist nicht mal ein Objekt im Spiel. Nur die Klassen-Instanz. Also das Statische. Evtl. wird eine Klasse von der benötigten Library (jdic.jar) versucht zu laden? Leider gibt es keine Ausgaben im Log :/

Kann es sein, dass unter Windows der Shutdown funktioniert? Dort funktioniert der Tray ja angeblich :)

Nachtrag: Wenn ich das Tray rausnehme gehts. (Die Zeile oben und dass im close() von plasmaSwitchboard.java)
DanielR
 
Beiträge: 395
Registriert: Di Feb 12, 2008 2:22 pm

Re: v0.593 / yacy hängt

Beitragvon miTreD » Mi Jul 23, 2008 5:58 pm

DanielR hat geschrieben:Nachtrag: Wenn ich das Tray rausnehme gehts. (Die Zeile oben und dass im close() von plasmaSwitchboard.java)
Dann check das bitte ein. Lieber mal ein paar Tage Probleme mit dem Tray als ein nicht funktionierender Shutdown mit Inkonsistenzen in der DB und permanenter Index-Neuerstellung.
miTreD
 
Beiträge: 1241
Registriert: Mi Jun 27, 2007 11:35 am
Wohnort: /home

Re: v0.593 / yacy hängt

Beitragvon DanielR » Mi Jul 23, 2008 8:27 pm

ich überlass' das lieber jemandem der sich mit dem Tray auskennt ;)
DanielR
 
Beiträge: 395
Registriert: Di Feb 12, 2008 2:22 pm

Re: v0.593 / yacy hängt

Beitragvon Lotus » Do Jul 24, 2008 7:51 am

Danke für den Hinweis. Passiert mir nicht noch einmal. ;)
fix in r5014

Das Tray sollte auch unter Linux ans Laufen gebracht werden können. Die Libs sind enthalten. Mir ist es nur zu umständlich den Link-Error zu beheben da ich das in einer VM teste. Mit der aktuellen JDIC 0.9.5 gibt es dann unter Windows Probleme. :roll:
Die Tray Demo funktioniert bei mir unter Linux (Ubuntu): https://jdic.dev.java.net/ unten auf der Seite.
Lotus
 
Beiträge: 1699
Registriert: Mi Jun 27, 2007 3:33 pm
Wohnort: Hamburg

Re: v0.593 / yacy hängt

Beitragvon Halborinda » Fr Jul 25, 2008 3:58 pm

Mein eigentliches Problem war aber ein anderes...

YaCy braucht wohl 1 1/2 Stunden um den Server hochzufahren bei mir...
Ich werde jetzt Mal ein Update durchführen und das ganz noch einmal ausprobieren...
Den Server dann geregelt herunterfahren und dann erneut starten...

Mal sehen...

Grüße
Halborinda
 
Beiträge: 217
Registriert: Sa Jul 07, 2007 7:58 am

Re: v0.593 / yacy hängt - solved

Beitragvon Halborinda » Fr Jul 25, 2008 6:42 pm

Schaut gut aus...

Einmal starten, dann ein bisschen laufen lassen und herunterfahren...
Danach startete YaCy wieder normal...

Wie wäre es, wenn YaCy am Daten reparieren ist, dass es auch etwas ins Log dazu schreibt...
...so etwas wie... "Was ist denn das für ein Chaos hier?! Da muss ich wohl erst einmal aufräumen... Das wird dauern... *Seufz*" ;-)

Weil die 1 1/2 Stunden, da ging ich dann davon aus, dass er hängt...

Grüße
Halborinda
 
Beiträge: 217
Registriert: Sa Jul 07, 2007 7:58 am

Re: v0.593 / yacy hängt - solved

Beitragvon ChrisJ » Di Jul 29, 2008 10:36 pm

Halborinda hat geschrieben:Wie wäre es, wenn YaCy am Daten reparieren ist, dass es auch etwas ins Log dazu schreibt...
...so etwas wie... "Was ist denn das für ein Chaos hier?! Da muss ich wohl erst einmal aufräumen... Das wird dauern... *Seufz*" ;-)

Weil die 1 1/2 Stunden, da ging ich dann davon aus, dass er hängt...


Mensch, jetzt habe ich meinen Yacy halbwegs ans laufen gebracht und schon die ersten Problemen.

Ich vermute es sind die selben wie bei dir? Der Shutdown hat nicht richtig funktioniert bzw ich war wohl zu ungeduldig und hab einfach nach 1 min.. startYACY.sh ausgeführt.

Anschleißend war ich 40 Minuten weg und als ich wiederkomme Reagiert der Server nicht mehr auf meine Anfragen. Weder bei yacy.. und auch nicht mehr auf einen anderen Test-Ruby-on-Rails-Dienst der dort ebenfalls läuft. SSH Verbindung steht aber noch funktionierte einwandfrei und auch ein neustarten der Rails-Anwendung funktinierte sie einwandfrei.

Nur Yacy macht jetzt ärger. Wenn ich ihn neu starte (startYACY.sh) kommt die Meldung das ein dienst wohl noch läuft.. am Anfang. Später viele Meldungen dieser Art:

Code: Alles auswählen
2008/07/29 23:03:18 KELONDRO KELONDRO FAILURE ......... array has different chunkcount than index: index = 1, array = 2; the index has been auto-fixed


Und sowas hatte ich auch:

Code: Alles auswählen
W 2008/07/29 23:03:06 FILEHANDLER Unexpected error while processing query.
....
Reason:  java.io.IOException: chunked stream ended unexpectedly


Was kann ich machen damit sich mein Yacy wieder bewegt? Ich gebe zu die Hardware ist ein wenig veraltet und recht Fehleranfällig was Wärme angeht. Leider kann ich die Temperatur im Moment nicht groß runterschrauben. Der Rechner neigt z.B beim Kompilieren großer Datenmengen (Linux-Kernel) gerne zu Speicherzugriffsfehlern. Aber "Kleine Dinge". Machen normalerweise keine Probleme. Proxy-Router usw laufen dort anstandslos.

Doch jetzt hängt sich Yacy immer auf(CPU Auslastung = 0, Reagiert nicht auf Anfragen - Timeout), gleich (3-4 Minuten)nachdem ich ihn neu gestartet habe... irgendwelche Vorschläge?? Ach ja die Rails Anwendung geht noch und auch der Rechner Reagiert normal.. nur Yacy nicht.

Hmm ich starte jetzt erst mal den Rechner neu (dabei wird ja alles beendet) und vielleicht geht es nachher besser. Wieso finde ich eigentlich keinen Prozess via ps -ALL | grep yacy?

Änderung-Anmerkung:
Ich wurde sehr wohl mit Fehlermeldungen Bombadiert.. das du das nicht gemerkt hast ist mir schleierhaft. Gerade z.B:
Code: Alles auswählen
W 2008/07/30 00:43:29 KELONDRO USEDC inconsistency at startup:


Bis jetzt läuft alles gut, mal sehen wie lange... ah. jetzt bleibt der Log stehen. Ich lasse ihn mal eine weile an.
ChrisJ
 
Beiträge: 8
Registriert: So Jul 27, 2008 5:19 pm

Re: v0.593 / yacy hängt - solved

Beitragvon Lotus » Mi Jul 30, 2008 8:57 am

@ChrisJ:
mit den Fehlermeldungen kann YaCy eigentlich weiterlaufen (sind ja "nur" Warnungen ;)).

Zu deinem "Hänger-Problem" versuche einmal "-XX:+UseLargePages" aus der startYACY.sh zu entfernen. (evtl. gleiches Phänomen wie hier: viewtopic.php?p=9276#p9276)
Lotus
 
Beiträge: 1699
Registriert: Mi Jun 27, 2007 3:33 pm
Wohnort: Hamburg

Re: v0.593 / yacy hängt - solved

Beitragvon ChrisJ » Mi Jul 30, 2008 1:14 pm

Hmm der Speicher ist mir bestimmt nicht ausgegangen. Dieses System verkraftet im Moment wohl einfach kein Locales Crawlen. Ich werde jetzt einfach mal abwarten. Als Java-Argumente hab ich lediglich dies:

Code: Alles auswählen
JAVA_ARGS: -Xms500m -Xmx500m


Ich denke dem Rechner ist einfach zu heiß und Zickt rum. Gestern Nacht ist er ohne laufendem Crawler noch relativ stabil gelaufen. Jetzt friert er gleich 1 Min. nach dem Start ein.
ChrisJ
 
Beiträge: 8
Registriert: So Jul 27, 2008 5:19 pm


Zurück zu Fragen und Antworten

Wer ist online?

Mitglieder in diesem Forum: 0 Mitglieder und 1 Gast

cron