Heavy loading

Keine Scheu, hier darf alles gefragt und diskutiert werden. Das ist das Forum für YaCy-Anfänger. Hier kann man 'wo muss man klicken' fragen und sich über Grundlagen zur Suchmaschinentechnik unterhalten.
Forumsregeln
Hier werden Fragen beantwortet und wir versuchen die Probleme von YaCy-Newbies zu klären. Bitte beantwortete Fragen im YaCy-Wiki http://wiki.yacy.de dokumentieren!

Heavy loading

Beitragvon mass » Mo Mai 05, 2014 3:16 pm

hello
The second day yacy process CPU time by 70-90%
Greatly slows down the web interface.
Rebooting does not help yacy

Code: Alles auswählen
E 2014/05/05 18:16:32 UNCAUGHT-EXCEPTION Thread Lucene Merge Thread #10: java.lang.OutOfMemoryError: GC overhead limit exceeded
org.apache.lucene.index.MergePolicy$MergeException: java.lang.OutOfMemoryError: GC overhead limit exceeded
   at org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:545)
   at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:518)
Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded

org.apache.lucene.index.MergePolicy$MergeException: java.lang.OutOfMemoryError: GC overhead limit exceeded
   at org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:545)
   at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:518)
Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
I 2014/05/05 18:16:32 HostBrowser collected 0 urls for path http://planetmaestro.com/zenphoto/test/
E 2014/05/05 18:16:32 ConcurrentLog java.lang.OutOfMemoryError: GC overhead limit exceeded
org.apache.lucene.index.MergePolicy$MergeException: java.lang.OutOfMemoryError: GC overhead limit exceeded
   at org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:545)
   at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:518)
Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded

Here's more from http://localhost:8091/ViewLog_p.html
Code: Alles auswählen
D 2014/05/05 19:32:51 SEARCH dropped RWI: wrong contentdom = image, domain = text

D 2014/05/05 19:32:51 SEARCH dropped RWI: wrong contentdom = image, domain = text

D 2014/05/05 19:32:51 SEARCH dropped RWI: wrong contentdom = image, domain = text

D 2014/05/05 19:32:51 SEARCH dropped RWI: wrong contentdom = image, domain = text

D 2014/05/05 19:32:51 SEARCH dropped RWI: wrong contentdom = image, domain = text

D 2014/05/05 19:32:51 SEARCH dropped RWI: wrong contentdom = image, domain = text

D 2014/05/05 19:32:51 SEARCH dropped RWI: hash not in metadata

D 2014/05/05 19:32:51 SEARCH dropped RWI: wrong contentdom = image, domain = text

D 2014/05/05 19:32:51 SEARCH dropped RWI: hash not in metadata

D 2014/05/05 19:32:51 SEARCH dropped RWI: wrong contentdom = image, domain = text

D 2014/05/05 19:32:51 SEARCH dropped RWI: wrong contentdom = image, domain = text

D 2014/05/05 19:32:51 SEARCH dropped RWI: wrong contentdom = image, domain = text

D 2014/05/05 19:32:51 SEARCH dropped RWI: wrong contentdom = image, domain = text

D 2014/05/05 19:32:51 SEARCH dropped RWI: wrong contentdom = image, domain = text

D 2014/05/05 19:32:51 SEARCH dropped RWI: wrong contentdom = image, domain = app

D 2014/05/05 19:32:51 SEARCH dropped RWI: wrong contentdom = image, domain = text

D 2014/05/05 19:32:51 SEARCH dropped RWI: wrong contentdom = image, domain = text

D 2014/05/05 19:32:51 SEARCH dropped RWI: wrong contentdom = image, domain = app

D 2014/05/05 19:32:51 SEARCH dropped RWI: wrong contentdom = image, domain = text

D 2014/05/05 19:32:51 SEARCH dropped RWI: wrong contentdom = image, domain = text

D 2014/05/05 19:32:51 SEARCH dropped RWI: wrong contentdom = image, domain = text

D 2014/05/05 19:32:51 SEARCH dropped RWI: hash not in metadata

D 2014/05/05 19:32:51 SEARCH dropped RWI: hash not in metadata

D 2014/05/05 19:32:51 SEARCH dropped RWI: wrong contentdom = image, domain = text

D 2014/05/05 19:32:51 SEARCH dropped RWI: wrong contentdom = image, domain = text

D 2014/05/05 19:32:51 SEARCH dropped RWI: hash not in metadata

D 2014/05/05 19:32:51 SEARCH dropped RWI: wrong contentdom = image, domain = text

D 2014/05/05 19:32:51 SEARCH dropped RWI: wrong contentdom = image, domain = text

D 2014/05/05 19:32:51 SEARCH dropped RWI: wrong contentdom = image, domain = text

D 2014/05/05 19:32:51 SEARCH dropped RWI: wrong contentdom = image, domain = text

D 2014/05/05 19:32:51 SEARCH dropped RWI: wrong contentdom = image, domain = text

D 2014/05/05 19:32:51 SEARCH dropped RWI: hash not in metadata

D 2014/05/05 19:32:51 SEARCH dropped RWI: hash not in metadata
Dateianhänge
ddo16.png
ddo16.png (6.67 KiB) 2598-mal betrachtet
ddo15.png
ddo15.png (56.56 KiB) 2598-mal betrachtet
mass
 
Beiträge: 56
Registriert: Do Jun 14, 2012 9:46 pm

Re: Heavy loading

Beitragvon David » Mo Mai 05, 2014 4:41 pm

mass hat geschrieben:java.lang.OutOfMemoryError

It seems like you are running out of RAM. Your index is probably getting too big :-(
David
 
Beiträge: 170
Registriert: Di Mär 05, 2013 5:35 pm

Re: Heavy loading

Beitragvon mass » Mo Mai 05, 2014 4:59 pm

ie do not use the hard disk? And directly with RAM.
If so, how can I fix it?
mass
 
Beiträge: 56
Registriert: Do Jun 14, 2012 9:46 pm

Re: Heavy loading

Beitragvon David » Mo Mai 05, 2014 6:13 pm

mass hat geschrieben:I do not use the hard disk? And directly with RAM.

No. Some parts of the index need to be loaded in the RAM, or your search engine won't work. And if your index grows, it requires not only more hard disk space, but also more RAM.

mass hat geschrieben:how can I fix it?

Unfortunately, I know only these two solutions: Add more RAM to your computer, or make your index smaller by removing some parts:
http://localhost:8090/IndexDeletion_p.html
David
 
Beiträge: 170
Registriert: Di Mär 05, 2013 5:35 pm

Re: Heavy loading

Beitragvon mass » Mo Mai 05, 2014 7:28 pm

Interestingly, before this time (about a year - a half ago) index was more than 100 million and enough RAM.
Increased RAM 3GB. Did not help.
very strange
mass
 
Beiträge: 56
Registriert: Do Jun 14, 2012 9:46 pm

Re: Heavy loading

Beitragvon David » Mo Mai 05, 2014 8:06 pm

It needs more RAM, because the developers have added Solr, which makes yacy more powerful, but also more resource hungry.

http://blog.yacy-websuche.de/2012/11/08 ... plus-solr/
https://en.wikipedia.org/wiki/Solr
David
 
Beiträge: 170
Registriert: Di Mär 05, 2013 5:35 pm

Re: Heavy loading

Beitragvon mass » Di Mai 06, 2014 9:07 pm

David hat geschrieben:It needs more RAM, because the developers have added Solr, which makes yacy more powerful, but also more resource hungry.

http://blog.yacy-websuche.de/2012/11/08 ... plus-solr/
https://en.wikipedia.org/wiki/Solr

Thank you
Add more RAM to 2 GB
Earned.
A run indexer rss http://localhost:8091/Load_RSS_p.html could not.
Displays a blank page.
mass
 
Beiträge: 56
Registriert: Do Jun 14, 2012 9:46 pm

Re: Heavy loading

Beitragvon mass » Sa Mai 10, 2014 7:19 pm

hello

sometimes CPU 90%
and writes in the log
Code: Alles auswählen
CollectionConfiguration collection - postprocessed 800 from 1018332 documents; 0 docs/second; 45308 minutes remaining

it is necessary to wait so long? :)
mass
 
Beiträge: 56
Registriert: Do Jun 14, 2012 9:46 pm

Re: Heavy loading

Beitragvon gTSj » Fr Mai 16, 2014 7:07 pm

@mass: seems like you try to generate a very big index. Try to reduce the index size by indexing less websites.
gTSj
 
Beiträge: 21
Registriert: Mo Jan 27, 2014 10:49 pm


Zurück zu Hilfe für Einsteiger und Anwender

Wer ist online?

Mitglieder in diesem Forum: 0 Mitglieder und 2 Gäste

cron