Yacy won't re-start

Discussion in English language.
Forumsregeln
You can start and continue with posts in english language in all other forums as well, but if you are looking for a forum to start a discussion in english, this is the right choice.

Yacy won't re-start

Beitragvon eros » Fr Mai 12, 2017 2:50 pm

Hi,

I've installed Yacy on a rather powerful server (48 cores, 256 GB RAM, Ubuntu 14.04, Java 1.7).

It ran smoothly for about two days, and I started a large crawl of the Italian web (I used about 10.000 URLs as seeds, I'm working on a Corpus Linguistics project at the University of Bologna).

I launched the crawl yesterday afternoon, but this morning I realized that maybe the 16GB of RAM I had assigned to Yacy was too small, so I decided to stop the server, increase the amount of RAM in the configuration file and then start the service again).

Here's what I did:

- I stopped the server using the web interface
- I waited for a few minutes for the Java process to terminate gracefully
- I changed this setting in /etc/yacy/yacy.conf: javastart_Xmx=Xmx32768m (the original amount was 16384)
- I restarted the service using: service yacy start

Now Yacy won't start. I waited for 30+ minutes and then I manually killed the Java process and tried again but to no avail. From what I can see, the problem seems to be here:

Code: Alles auswählen
java.io.FileNotFoundException: /usr/share/yacy/DATA/INDEX/freeworld/QUEUES/CrawlerLimitStacks/toni.org-#gctLmQ.80/0003.stack (Too many open files)
   at java.io.FileInputStream.open(Native Method)
   at java.io.FileInputStream.<init>(FileInputStream.java:146)
   at net.yacy.kelondro.table.ChunkIterator.<init>(ChunkIterator.java:65)
   at net.yacy.kelondro.table.Table.<init>(Table.java:161)
   at net.yacy.kelondro.index.OnDemandOpenFileIndex.getIndex(OnDemandOpenFileIndex.java:61)
   at net.yacy.kelondro.index.OnDemandOpenFileIndex.size(OnDemandOpenFileIndex.java:153)
   at net.yacy.kelondro.index.BufferedObjectIndex.size(BufferedObjectIndex.java:152)
   at net.yacy.crawler.HostBalancer$1.run(HostBalancer.java:101)
W 2017/05/12 14:30:22 ConcurrentLog net.yacy.kelondro.util.kelondroException: /usr/share/yacy/DATA/INDEX/freeworld/QUEUES/CrawlerLimitStacks/toni.org-#gctLmQ.80/0003.stack (Too many open files)
net.yacy.kelondro.util.kelondroException: /usr/share/yacy/DATA/INDEX/freeworld/QUEUES/CrawlerLimitStacks/toni.org-#gctLmQ.80/0003.stack (Too many open files)
   at net.yacy.kelondro.table.Table.<init>(Table.java:228)
   at net.yacy.kelondro.index.OnDemandOpenFileIndex.getIndex(OnDemandOpenFileIndex.java:61)
   at net.yacy.kelondro.index.OnDemandOpenFileIndex.size(OnDemandOpenFileIndex.java:153)
   at net.yacy.kelondro.index.BufferedObjectIndex.size(BufferedObjectIndex.java:152)
   at net.yacy.crawler.HostBalancer$1.run(HostBalancer.java:101)


But I don't know how to fix it.

The directory /usr/share/yacy/DATA/INDEX/freeworld/QUEUES/CrawlerLimitStacks/ contains 180330 files, that seems like a lot, but it shouldn't be a problem on a EXT4 filesystem.

I tried changing the RAM value back to the initial setting but it didn't help. I tried moving the data directory but in that case Yacy won't start.

During one (just one!) of the various page refreshes I did, I got this error page on the web browser:

Code: Alles auswählen
HTTP ERROR 500

Problem accessing /. Reason:

    Server Error
Caused by:

javax.servlet.ServletException: /usr/share/yacy/htroot/index.html
   at net.yacy.http.servlets.YaCyDefaultServlet.handleTemplate(YaCyDefaultServlet.java:895)
   at net.yacy.http.servlets.YaCyDefaultServlet.doGet(YaCyDefaultServlet.java:312)
   at javax.servlet.http.HttpServlet.service(HttpServlet.java:687)
   at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
   at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:812)
   at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:587)
   at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
   at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:595)
   at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
   at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
   at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
   at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
   at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
   at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
   at org.eclipse.jetty.server.Dispatcher.forward(Dispatcher.java:191)
   at org.eclipse.jetty.server.Dispatcher.forward(Dispatcher.java:72)
   at net.yacy.http.servlets.YaCyDefaultServlet.doGet(YaCyDefaultServlet.java:349)
   at javax.servlet.http.HttpServlet.service(HttpServlet.java:687)
   at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
   at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:812)
   at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:587)
   at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
   at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:553)
   at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
   at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
   at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
   at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
   at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
   at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
   at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
   at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
   at net.yacy.http.CrashProtectionHandler.handle(CrashProtectionHandler.java:33)
   at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
   at org.eclipse.jetty.server.Server.handle(Server.java:499)
   at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311)
   at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
   at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)
   at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
   at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
   at java.lang.Thread.run(Thread.java:745)


Any suggestions?

Thanks in advance,
Eros
eros
 
Beiträge: 7
Registriert: Fr Mai 12, 2017 1:56 pm

Re: Yacy won't re-start

Beitragvon luc » Sa Mai 13, 2017 8:41 am

Hi eros,
the error you report suggests there might be a leak somewhere in YaCy (maybe some missing InputStream.close() or OuputStream.close() instructions) or a too low "file-max" setting on you machine (I guess you already checked "/proc/sys/fs/file-max" and ulimit, but checking "/proc/sys/fs/file-nr" when YaCy is running could also be interesting).

I am not sure this is directly related to your new memory setting. Maybe it was rather caused by the fact that you stopped the server while running a large crawl.

Which version do you use : 1.92/9000?

Did you try to remove part or all files from your "/usr/share/yacy/DATA/INDEX/freeworld/QUEUES/CrawlerLimitStacks" folder (after eventually backing them up for later re-insert)?

There is also a setting that might help you in yacy.conf : "crawler.onDemandLimit". Its default value is 1000, meaning that when more than 1000 hosts are in the crawler stack, each queue backing file will be open, read and closed each time it is needed rather than loading in memory only once. Maybe increasing this value could help you, moreover with the large memory available you have.

Edit : if you can check your logs around the "HTTP ERROR 500" log trace there may be additional details that could be aslo interesting.

Best regards
luc
 
Beiträge: 263
Registriert: Mi Aug 26, 2015 1:04 am

Re: Yacy won't re-start

Beitragvon eros » Sa Mai 13, 2017 8:28 pm

Thanks for your reply Luc.

Code: Alles auswählen
cat /proc/sys/fs/file-max
26251184

cat /proc/sys/fs/file-nr
1248   0   26251184

ulimit
unlimited


it looks like the crawl was within the limits. In any case I deleted the "freeworld" directory in the index and I was able to start yacy. I then restarted the crawl from scratch, but I'm afraid it will happen again in case yacy crashes or if I need to reboot the machine. I'll try and terminate (or at least pause) the crawl before shutting down yacy, but what worries me is that the whole application doesn't seem to be very robust, if a system crash can compromise days of crawling.

Also: does the crawl ever really end on it's own? I suppose it could go on potentially forever, do I need to stop it manually after a while?

I'll experiment some more.
eros
 
Beiträge: 7
Registriert: Fr Mai 12, 2017 1:56 pm

Re: Yacy won't re-start

Beitragvon luc » So Mai 14, 2017 9:52 am

For sure it is very annoying to loose your work. At least if this would happen again, you should try keeping or backing up your /freeworld/SEGMENTS/ folder which holds the data you already indexed in your local Solr and RWI indexes.

The main parameter to control the end of a crawl is the "Crawling depth", i.e. the number of links followed in the webgraph starting from your crawl starting point. So if you set a too high value (over 8 according to Yacy help), it is likely you are indeed trying to crawl the whole Internet, and you should certainly stop it manually after some time.
luc
 
Beiträge: 263
Registriert: Mi Aug 26, 2015 1:04 am

Re: Yacy won't re-start

Beitragvon eros » So Mai 14, 2017 10:50 am

Yup, it happened again.

I let the crawler run for about 14 hours, then I paused it before shutting down the server. It didn't help and I got the same error message:

Code: Alles auswählen
java.io.FileNotFoundException: /usr/share/yacy/DATA/INDEX/freeworld/QUEUES/CrawlerLimitStacks/toni.org-#gctLmQ.80/0003.stack (Too many open files)


I moved the /INDEX/freeworld/QUEUES/ directory: the system started in no time and the index seems to be safe.

BTW, this is the size of QUEUES:

Code: Alles auswählen
root@amelia:/storage# ls -R QUEUES/| wc -l
536442


I'll try increasing the "crawler.onDemandLimit" value to 50000, any suggestions on suitable values? Currently I assigned 64 GB of RAM to Yacy, but I could bump it up to 96 GB.

I'm using the default Crawling depth of 3 for now, so that shouldn't be the problem. I don't need to restart the server that often, but I'm experimenting to see if Yacy is a viable solution to the problems I'm having with my research project, so I need to know that I can rely on it.

Unfortunately, most of the documentation is is German and I don't speak German, so I need to do a bit of trial and error and use Google Translate... ;)

Thanks for your help Luc!
eros
 
Beiträge: 7
Registriert: Fr Mai 12, 2017 1:56 pm

Re: Yacy won't re-start

Beitragvon luc » Mo Mai 15, 2017 6:15 am

I'll try increasing the "crawler.onDemandLimit" value to 50000, any suggestions on suitable values?

With the amount of RAM you have I guess you can even increase this value, but that's only a supposition : personally I always run YaCy on mid-range desktop machines or on low-end Virtual Machines so I am not really experienced with large scale crawls on high performance servers...
But I can say that some months ago I ran successfully some crawls with 100 000 links by crawl start file, needing some days to end, but working rather well on a machine with only 2GB RAM. I have to admit that I didn't try to restart YaCy while these crawls where running. And I was not aware that so many files could be created in the QUEUES folder... It looks like there is definitely something to do, at least at start to avoid exhausting the file descriptors made available by the OS.
luc
 
Beiträge: 263
Registriert: Mi Aug 26, 2015 1:04 am

Re: Yacy won't re-start

Beitragvon eros » Mo Mai 15, 2017 1:48 pm

Increasing "crawler.onDemandLimit" to 50000 didn't work: I started getting "Too many open files" error in the web frontend, I decreased it to 5000 and again I got errors, so I reverted the value to 1000 since I figured that probably that wasn't the problem anyway.

I tried terminating the crawler before restarting the service and this time it worked. I then started a new crawl using the same seed URLs but, after 24 hours no new pages had been indexed, so I removed the index and I started over. Now I'm going to let the crawler run for a few days.

I noticed that Yacy uses *a lot* of memory: I assigned 96 GB to stay on the safe side, after 24 hours it was running steadily on 20 GB and now (after a restart and 3 hours of crawling) it's already using 19 GB. I guess it's normal for Java to use large amounts of memory when it's available, I just wanted to let you know my experience.

Another question: I installed Yacy on my laptop too, and I tried submitting the same query on the server and on the laptop. The results are different (I get less results on the laptop) even though theoretically the server's index should be reachable since it's running in "senior mode" (i.e. port 8090 is open on the firewall and the server claims to be "senior"). Is that a behaviour to be expected?
eros
 
Beiträge: 7
Registriert: Fr Mai 12, 2017 1:56 pm

Re: Yacy won't re-start

Beitragvon luc » Mo Mai 15, 2017 6:16 pm

Thanks for sharing your experience. It looks like there is really something to dig regarding this problem with file descriptors... I will try to reproduce your scenario when having some time.

I guess it's normal for Java to use large amounts of memory when it's available, I just wanted to let you know my experience.

For sure Java lets more easily fill the available memory when using some structures without limitations. But in the end to my mind this really depends on how the application code is organized an is not so much related to Java. In YaCy there are many places where controls are performed on the available memory, so it is not a surprise to me that it uses much when much is available... which doesn't mean that YaCy necessarily needs so large amounts of memory to run fine.

I installed Yacy on my laptop too, and I tried submitting the same query on the server and on the laptop. The results are different

Yes, in peer-to-peer mode this may sound a bit surprising but it is the expected behavior, at least with the current YaCy architecture. When performing a search in p2p mode, some peers among the network are selected to be requested and to aggregate results from, and the selection rules even include some Randomness. Thus we can not obtain a deterministic behavior (like in Solr Cloud) in this p2p mode because each node has its own index and its own blacklisting and crawling rules, and is not supposed to obey to some master(s) node rules. The index distribution over the nodes accepting to receive remote index parts (DHT-in) is made to homogenize a bit the whole distributed index, but in the end the behavior is not deterministic... but after a long running time of your two YaCy peers I guess you should obtain more similar results.
I hope this answer a bit to your question.
luc
 
Beiträge: 263
Registriert: Mi Aug 26, 2015 1:04 am

Re: Yacy won't re-start

Beitragvon luc » Fr Mai 19, 2017 6:19 am

Hello eros, did you keep a copy of the full log file containing the "(Too many open files)" error? If you would like to share it it may be helpful for a deeper analysis of the issue.

Indeed I have run again some tests with large crawl queues (around 100000 to 220000 files in the QUEUES folder), on a Debian Jessie with its file-max system setting value default set at 404027. But until now, after many stops and restarts of YaCy, including change of the memory settings, I did not reproduce the error you had.

I was running with at max 2GB RAM memory dedicated to YaCy on a domestic DSL connection, so not really in the same conditions as you. But if you wish, I am still interested to have a look at your full log trace before the error to find some clue about what was going wrong.
luc
 
Beiträge: 263
Registriert: Mi Aug 26, 2015 1:04 am

Re: Yacy won't re-start

Beitragvon eros » Mo Mai 22, 2017 11:21 am

Hi Luc,

unfortunately I don't have the logs. I suspect the problem might be tied to the version of Java I was using (the server had Java 1.7)

So I tried running 2 crawls on a regular PC (Ubuntu 16.04.2, 4 GB RAM, Java 1.8.0_131) and I noticed that:

1. the crawl is running much faster
2. I didn't have any problems restarting the server

Caveats: for these new crawls I started from a single URL https://it.wikipedia.org/wiki/Portale:Portali for the first and https://en.wikipedia.org/wiki/Portal:Contents/Indices for the second. Both were set with a crawling depth of 6, they are still running (and much faster than the crawls on the server ever did) and have indexed about 1.1 million pages from wikipedia.

Now I upgraded Java to version Java 1.8.0_131 on the server (I obviously stopped the crawls before restarting Yacy) and it started without incident. The problem now is that I cannot export the large server's index to XML, whenever I try (using the default settings) I get this error message:

Code: Alles auswählen
HTTP ERROR 500

Problem accessing /IndexExport_p.html. Reason:

    Server Error
Caused by:

javax.servlet.ServletException: /usr/share/yacy/htroot/IndexExport_p.html
   at net.yacy.http.servlets.YaCyDefaultServlet.handleTemplate(YaCyDefaultServlet.java:895)
   at net.yacy.http.servlets.YaCyDefaultServlet.doGet(YaCyDefaultServlet.java:312)
   at net.yacy.http.servlets.YaCyDefaultServlet.doPost(YaCyDefaultServlet.java:374)
   at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
   at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
   at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:812)
   at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:587)
   at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
   at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:542)
   at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
   at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
   at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
   at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
   at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
   at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
   at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
   at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
   at net.yacy.http.CrashProtectionHandler.handle(CrashProtectionHandler.java:33)
   at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
   at org.eclipse.jetty.server.Server.handle(Server.java:499)
   at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311)
   at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
   at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)
   at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
   at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
   at java.lang.Thread.run(Thread.java:748)

YaCy 1.92 - powered by Jetty -


Any ideas?
eros
 
Beiträge: 7
Registriert: Fr Mai 12, 2017 1:56 pm

Re: Yacy won't re-start

Beitragvon eros » Mo Mai 22, 2017 11:33 am

Update, these are the errors I get on the log file:

Code: Alles auswählen
I 2017/05/22 12:30:04 Fulltext HOT DUMP dump path = /usr/share/yacy/DATA/ARCHIVE
W 2017/05/22 12:30:04 ConcurrentLog java.lang.reflect.InvocationTargetException
java.lang.reflect.InvocationTargetException
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   at java.lang.reflect.Method.invoke(Method.java:498)
   at net.yacy.http.servlets.YaCyDefaultServlet.invokeServlet(YaCyDefaultServlet.java:670)
   at net.yacy.http.servlets.YaCyDefaultServlet.handleTemplate(YaCyDefaultServlet.java:881)
   at net.yacy.http.servlets.YaCyDefaultServlet.doGet(YaCyDefaultServlet.java:312)
   at net.yacy.http.servlets.YaCyDefaultServlet.doPost(YaCyDefaultServlet.java:374)
   at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
   at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
   at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:812)
   at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:587)
   at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
   at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:542)
   at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
   at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
   at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
   at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
   at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
   at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
   at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
   at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
   at net.yacy.http.CrashProtectionHandler.handle(CrashProtectionHandler.java:33)
   at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
   at org.eclipse.jetty.server.Server.handle(Server.java:499)
   at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311)
   at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
   at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)
   at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
   at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
   at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NullPointerException
W 2017/05/22 12:30:04 org.eclipse.jetty.servlet.ServletHandler
javax.servlet.ServletException: /usr/share/yacy/htroot/IndexExport_p.html
   at net.yacy.http.servlets.YaCyDefaultServlet.handleTemplate(YaCyDefaultServlet.java:895)
   at net.yacy.http.servlets.YaCyDefaultServlet.doGet(YaCyDefaultServlet.java:312)
   at net.yacy.http.servlets.YaCyDefaultServlet.doPost(YaCyDefaultServlet.java:374)
   at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
   at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
   at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:812)
   at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:587)
   at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
   at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:542)
   at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
   at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
   at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
   at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
   at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
   at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
   at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
   at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
   at net.yacy.http.CrashProtectionHandler.handle(CrashProtectionHandler.java:33)
   at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
   at org.eclipse.jetty.server.Server.handle(Server.java:499)
   at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311)
   at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
   at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)
   at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
   at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
   at java.lang.Thread.run(Thread.java:748)
I 2017/05/22 12:30:08 SWITCHBOARD dhtTransferJob: no selection, too many entries in transmission buffer: 17
I 2017/05/22 12:30:08 SWITCHBOARD dhtTransferJob: result from dequeueing: true


I see a NullPointerException, which is never good...
eros
 
Beiträge: 7
Registriert: Fr Mai 12, 2017 1:56 pm

Re: Yacy won't re-start

Beitragvon luc » Di Mai 23, 2017 6:50 am

Hi eros,
thanks for your feedback, this is interesting to know that the upgrade to Java 1.8 seems to solve your initial problem.

Regarding the error on index export, I recently fixed some issues with this feature notably a NullPointerException case (see commit e5858bc on the GitHub repository). So if you want to have a try... you just have to build and install your own deb package from latest source. Not very difficult, and you will also benefit from all the other fixes and improvements made since release 1.92/9000.
luc
 
Beiträge: 263
Registriert: Mi Aug 26, 2015 1:04 am

Re: Yacy won't re-start

Beitragvon eros » Di Mai 23, 2017 3:31 pm

I tried building a deb package but apparently something went wrong, do I need to use Java 7 to compile it?

Code: Alles auswählen
Buildfile: /home/eros/yacy_search_server/build.xml

buildGitRevTask:
   [delete] Deleting: /home/eros/yacy_search_server/libbuild/GitRevTask.jar
      [jar] Building jar: /home/eros/yacy_search_server/libbuild/GitRevTask.jar

determineGitRevision:
   [gitRev] SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
   [gitRev] SLF4J: Defaulting to no-operation (NOP) logger implementation
   [gitRev] SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.

readBuildProperties:

init:
     [echo] YaCy Branch:
     [echo] YaCy Version number: 1.92
     [echo] YaCy Release number: 9216
   [delete] Deleting: /home/eros/yacy_search_server/classes/net/yacy/peers/operation/yacyBuildProperties.java
     [copy] Copying 1 file to /home/eros/yacy_search_server/classes/net/yacy/peers/operation
     [copy] Copying 1 file to /home/eros/yacy_search_server/classes

deb:
     [exec] dpkg-buildpackage: source package yacy
     [exec] dpkg-buildpackage: source version 1.92.9216
     [exec] dpkg-buildpackage: source distribution unstable
     [exec] dpkg-buildpackage: source changed by Michael Peter Christen <mc@yacy.net>
     [exec]  dpkg-source --before-build yacy_search_serverdpkg-buildpackage: host architecture amd64
     [exec]
     [exec] dpkg-checkbuilddeps: error: Unmet build dependencies: openjdk-7-jdk debhelper (>= 5) m4
     [exec] dpkg-buildpackage: warning: build dependencies/conflicts unsatisfied; aborting
     [exec] dpkg-buildpackage: warning: (Use -d flag to override.)
     [exec] Result: 3

BUILD SUCCESSFUL
eros
 
Beiträge: 7
Registriert: Fr Mai 12, 2017 1:56 pm


Zurück zu English

Wer ist online?

Mitglieder in diesem Forum: 0 Mitglieder und 2 Gäste