1 of 2 identical machines is incredibly slow

Discussion in English language.
Forumsregeln
You can start and continue with posts in english language in all other forums as well, but if you are looking for a forum to start a discussion in english, this is the right choice.

1 of 2 identical machines is incredibly slow

Beitragvon xioc752 » Do Okt 23, 2014 7:57 am

HI, we have 2 engines on Ubuntu. Ubuntu 12.04.5 LTS
(We have not done the ' offered ' upgrade to 14.04.1 LTS as we lost all data the last time we attempted it)

New release '14.04.1 LTS' available.
Run 'do-release-upgrade' to upgrade to it.


Both are cloud installations. Each on a 30 GB block, 1 GB Ram, 1 VCPU.
We always have the latest Ubuntu & YaCy versions and we update them every day - whenever presented the opportunity in PuTTy.
The have ~30 million web edges, each.
They are both nearly constantly crawling - with a long list of scheduled RSS feeds to process, each.

They read each other, but only write locally.
Memory reserved for JVM 768 on both machines
"Online Caution Case indexer delay" Proxy, Local Search, Remote Search (not active) all - 5000

Survey the Problem:
A) Server 1 is fine. Fast, crawls quickly - at different speeds-sometimes very quickly! Boots reasonably quickly from 'cold start' including cold reboot of Ubuntu. Good 'warm reboot' times with Ubuntu running.
B) Server 2 unbelievably SLOW. Horribly Slow to boot up from a cold start (including cold boot of Ubuntu) or restart with Ubuntu running (take extra minutes, sometimes many), visibly much more slow to crawl...really miserably slowly, sometimes. Sometimes does not start form cold boot of Ubuntu+crawler. Uses its available memory quickly, hence the need to reboot ~ 2 X - 3 X in a 24 hour ' day '. Sometimes it stops & fails to respond if it doesn't get the most current Ubuntu updates (and YaCy updates. Then it will start. Without it, it is dead and won't re-start, apparently.)
C) Server 2 -blocks- every day and needs manual reboots from a 'cold' start to stay online, even slowly. Crawled data seems OK + saved, however.
D) Remote requests for information from the clouds go through Slow Server 2, presently. It often takes 1-2 minutes for a "searched phrase" in red to appear in the window at / Status.html
E) Users are frustrated because the server takes sometimes x minutes to respond, prompting multiple clicks on "Search"

Note: Boot up times for Ubuntu are comparable when measured by the time needed to get a web prompt 'available' message at the root address, without the port number for the crawler

Current specs:
Fast Machine:
Memory Usage
RAM used: 244.73 MB
RAM max: 742.44 MB
DISK used: (approx.) 18.27 GB
DISK free: 6.1 GB

Slow Machine:
Memory Usage
RAM used: 717.33 MB
RAM max: 742.44 MB
DISK used: (approx.) 15.61 GB
DISK free: 8.31 GB

Updates and upgrades to ubuntu make no change in this. Updates to YaCy also have no effect on the situation.

What we want to do:
1) Get Server 2 running at the same healthy speed as Server 1
2) Give priority to searchers on one of the machines, so that they get a result quickly.

Thanks so much for your advice! :D
xioc752
 
Beiträge: 68
Registriert: Mo Jul 28, 2014 5:01 pm

Re: 1 of 2 identical machines is incredibly slow

Beitragvon smokingwheels » Do Okt 23, 2014 11:34 am

Hi my cloud machine suffered today too.
Do you download the logs on the slow machine?
I got lots of
W 2014/10/23 09:38:28 org.eclipse.jetty.server.ServerConnector
java.io.IOException: Too many open files
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:241)
at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:377)
at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:500)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:610)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:539)
at java.lang.Thread.run(Thread.java:745)
and took forever to restart but then got
W 2014/10/23 09:39:33 ConcurrentLog java.lang.OutOfMemoryError: Requested array size exceeds VM limit

What size is you Swap file?
Not too sure but your VM seems too large mine is 700 mb and there is nothing left free in Debian.
In Debian you type free to see the stats of memory.

All seems to be ok now after a slow reboot.

or tell Erik_S to abandon his tool viewtopic.php?f=23&t=5302
smokingwheels
 
Beiträge: 102
Registriert: Sa Aug 31, 2013 7:16 am

Re: 1 of 2 identical machines is incredibly slow

Beitragvon xioc752 » Fr Okt 24, 2014 2:37 pm

HI & thanks for your thoughts.
I changed / lowered the JVM to 700 on both machines and restarted.
The slow machine is still horribly slow.
I cannot see the difference, though ...perhaps ... a little faster, not so sluggish. But only 'perhaps.'
Sometimes things are faster, sometimes much slower, but always the slow machine is visibly horribly slow.

Where is the swap file setting, please?
Is there anything else that can make the machine horrifically slow?
We do updates and upgrades to the clouds whenever they are available, though we are only in 12 Ubuntu, upgraded from 10.4
We discovered the rolling update to 14 is buggy.
Many thanks
xioc752
 
Beiträge: 68
Registriert: Mo Jul 28, 2014 5:01 pm

Re: 1 of 2 identical machines is incredibly slow

Beitragvon smokingwheels » Fr Okt 24, 2014 10:09 pm

xioc752 hat geschrieben:Where is the swap file setting, please?

See https://www.digitalocean.com/community/ ... untu-12-04

xioc752 hat geschrieben:Is there anything else that can make the machine horrifically slow?

Have you checked the queries.log to see how much work the peer is doing?
If it is no more or less than the good one I would consider a backup and create a 3rd machine and I think you restore the data directory to the new os with a fresh install of yacy on it. I would need someone to confirm how to do it though. In the past I have had problems with a VPS and I ended up destroying it and starting again.
How do you access your machines by IP or DNS eg http://smokingwheels.evils.in:8090 If it is DNS I would go to http://www.webpagetest.org/ and test both servers, go to the advanced settings and set 9 tests then look at the waterfall graphs and check the response time, on my server it sometimes takes 10-30 seconds to load the DNS redirect.

xioc752 hat geschrieben:We do updates and upgrades to the clouds whenever they are available, though we are only in 12 Ubuntu, upgraded from 10.4
We discovered the rolling update to 14 is buggy.
Many thanks

Its probably got to do with the swap file setting thats my guess

I tried a second VPS Debian system yesterday on a SATA and its Load time for the yacy search page was about 50% faster than my SSD but I will see how it is today. I mention this because you are running out of space on one of your servers.
smokingwheels
 
Beiträge: 102
Registriert: Sa Aug 31, 2013 7:16 am

Re: 1 of 2 identical machines is incredibly slow

Beitragvon xioc752 » Sa Okt 25, 2014 4:19 pm

Thank you very kindly for the detailed analysis, discussion & proposed testing routines.
I have to look at how to do all of that...whew...not easy.

Alternatively, but perennially,...
Our #1 problem ultimately is in taking the DATA folder and reinstalling elsewhere.
It is quite large, now.

But the worst part is huge risk of
both
1) losing the feeding trail of URLs & particularly RSS links that are programmed for daily updating.

2) Web edge collection accompanying the built in Solr files

Is there any way to copy that Process Scheduler History of Instructions INTACT with the settings?
Table_API_p.html

We cannot afford to lose the data and web edge cross references built up, now, inside ...
In the worst case, we would like to take the Solr save (and what else?) and match it with the Table_API_p.html instructions in a clean copy of YaCy.

Many thanks...
xioc752
 
Beiträge: 68
Registriert: Mo Jul 28, 2014 5:01 pm

Re: 1 of 2 identical machines is incredibly slow

Beitragvon smokingwheels » So Okt 26, 2014 9:20 am

xioc752 hat geschrieben:Our #1 problem ultimately is in taking the DATA folder and reinstalling elsewhere.
It is quite large, now.

Your VPS has a 30 GB drive, I think you will have to upgrade its space in a few months or less.

How long are you going to keep using YaCy for?

xioc752 hat geschrieben: But the worst part is huge risk of
both
1) losing the feeding trail of URLs & particularly RSS links that are programmed for daily updating.

2) Web edge collection accompanying the built in Solr files

Is there any way to copy that Process Scheduler History of Instructions INTACT with the settings?
Table_API_p.html

If you go to /Tables_p.html change to API change number to ALL then search you have a html page with all the info on.
Simply File Save As TXT and open with a txt editor where you can copy all the various URL's etc and put them in a new install. But also create a backup file of all the URLs etc

xioc752 hat geschrieben:We cannot afford to lose the data and web edge cross references built up, now, inside ...
In the worst case, we would like to take the Solr save (and what else?) and match it with the Table_API_p.html instructions in a clean copy of YaCy.

I only know if you take your server offline you can backup all the files, but SSH is too slow for this pursuance.
smokingwheels
 
Beiträge: 102
Registriert: Sa Aug 31, 2013 7:16 am

Re: 1 of 2 identical machines is incredibly slow

Beitragvon xioc752 » Mi Okt 29, 2014 7:45 pm

Hello,
well, we have planned to continue using YaCy. There are no plans or interest in going somewhere else as there are security functions in it we feel are important.
as you know, however, the update from yesterday has trashed both of our servers.
YaCy version 1.81/9116 causes a critical and complex failure documented by the error notices / logs at the other forum address.
viewtopic.php?f=23&t=5429

So now we need serious help to fix this.
Who knows how to fix, bypass, work around or whatever to repair this mess, please?
Both servers were effected immediately.
Both show the illogical peer notice.
interestingly, the key message never seen before in our use of this is
The peer must go online to get a peer address.

As they were never set up as peers, this is very strange.
These are Robinson servers that read each other but do not write to each other
...precisely to avoid the chance that one sick server will infect the other.
However, as we can see a bug in the update applied to both has caused the massive dual failure.
xioc752
 
Beiträge: 68
Registriert: Mo Jul 28, 2014 5:01 pm

Re: 1 of 2 identical machines is incredibly slow

Beitragvon Orbiter » Mi Nov 12, 2014 10:45 am

as you say that the second server already is slow at boot time
xioc752 hat geschrieben:Horribly Slow to boot up from a cold start (including cold boot of Ubuntu) or restart with Ubuntu running (take extra minutes, sometimes many)

this shows that there is no YaCy-specific problem on that second server. If this runs 'in a cloud' then possibly the second server is probably just hosted on a machine which has busy VM vom other users.

The "The peer must go online to get a peer address." is maybe caused my recent IPv6 changes, I believe its just a wrong message, not a real communication error. I am not monitoring any 'massive failure' due to recent release changes. Please report detailed observations which can be used to track down what you name 'massive failure'. From the information you gave so far there is no problem solving possible.
Orbiter
 
Beiträge: 5769
Registriert: Di Jun 26, 2007 10:58 pm
Wohnort: Frankfurt am Main

Re: 1 of 2 identical machines is incredibly slow

Beitragvon xioc752 » Sa Nov 22, 2014 4:50 pm

Thank you.
Conditions seem to have changed with the upgrades to YaCy; thanks.
Both machines have been upgraded to 1.81/9177 & latest Ubuntu

The slow machine is now showing
HTTP ERROR: 403

Problem accessing /Status.html. Reason:

proxy use not allowed (see Advanced Settings -> HTTP Networking -> Transparent Proxy; switched off).
Powered by Jetty://


Please advise the manual work around to fix this. i.e., what file to adjust manually.

We feel there may be more than 1 step required to restore system visibility in the web admin and online
Thanks very kindly

The faster machine which does display results but does not process RSS instructions shows a very curious address:
Address
Host: []:8090
peer address not assigned
Proxy
Transparent off URL off
Remote: not used


It also shows
The peer must go online to get a peer address.

We note that both the machines displayed this and neither crawled despite many RSS feeds programmed. Also manual crawl instructions were not processed.
There are no firewall changes in the clouds.

Thank you again for your help!
xioc752
 
Beiträge: 68
Registriert: Mo Jul 28, 2014 5:01 pm

Re: 1 of 2 identical machines is incredibly slow

Beitragvon xioc752 » Sa Nov 22, 2014 5:31 pm

Further note:
On the faster machine that is displaying results and admin pages, but does not search, the admin page ConfigPortal.html shows the following error results:

HTTP ERROR 500

Problem accessing /ConfigPortal.html. Reason:

Server Error
Caused by:

javax.servlet.ServletException: /usr/share/yacy/htroot/ConfigPortal.html
at net.yacy.http.servlets.YaCyDefaultServlet.handleTemplate(YaCyDefaultServlet.java:815)
at net.yacy.http.servlets.YaCyDefaultServlet.doGet(YaCyDefaultServlet.java:317)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:687)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:769)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:553)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1125)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1059)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
at net.yacy.http.CrashProtectionHandler.handle(CrashProtectionHandler.java:33)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
at org.eclipse.jetty.server.Server.handle(Server.java:497)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:248)
at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:610)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:539)
at java.lang.Thread.run(Thread.java:745)

YaCy 1.81 - powered by Jetty -

others seem to display normally. Browser page back is possible.
How do we repair this, please?

Thank you!
xioc752
 
Beiträge: 68
Registriert: Mo Jul 28, 2014 5:01 pm

Re: 1 of 2 identical machines is incredibly slow

Beitragvon xioc752 » So Nov 23, 2014 1:50 pm

The faster machine - the one with visible results & pages
now processes advanced crawler specific pages
but does not process RSS feeds, neither old nor attempted to be newly added
thanks for your expterise...
xioc752
 
Beiträge: 68
Registriert: Mo Jul 28, 2014 5:01 pm


Zurück zu English

Wer ist online?

Mitglieder in diesem Forum: 0 Mitglieder und 1 Gast

cron