YaCy version 1.81/9116 knocks servers off Adressing

Discussion in English language.
Forumsregeln
You can start and continue with posts in english language in all other forums as well, but if you are looking for a forum to start a discussion in english, this is the right choice.

YaCy version 1.81/9116 knocks servers off Adressing

Beitragvon xioc752 » Di Okt 28, 2014 7:31 pm

This afternoon we upgraded the latest patch YaCy version 1.81/9116 to both systems we have.

The result is that both of our Robinson mode servers are now reporting

The peer must go online to get a peer address.


They are Robinson servers that read each other's DBs but do not write to each other.

Everything concerning external crawling and indexing and processing of our RSS links that are programmed into our systems has now stopped.
The DBs can be read, but no growth is possible.
How do we fix this please?
Thank you
xioc752
 
Beiträge: 68
Registriert: Mo Jul 28, 2014 5:01 pm

AdditionalRe:YaCy version 1.81/9116 knocks servers off Adres

Beitragvon xioc752 » Di Okt 28, 2014 7:42 pm

ADDITIONALLY

The update caused both servers to generate a major fault at /ConfigPortal.html
This is a disaster.
Our entire environment and data on both systems is now at risk.
How can we roll back safely, please? Thank you.

HTTP ERROR 500

Problem accessing /ConfigPortal.html. Reason:

Server Error
Caused by:

javax.servlet.ServletException: /usr/share/yacy/htroot/ConfigPortal.html
at net.yacy.http.servlets.YaCyDefaultServlet.handleTemplate(YaCyDefaultServlet.java:815)
at net.yacy.http.servlets.YaCyDefaultServlet.doGet(YaCyDefaultServlet.java:317)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:687)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:769)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:553)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1125)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1059)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
at net.yacy.http.CrashProtectionHandler.handle(CrashProtectionHandler.java:33)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
at org.eclipse.jetty.server.Server.handle(Server.java:497)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:248)
at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:610)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:539)
at java.lang.Thread.run(Thread.java:745)

YaCy 1.81 - powered by Jetty -
xioc752
 
Beiträge: 68
Registriert: Mo Jul 28, 2014 5:01 pm

Re: YaCy version 1.81/9116 knocks servers off Adressing

Beitragvon xioc752 » Di Okt 28, 2014 8:03 pm

ALSO...
DATA doesn't show in the search results window
just the generic framework at the top of the page.
How Do We ROLL THIS BACK?
This is a critical state.

...and on Both Servers....which operate independently of each other!
xioc752
 
Beiträge: 68
Registriert: Mo Jul 28, 2014 5:01 pm

Re: YaCy version 1.81/9116 knocks servers off Adressing

Beitragvon xioc752 » Di Okt 28, 2014 8:09 pm

System Administration is Offline/trashed, too

/Tables_p.html?table=robots&count=100

HTTP ERROR 500

Problem accessing /Tables_p.html. Reason:

Server Error
Caused by:

javax.servlet.ServletException: /usr/share/yacy/htroot/Tables_p.html
at net.yacy.http.servlets.YaCyDefaultServlet.handleTemplate(YaCyDefaultServlet.java:815)
at net.yacy.http.servlets.YaCyDefaultServlet.doGet(YaCyDefaultServlet.java:317)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:687)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:769)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:542)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1125)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1059)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
at net.yacy.http.CrashProtectionHandler.handle(CrashProtectionHandler.java:33)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
at org.eclipse.jetty.server.Server.handle(Server.java:497)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:248)
at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:610)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:539)
at java.lang.Thread.run(Thread.java:745)

YaCy 1.81 - powered by Jetty -
xioc752
 
Beiträge: 68
Registriert: Mo Jul 28, 2014 5:01 pm

Re: YaCy version 1.81/9116 knocks servers off Adressing

Beitragvon smokingwheels » Mi Okt 29, 2014 10:17 am

You would need to unpack a previous release to get back a version but I do not have the experience to tell you how.

Could you download the log files and report it here http://mantis.tokeek.de/my_view_page.php.

Thanks for the tip I wont update mine yet unless I have a full recent Backup.

If /CrawlProfileEditor_p.html still works then save page as TXT file to backup all your crawling stuff incase there is no return.
smokingwheels
 
Beiträge: 107
Registriert: Sa Aug 31, 2013 7:16 am

Re: YaCy version 1.81/9116 knocks servers off Adressing

Beitragvon xioc752 » Mi Okt 29, 2014 7:04 pm

This is very bad.
I have posted a bug report at http://mantis.tokeek.de/view.php?id=492
You would need to unpack a previous release to get back a version but I do not have the experience to tell you how.

Who knows how to do this, please? This is critical failure, & it is not a hobby project.

If /CrawlProfileEditor_p.html still works then save page as TXT file to backup all your crawling stuff incase there is no return.

Yes, it works, but there are 700+ entries on one server alone.
What we need is to recover an instruction set, not oceans of lines to copy & paste.
There are weeks of continuous crawls, 24/7 on each server.
How to do this, please?
Many thanks
xioc752
 
Beiträge: 68
Registriert: Mo Jul 28, 2014 5:01 pm

Re: YaCy version 1.81/9116 knocks servers off Adressing

Beitragvon smokingwheels » Do Okt 30, 2014 5:16 am

I have posted a bug report at http://mantis.tokeek.de/view.php?id=492

I seen your report but there is no system logs on debian there here /var/lib/yacy/LOG you should use 7zip or similar to compress them and upload it to the bug report.
This is critical failure, & it is not a hobby project.

So a production servers. No backup. SSD is going to fill up. If the Cloud drive fails? Spells disaster to me.
I'm not being nasty just trying to educate you.

Yes, it works, but there are 700+ entries on one server alone.
What we need is to recover an instruction set, not oceans of lines to copy & paste.
There are weeks of continuous crawls, 24/7 on each server.


Try saving this page as a TXT file to backup your API Table.
/Tables_p.html?table=api&count=all&search=
Copy & paste oceans of lines, well my typist she can run at 500 WPM.
smokingwheels
 
Beiträge: 107
Registriert: Sa Aug 31, 2013 7:16 am

Re: YaCy version 1.81/9116 knocks servers off Adressing

Beitragvon xioc752 » Do Okt 30, 2014 8:54 am

Thanks for your comments & suggestions.
I will find those logs... Thanks
Well, the background is 'welcome to the world of big cloud providers always blaming the other guy' ...especially when there is a hint of developer work going on! = The "perfect excuse."
We will try your tricks. Thanks.
By the way... Lack of space has been the recent problem for creating backups...+ not everyone has gargantuan institutional backbones for flinging around multi-GB folders in less than lightyears... and with now substantial sized data folders, this is not a funny problem. :(

So, back to the Basic Question please:
FINDING The Best Practices based work around, patch or rollback...

How is it that this lack of displaying the search results in a standard off the shelf window - using one of the display pages provided, and several html pages of errors inside admin (which does display) results in no search results being shown to searchers (only the top of page framework) and how is that our fault on -2- simultaneous separate, not-interconnected, cloud servers - each in in Robinson mode that got the same ~very slightly~ flaky update within minutes of each other (that clearly doesn't 'work' everywhere)?
So at least in clouds with this version of Ubuntu, this was a "repeatable failure."
Call this a dumb sincere question, please?
We're not angry, just needing seasoned creative help to get past what clearly is "un petit bete noire," as the French say.
This is all work in development, on all sides, and it takes inter-team creativity, across & between different teams.
Time to think :arrow: past the error state, please...
Many thanks!
:D
xioc752
 
Beiträge: 68
Registriert: Mo Jul 28, 2014 5:01 pm

Re: YaCy version 1.81/9116 knocks servers off Adressing

Beitragvon Orbiter » Do Okt 30, 2014 10:52 am

a migration back to a previous version of YaCy is exactly the same as upgrading: just open /ConfigUpdate_p.html, select an older version and install that.
Orbiter
 
Beiträge: 5778
Registriert: Di Jun 26, 2007 10:58 pm
Wohnort: Frankfurt am Main

Re: YaCy version 1.81/9116 knocks servers off Adressing

Beitragvon xioc752 » Do Okt 30, 2014 9:03 pm

HI and thank you, but unfortunately we installed these by hand in the 2 clouds, so we have this displaying.

System Update
You installed YaCy with a package manager. To update YaCy, use the package manager:
manual update:
apt-get update && apt-get install yacy
automatic update: add the following line to /etc/crontab
0 6 * * * root apt-get update && apt-get -y --force-yes install yacy


We have since upgraded the two servers to version 1.81/9123 - on both machines.
Crawling resumes on the slow server (the other is out of space, so it is paused)

However, although search requests look like they process, NO Data Results show...
We are using one of the provided templates, at the moment.
Only the header lines of the results page show with the search term in the window

Can we have an upgrade/update to fix that, please?
Many Thanks!
xioc752
 
Beiträge: 68
Registriert: Mo Jul 28, 2014 5:01 pm

Re: YaCy version 1.81/9116 knocks servers off Adressing

Beitragvon smokingwheels » Fr Okt 31, 2014 4:58 am

If /CrawlProfileEditor_p.html still works then save page as TXT

I tell a lie.
In order to save all URL's you need a backup from the API table you need this to work.
/Tables_p.html?table=api&count=all&search=
Not sure what else is needed for a complete fresh rebuild though.
I may be able to help you rebuild a new yacy server if you can get the API info. PM me.

On the http error 500 try clicking back then the link again.

How many crawls per hour are you loading the yacy peer up with?

How often do you crawl each one?

From what you said before, over 700 items you crawl in the list, pushed to the max that's like 4000+ crawls/hour on a single CPU. Correct me if I am wrong.

The VPS provider I use has a Backup service they will keep 2 backups on there system and it runs once a day.
smokingwheels
 
Beiträge: 107
Registriert: Sa Aug 31, 2013 7:16 am

Re: YaCy version 1.81/9116 knocks servers off Adressing

Beitragvon xioc752 » Fr Okt 31, 2014 12:07 pm

Thank you for your thoughtful and helpful comments and suggestions.
We have updated the ticket at http://mantis.tokeek.de/view.php?id=492#bugnotes

Summary;
We have upgraded both servers to today's available ver 1.81/9130
While both are apparently stable,
>>> No change in observed systems behavior.
>>>NO SEARCH RESULTS DISPLAY when searches are made. Only the page header details including the search button, when using a template provided with the kit.
>>> errors of 500 type lists are still showing, as detailed previously, in administration pages
This condition persists despite that we have applied -2- updates since the incident began on both servers, as a result of applying ( 1.81/9116 ), sequentially to each cloud in the scope of ~10 min.

Good idea...we know the clicking back trick and have tried it, but it doesn't give us any joy, unfortunately. But thanks!

The some 700+ mostly custom RSS crawls are divided between the presently two servers.
This would tend to indicate a somewhat lower loading than you suggest, however, due to some things, among which the loading in "rush hours," it can become heavy and at those times, the engines are visibly running very quickly - depending upon the remote sources being addressed.
While the crawls are typically once ( 1X ) a day, there are some important characteristics which define system loading.
Each RSS can include a long secondary list of hundreds of external addresses of URLs that have been pre-retrieved by another toolset, and then those results must be crawled and locally indexed. The list of each one can be over 1000 entries long, upon occasion ( i.e., frequently ).
There are also time of day issues. Typically evening and late night until very early morning (in the night) are especially busy times due to the availability of "fresh data" to harvest. Then a few hours later in the mornings, for another large but shorter block of high traffic times.

Thanks for your thoughts. I will pm.
Our #1 issue now is to remove the persistent condition where the The Key Error Message still displayed is:

on page: / Status.html?noforward=
"
The peer must go online to get a peer address.
"

Note: These was never in peer mode. These are stand alone Robinson Servers that read each other to fulfill search requests, but do not write to each other.

Both servers were knocked offline, disconnected by the version update applied, at the start of this ticket ( 1.81/9116 ). They were NOT Reconnected by the -2- two subsequent updates we have applied since our first report.
Therefore it appears a manual re-connection is needed. What & Where and what is the correct Syntax are the questions to answer, please. We suspect that restoring this manually may have a beneficent effect on the whole environment in each cloud, but it is only a theory.
Thank You Very, Very Kindly, Everybody...
xioc752
 
Beiträge: 68
Registriert: Mo Jul 28, 2014 5:01 pm

Re: YaCy version 1.81/9116 knocks servers off Adressing

Beitragvon xioc752 » Sa Nov 01, 2014 10:04 am

Application Level

We have consulted extensively with the cloud provider which has done extensive testing.
We suspected this was firewall related.
out Theory was that the YaCy servers could not communicate with each other to get data on every search - and that this could block the results from being displayed.
However, the cloud provider did independent internal testing.

Results of their internal testing.
1) Port 8090 is OPEN and listening on both clouds.
2) Internet access is unrestricted at the virtual rack level.
3) Communications between the clouds is not blocked
4) Nothing in the firewalls in either cloud should block the YaCy servers from speaking with each other to share / cross- read data before displaying combined results.

Therefore we return to an Application Layer Failure in YaCY connectivity, dating from a mis-configuration caused when patch YaCy version 1.81/9116 was applied to both servers, consecutively.

Can someone please tell us how to restore connectivity manually and fix the display of data for searchers and reconnect the admin screens?
Thank you
xioc752
 
Beiträge: 68
Registriert: Mo Jul 28, 2014 5:01 pm

Re: YaCy version 1.81/9116 knocks servers off Adressing

Beitragvon xioc752 » Sa Nov 01, 2014 10:51 am

Application Level Failure
Independent Fault Testing Results

We have consulted extensively with the cloud provider which has done extensive testing.
We suspected this was shared firewall related.

Remembering our servers are Robinson servers that read each other for search results but do not write to each other,...
And that the clouds are logically and geographically physically separated from each other though accessible to each other in read only mode, through multiple level 3 carrier pipes belonging to the same leading European facilities based carrier and provider.
Our theory was that the YaCy servers could not communicate with each other to get data on every search - and that this could block the results from being displayed.

We thought that the message
The peer must go online to get a peer address.

on Status.html?noforward= was indicating a cascade of errors caused by the servers being unable to find their previous IP addresses when patch version 1.81/9116 was applied to both servers, consecutively.
We still believe this is the case.

We further believe that the 500 Type Error pages for certain Administration functions is probably caused by the servers not being able to find their own previously known IP addresses, and this in turn, generates the errors manifested by 500 Type Error pages shown in the administration of each server. We note the faulty pages with 500 Type Errors are identical on both servers - that both received the 1.81/9116 update within minutes of each other, though consecutively.

However, we suspected this was shared firewall related - where a firewall key setting was potentially damaged.
We have discovered that the fault is not in the firewalls.


At our request, the cloud provider (yesterday) did independent internal examination & testing on both clouds - specifically examining the shared firewall environment related to the hypervisor configurations.

Results of their internal testing.
1) Port 8090 is indeed OPEN and listening on both clouds.
2) Internet access is unrestricted at their virtual rack level. ( We noted that our tests showed 8090 OPEN form the outside and we can see both Admin areas (even if some pages are still showing 500 Errors (unchanged), as noted previously).
3) Communications between the clouds is not blocked the cloud provider has determined
4) Nothing in the firewalls in either cloud should block the YaCy servers from speaking with each other to share / cross- read data before displaying combined results.

We have today applied the currently available patch version version 1.81/9132 to both servers.
We applied all recommended Ubuntu security and related updates for our version Ubuntu and have rebooted both from PuTTY and from the cloud provider interface.
The previous error state and error pages remain on both servers, as before.

Therefore we return to an Application Layer Failure in YaCy connectivity, dating from an internal mis-configuration caused when patch YaCy version 1.81/9116 was applied to both servers, consecutively.

Can someone please tell us how to
1) restore connectivity manually and
2) fix the display of data for searchers so that search results are shown again - and not just the framework at the top of the display page, and
3) reconnect the admin screens such as they remove the 500 Errors, please?

As noted previously, it is our hunch that restoring the IP address connectivity will remove the other errors - though we did not participate in the engine's development, so we really do not know if this is 'the simple fix' that we are seeking to all three problems. This seems to be the unanticipated error state and hence indication that we need to resolve.
The peer must go online to get a peer address.


Thank you very kindly and have a nice day. :)
xioc752
 
Beiträge: 68
Registriert: Mo Jul 28, 2014 5:01 pm

Re: YaCy version 1.81/9116 knocks servers off Adressing

Beitragvon smokingwheels » So Nov 02, 2014 7:16 am

How powerful are your processor/s?
See http://www.linuxandlife.com/2011/11/how ... on-in.html
I'm interested in output from sudo lshw -class processor.

Just confirm you managed to save the API Table and the Content Importer pages as TXT?


I use Bitvise to upload and download files and as a terminal to my Cloud server see http://www.bitvise.com/ssh-client.
But I run Windows at home though there should be something similar for Linux.
smokingwheels
 
Beiträge: 107
Registriert: Sa Aug 31, 2013 7:16 am

Re: YaCy version 1.81/9116 knocks servers off Adressing

Beitragvon xioc752 » Mo Nov 03, 2014 7:59 pm

Thanks for the thoughts...We will do all that....

We got to focus please
Please 1st things 1st....
MANUALLY...
PLEASE Where do we ADD the IP address they need???
The peer must go online to get a peer address.


However the 500 Error Screens are still there and WE Cannot add URLs or RSS links to work on.
It only processes links it already has.


We applied the current updates ( 1.81/9140 ) and things are better, but the Error pages are still there.
We must RECONNECT MANUALLY...but HOW+WHERE, please????

Thanks
xioc752
 
Beiträge: 68
Registriert: Mo Jul 28, 2014 5:01 pm


Zurück zu English

Wer ist online?

Mitglieder in diesem Forum: 0 Mitglieder und 1 Gast

cron