Demo portal has got too many connections

Discussion in English language.
Forumsregeln
You can start and continue with posts in english language in all other forums as well, but if you are looking for a forum to start a discussion in english, this is the right choice.

Demo portal has got too many connections

Beitragvon smokingwheels » Sa Aug 16, 2014 2:52 pm

Demo portal has got too many connections error 503. When I have tried a few times today.
Thanks
smokingwheels
 
Beiträge: 129
Registriert: Sa Aug 31, 2013 7:16 am

Re: Demo portal has got too many connections

Beitragvon Orbiter » Mo Aug 18, 2014 12:07 am

hm, das Demo-Portal soll auch nur das sein: eine Demo. Für großen Andrang nicht gedacht. Es ist immer besser eine eigene YaCy-Instanz zu benutzen. Eigentlich sollte es das Demo-Portal ja gar nicht geben weil es dem Konzept widerspricht, eine verteilte p2p Suchmaschine zu errichten. Insofern bin ich immer froh wenn das Demo-Portal versagt und ich darauf hinweisen kann das es nicht dafür gedacht ist. Wir wollen ja gar kein Suchportal bauen, sondern eine Suchmaschinensoftware für Suchportale.
Orbiter
 
Beiträge: 5787
Registriert: Di Jun 26, 2007 10:58 pm
Wohnort: Frankfurt am Main

Re: Demo portal has got too many connections

Beitragvon tinkerphone » Mo Okt 13, 2014 2:51 pm

Hi,
i thought the "Demo Portal" is supposed to be a "load balancer" :!: :?:

I think it´s quite bad advertising for YaCy if the demo does not work. I really think the YaCy demo should be entrance to the YaCy website, not hidden in some corner. If you don´t like this centralized view on the network, maybe the loadbancer should "just" iframe a host from the list. This would make the decentralized concept of YaCy visible.
tinkerphone
 
Beiträge: 26
Registriert: Fr Okt 10, 2014 10:38 am

Re: Demo portal has got too many connections

Beitragvon Erik_S » Di Okt 14, 2014 8:42 pm

Hello,

tinkerphone hat geschrieben:maybe the loadbancer should "just" iframe a host from the list. This would make the decentralized concept of YaCy visible.
Please try http://130.255.73.69:5353/forward?random for a random YaCy-Peer. But be warned, this Forwarder is a new and not well tested function in my Tool.

Greetings
Erik
Erik_S
 
Beiträge: 185
Registriert: Sa Aug 30, 2014 11:13 am

Re: Demo portal has got too many connections

Beitragvon smokingwheels » Mi Okt 15, 2014 2:32 am

Erik_S hat geschrieben:Hello,

tinkerphone hat geschrieben:maybe the loadbancer should "just" iframe a host from the list. This would make the decentralized concept of YaCy visible.
Please try http://130.255.73.69:5353/forward?random for a random YaCy-Peer. But be warned, this Forwarder is a new and not well tested function in my Tool.

Greetings
Erik


Tried a few times and it works ok but.

Can you factor in a if version number is above 1.81/xxxx then forward to those random peers because the httpd Session Pool max of 50 was only just recently raised it was previous it was 20.
Ask one of the developers which version the setting was raised.

Another thing to take into account is peer response time to do a search, my old YaCy peer was in the order of 12 seconds which is way too long for someone just trying Yacy search out.

Another thing Orbiter wrote German in the English forum but here is what he said.

hm, the demo-portal should be just that: a demo. For large crowds not thought of. It is always better to use your own YaCy instance. Actually, it should not even exist because it contradicts the concept of building a distributed p2p search engine, the demo portal yes. In this respect I am always glad when the demo portal failed and I can point out that it is not intended. We want to build no search portal, but a search engine software for Web portals.

I have my web portal on my twitter page and I left my computer off for a week and lost over 10 Followers 6%.
I now run a Debian VPS with my yacy portal linked to my profile and slowly gained new followers again.

YaCy gives you the freedom to index your liked content and search the results.
smokingwheels
 
Beiträge: 129
Registriert: Sa Aug 31, 2013 7:16 am

Re: Demo portal has got too many connections

Beitragvon Erik_S » Mi Okt 15, 2014 10:07 am

Hello,

smokingwheels hat geschrieben:Can you factor in a if version number is above 1.81/xxxx then forward to those random peers because the httpd Session Pool max of 50 was only just recently raised it was previous it was 20.
Sorry, but no. I do not think that this is a meaningful way to solve the expected Problem. The Version "1.81/xxxx" is a Developer-Version and not a stable Release and there are only a few Peers that update its Software frequently. Increasing the Setting for HTTP-Server-Worker-Threads was one of the Settings that i have changed in the beginning of hosting a YaCy-Peer on a Root-Server, i think the People that operate a YaCy-Peer for the Public have increased this Setting. A random Selection of the YaCy-Peers can help to avoid Problems with overload a small amount of the YaCy-Peers.
In my Opinion, on yacy.net should be a iframe with a real random YaCy-Peer be present. This would show the Concept of Distribution.

smokingwheels hat geschrieben:Another thing to take into account is peer response time to do a search, my old YaCy peer was in the order of 12 seconds which is way too long for someone just trying Yacy search out.
I do not have any statistics about the response times of the YaCy-Peers. My Tool use a short timeout for querying the Peers and all Peers that are able too respond quickly enough are market as "available" in my Data-Base.

smokingwheels hat geschrieben:Another thing Orbiter wrote German in the English forum but here is what he said.
I am a German Native Speaker, English is the language i have problems with.

It is always better to use your own YaCy instance.
I have exactly the same Opinion. For me the Demo-Portal is only a Show-Case for YaCy-Newbies and not a real Product.

Greetings
Erik
Erik_S
 
Beiträge: 185
Registriert: Sa Aug 30, 2014 11:13 am

Re: Demo portal has got too many connections

Beitragvon tinkerphone » Mi Okt 15, 2014 2:30 pm

It is always better to use your own YaCy instance. I have exactly the same Opinion. For me the Demo-Portal is only a Show-Case for YaCy-Newbies and not a real Product.

Greetings
Erik


Well, I don´t want to sound harsh, but i often do... so anyway:

I want to "sell" a new and cool kind of car – it will save the earth if everybody would use it!
But, often when i show it around the wheels fly off and the windshield is broken. Who would "buy" it?

Point is: a defunct demo is a total show stopper

Beside that, I like yacy anyway (even if the autocorrect always give me yack :lol:
tinkerphone
 
Beiträge: 26
Registriert: Fr Okt 10, 2014 10:38 am

Re: Demo portal has got too many connections

Beitragvon Orbiter » Mi Okt 15, 2014 4:40 pm

If YaCy gets successfull and the demo portal would scale to so much people who wants to use that, than at any point we would need a data center and a business model. But that is not what this project is about.

If you think this to the end then you must conclude that the demo portal is not a good idea at all. so.. this brings me (again) to the point where I believe that removing the demo is the only way to protect YaCy from criticism in the way you argued.
Orbiter
 
Beiträge: 5787
Registriert: Di Jun 26, 2007 10:58 pm
Wohnort: Frankfurt am Main

Re: Demo portal has got too many connections

Beitragvon Erik_S » Mi Okt 15, 2014 10:24 pm

Hello,

tinkerphone hat geschrieben:Point is: a defunct demo is a total show stopper
Yes, that is also true.
I think on yacy.net should be an iframe (or something similar) with a real randomly selected YaCy-Peer and a sentence above with
"You see here a random YaCy-Peer, because not all Peers are operated as permanent online with perfect responsiveness, use the reload button for selecting an other Peer."
and all People should understand sporadically problems.

tinkerphone hat geschrieben:Beside that, I like yacy anyway
Me too!

Orbiter hat geschrieben:this brings me (again) to the point where I believe that removing the demo is the only way to protect YaCy from criticism in the way you argued.
Na immerhin bringt Dich seine Kritik zu überhaupt einer Reaktion, so wirksam war meine (zahlreiche) Kritik bisher noch nicht.

Grüße
Erik
Erik_S
 
Beiträge: 185
Registriert: Sa Aug 30, 2014 11:13 am

Re: Demo portal has got too many connections

Beitragvon sixcooler » Mi Okt 15, 2014 11:09 pm

Hello,

the more I think about this, the more I like the idea of the random used YaCy-Host.
Using the freeworld-peers for demonstration could be a great community advertising.

But I remember the days we released 1.0:
We had something similar and the high rate of queries overstressed my machine :-)

cu, sixcooler.
sixcooler
 
Beiträge: 494
Registriert: Do Aug 14, 2008 5:22 pm

Re: Demo portal has got too many connections

Beitragvon Erik_S » Do Okt 16, 2014 1:32 pm

Hello,

sixcooler hat geschrieben:the more I think about this, the more I like the idea of the random used YaCy-Host.
Using the freeworld-peers for demonstration could be a great community advertising.
Then do it!
Showing a random selected YaCy-Peer is a powerful demonstration of the concept distributed.
After heavily testing http://130.255.73.69:5353/forward?random i can say there are no problems, except a few 403-Errors from some YaCy-Peers but liable for this are the YaCy-Developers.
Okay, some of the Peers are slow but if this is really a problem i can add a test for the response-time and accept only fast peers for the internal Random-List.
And my Random-List is very up to date, all IP-Addresses are positive tested within the last 30 seconds.

sixcooler hat geschrieben:We had something similar and the high rate of queries overstressed my machine
One of the positive effects of a load-balancer is too avoid overstressing a single machine.

Greetings
Erik
Erik_S
 
Beiträge: 185
Registriert: Sa Aug 30, 2014 11:13 am

Re: Demo portal has got too many connections

Beitragvon smokingwheels » Do Okt 16, 2014 2:38 pm

Erik_S hat geschrieben:Hello,

Showing a random selected YaCy-Peer is a powerful demonstration of the concept distributed.
After heavily testing http://130.255.73.69:5353/forward?random i can say there are no problems, except a few 403-Errors from some YaCy-Peers but liable for this are the YaCy-Developers.
Okay, some of the Peers are slow but if this is really a problem i can add a test for the response-time and accept only fast peers for the internal Random-List.
And my Random-List is very up to date, all IP-Addresses are positive tested within the last 30 seconds.
Greetings
Erik


Erik,
But I could pick on your URL with my robot and down the YaCy Senior network...not that I would ever do that with out permission to execute. When Yacy is bullet proof I may support your thinking. I have 67 bug reports that I have found some have been fixed but YaCy is not a Google replacement it has lots of options and you never have to Submit anything.
smokingwheels
 
Beiträge: 129
Registriert: Sa Aug 31, 2013 7:16 am

Re: Demo portal has got too many connections

Beitragvon Erik_S » Do Okt 16, 2014 3:27 pm

Hello,

smokingwheels hat geschrieben:But I could pick on your URL with my robot and down the YaCy Senior network
Yes, that is true. You can also enhance your robot with its own YaCy-Network-Tracker (this is lesser than 2000 Lines of C-Code), this kind of problem is independent of my Service.
The only way to make YaCy "bullet proof" is to make it much bigger (more than 100'000 Peers, for just saying a number). Okay, there are a lot of problems and weaknesses in the Peer-to-Peer-Network-Managment-Logic inside the YaCy-Code, this makes it difficult. In my opinion, the current existing YaCy can not benefit enough from a growing number of peers.

smokingwheels hat geschrieben:you never have to Submit anything
I have offered my help multiple times in the last 6 weeks, but without any kind of reaction.

Greetings
Erik
Erik_S
 
Beiträge: 185
Registriert: Sa Aug 30, 2014 11:13 am

Re: Demo portal has got too many connections

Beitragvon tinkerphone » Do Okt 16, 2014 4:37 pm

Erik_S hat geschrieben:Hello,

smokingwheels hat geschrieben:But I could pick on your URL with my robot and down the YaCy Senior network
Yes, that is true. You can also enhance your robot with its own YaCy-Network-Tracker (this is lesser than 2000 Lines of C-Code), this kind of problem is independent of my Service.
The only way to make YaCy "bullet proof" is to make it much bigger (more than 100'000 Peers, for just saying a number). Okay, there are a lot of problems and weaknesses in the Peer-to-Peer-Network-Managment-Logic inside the YaCy-Code, this makes it difficult. In my opinion, the current existing YaCy can not benefit enough from a growing number of peers.

smokingwheels hat geschrieben:you never have to Submit anything
I have offered my help multiple times in the last 6 weeks, but without any kind of reaction.

Greetings
Erik


Well, might be a load balancing function should be implemented into YaCy? Until then the iFrame Balancer which Erik_S is a really nice alternative!
If my peer has too many connections, why does the frontend get killed? (not to speak about the backend which does the hobo as well)

If my peer has too many connection it means that my YaCy has too many connections - right? Not necessarily my server.
If YaCy would provide a list with available connections along with the node list, my peer could forward the query to an other peer and display the results (with a note that this happened). -> decouple the frontend!!
A query could take along a counter of hops (numbers of unsuccessfully connected peers). If a query had too many hops, it is delayed and reinvoked after a certain time. After x tries the query gets deleted and the user a message:
"Sorry. The YaCy network is too busy. Please try again in a second...
-> Download and run your own YaCy Server and help to expand the network!" <-


My idea does certainly not take all implemented features of YaCy into account. But i simply can´t understand why the frontend goes down if my YaCy has too much to do. The frontend should be able to find atlast one working peer which can deliver some data...
tinkerphone
 
Beiträge: 26
Registriert: Fr Okt 10, 2014 10:38 am

Re: Demo portal has got too many connections

Beitragvon smokingwheels » Fr Okt 17, 2014 12:30 am

smokingwheels hat geschrieben:
Erik_S hat geschrieben:Hello,

Showing a random selected YaCy-Peer is a powerful demonstration of the concept distributed.
After heavily testing http://130.255.73.69:5353/forward?random i can say there are no problems, except a few 403-Errors from some YaCy-Peers but liable for this are the YaCy-Developers.
Okay, some of the Peers are slow but if this is really a problem i can add a test for the response-time and accept only fast peers for the internal Random-List.
And my Random-List is very up to date, all IP-Addresses are positive tested within the last 30 seconds.
Greetings
Erik


Erik,
But I could pick on your URL with my robot and down the YaCy Senior network...not that I would ever do that with out permission to execute. When Yacy is bullet proof I may support your thinking. I have 67 bug reports that I have found some have been fixed but YaCy is not a Google replacement it has lots of options and you never have to Submit anything.


What about creating a web hop for your url at http://freedns.afraid.org just an idea.


tinkerphone hat geschrieben:Well, might be a load balancing function should be implemented into YaCy? Until then the iFrame Balancer which Erik_S is a really nice alternative!
If my peer has too many connections, why does the frontend get killed? (not to speak about the backend which does the hobo as well)

-> Download and run your own YaCy Server and help to expand the network!" <-

My idea does certainly not take all implemented features of YaCy into account. But i simply can´t understand why the frontend goes down if my YaCy has too much to do. The frontend should be able to find atlast one working peer which can deliver some data...


Tinkerphone you can increase the log file length as per http://mantis.tokeek.de/view.php?id=353 to see whats going on with your peer if it crashes a lot. I add a extra 0 to the length setting this seems to work well.

Erik_S
I had a look at my youtube Analytics and only 55% of my views come from a PC the rest was mobile phone, Tablet, Game Console and TV. Maybe YaCy in being accessed by devices other than a PC who knows.
smokingwheels
 
Beiträge: 129
Registriert: Sa Aug 31, 2013 7:16 am

Re: Demo portal has got too many connections

Beitragvon tinkerphone » Fr Okt 17, 2014 6:44 am

smokingwheels hat geschrieben:Tinkerphone you can increase the log file length as per http://mantis.tokeek.de/view.php?id=353 to see whats going on with your peer if it crashes a lot. I add a extra 0 to the length setting this seems to work well. ... snip..


The last days i simply used max connections = 200 which "solved" the problem (but that setting was not saved - it backs to 50 after restart).
For now all servers are in Robinson. I check again when i open up again.
tinkerphone
 
Beiträge: 26
Registriert: Fr Okt 10, 2014 10:38 am

Re: Demo portal has got too many connections

Beitragvon Erik_S » Fr Okt 17, 2014 11:19 am

Hello,

tinkerphone hat geschrieben:Well, might be a load balancing function should be implemented into YaCy?
I do not think that this is a good idea, the design goal of the Peer-to-Peer-Management-Logic in YaCy is too much different. And the Logic in the most YaCy-Peers does not know really all active YaCy-Peers, and the Logic in YaCy is too slow for a fast changing network.

smokingwheels hat geschrieben:What about creating a web hop for your url at http://freedns.afraid.org
Nice idea! If you can do that, i would be happy. When the problems with IPv6 in YaCy are solved, we can add the IPv6-Address of my Server.
But please note that the HTTP-Server in my Tool does not answer for the URL '/' (it generates a 404) and it is not available at Port 80. If you create a real Domain-Name than there must be a Forwarder to my URL that do listen on Port 80 and for the URL '/' (or ignoring the URL at all).
I have a small Web-Space (on a shared Web Hoster) and i will try a similar Feature for a subdirectory with a special .htaccess (i have talked about this kind of idea with flegno 2 weeks ago, see here).

smokingwheels hat geschrieben:Maybe YaCy in being accessed by devices other than a PC who knows.
This is exactly the reason why i think on yacy.net should be an iframe or a Link or something similar to http://130.255.73.69:5353/forward?random.
This would show the Power of a distributed System and enable all Users, that are not able to run their own YaCy-Instance, for using the YaCy-Search-Engine without overstressing a single machine (or a small number of "special selected" machines).

I am not sure but i think my service should be able to handle more than 100 forwarding-operations per second. This means there must be at least 100 new Searchers per Second to overstressing my service. This means not that my Service is absolute "DoS proof" but you need much more power to DoS my Service compared to the DoS-Power needed for overstressing multiple YaCy-Peers.

Greetings
Erik
Erik_S
 
Beiträge: 185
Registriert: Sa Aug 30, 2014 11:13 am

Re: Demo portal has got too many connections

Beitragvon smokingwheels » Sa Okt 18, 2014 7:10 am

Erik_S,

I created a web forward to your url it did not work at first so I did one other action then it worked.

Is it possible for you to create a random list and then step thru them one by one because while I was testing I hit 1 server 4 times out of 64?

Some are very slow in loading the search page even one dead, search.yacy.de loads in 1.8 seconds.

It might be a good idea if you have a list of subscribers that users can register with you if they want to be on the random list.
smokingwheels
 
Beiträge: 129
Registriert: Sa Aug 31, 2013 7:16 am

Re: Demo portal has got too many connections

Beitragvon Erik_S » Sa Okt 18, 2014 9:35 am

Hello,

smokingwheels hat geschrieben:Is it possible for you to create a random list and then step thru them one by one because while I was testing I hit 1 server 4 times out of 64?
Are the 4 times uniformly distributed over the 64 tries? What is the over-all time of the 64 tries?
The Random-List works in this way: there are two small Lists (one for IPv4 and one for IPv6) and every first "working" IPv4-Address and every first "working" IPv6-Address of each running Peer is inserted in this Lists (each List has a cycling Write-Pointer), then you call the '/forward?random' Servlet it selects per random one of the IP-Addresses (the List is selected by the IP-Version of the incoming HTTP-Request, my Server is fully dual stacked) and returns a HTTP-307-Redirect.

smokingwheels hat geschrieben:Some are very slow in loading the search page even one dead
I can add a test for the response-time of the complete transfer of the entire Seed-List and only use Peers for the Random-List that send their Seed-List in lesser than 3 seconds. Is this okay?
But this is a trade-off between a wide variety of Peers and a selection for "high-quality" Peers. Is in your opinion the load-time of the majority of Peers fast enough?
What was the symptom of the one "dead" Peer?
My random List is filled only with IP-Addresses that are positive tested within the last approximately 30 seconds. But i must say 1 fail on 64 tries is a good result for a distributed system that contains mostly Peers on private Home-Computers.

smokingwheels hat geschrieben:It might be a good idea if you have a list of subscribers that users can register with you if they want to be on the random list.
You mean an opt-in? I think this is the opposite of the goal of my current random Selection.
I would prefer a list with "special announced" YaCy-Peers on yacy.net, this List can be a List of Links as
Code: Alles auswählen
<a href="http://130.255.73.69:5353/forward?name=Peer-Name">Peer-Name</a>
The Forwarder for Peer-Names and Peer-Hashs is present in my Tool but do not work correctly, in the Source-Code this problem is well fixed but i will finish an other important Feature prior to updating my Tool on the Server.

Greetings
Erik
Erik_S
 
Beiträge: 185
Registriert: Sa Aug 30, 2014 11:13 am

Re: Demo portal has got too many connections

Beitragvon Erik_S » Sa Okt 18, 2014 12:56 pm

Hello,

smokingwheels hat geschrieben:What about creating a web hop for your url at http://freedns.afraid.org
flegno has created a Web-Hop at http://random-yacy.sprechrun.de/, see here.
But he use the IP-Address of my Server as a static Constant, i think it could be a good idea to take a real Domain-Name (for example from a free DynDNS-Service like freedns.afraid.org) and connect my IPv4-Address and my IPv6-Address to it. This would automagically enable the IPv6-Capabilities of my Forwarder- and Analysis-Tool for all Users.

Greetings
Erik
Erik_S
 
Beiträge: 185
Registriert: Sa Aug 30, 2014 11:13 am

Re: Demo portal has got too many connections

Beitragvon smokingwheels » Sa Okt 18, 2014 1:03 pm

I can add a test for the response-time of the complete transfer of the entire Seed-List and only use Peers for the Random-List that send their Seed-List in lesser than 3 seconds. Is this okay?


Edit is Seed list http://smokingwheels.evils.in:8090/Netw ... Count=1000
If so my time is around 3 sec but my search page load is approx 1.2 see below.
A 3 word search http://www.webpagetest.org/result/141018_3T_GGW/ 1.8 sec
You may add complexity into it
If you could possibly test the peer load time at http://www.webpagetest.org/ just to complete the home search page for the ones that are faster than 3-4 seconds this would be good. But in the early days search delays for a user was an issue. Again search.yacy.de load is 1.8 seconds.
You would manually go thru the list to obtain a list of peers that are capable. I know thats not your idea of what you want.
Also listing your IP is not as good as http://yacy.mooo.com or similar.

Are the 4 times uniformly distributed over the 64 tries? What is the over-all time of the 64 tries?


I increased the tabs to over 100 in one go and some times the same peer was listed 3 in a row but not very often.
I ran a test over 100 browser tabs at 1 second interval after a few roll overs not many peers could keep up this was with the least load possible a 1 word search, as you increase the number of word it slows down. Not too sure if my slow PC is affecting it though.

You mean an opt-in? I think this is the opposite of the goal of my current random Selection.
I would prefer a list with "special announced" YaCy-Peers on yacy.net, this List can be a List of Links as


Special announced... hmmm
You can add my server http://smokingwheels.evils.in:8090 my IP will some times change when I kill my VPS server, but its response time is around 1.2 sec see http://www.webpagetest.org/result/141018_MB_G21/

Just seen reply by flegno I will give it a test.
smokingwheels
 
Beiträge: 129
Registriert: Sa Aug 31, 2013 7:16 am

Re: Demo portal has got too many connections

Beitragvon smokingwheels » Sa Okt 18, 2014 1:32 pm

Erik_S hat geschrieben:Hello,

smokingwheels hat geschrieben:What about creating a web hop for your url at http://freedns.afraid.org
flegno has created a Web-Hop at http://random-yacy.sprechrun.de/, see here.
But he use the IP-Address of my Server as a static Constant, i think it could be a good idea to take a real Domain-Name (for example from a free DynDNS-Service like freedns.afraid.org) and connect my IPv4-Address and my IPv6-Address to it. This would automagically enable the IPv6-Capabilities of my Forwarder- and Analysis-Tool for all Users.

Greetings
Erik


At freedns.afraid.org you can setup IPv6 but not sure how to do that just yet. You can also Cloak end url see http://yacy.mooo.com .
First test of http://random-yacy.sprechrun.de it seems to load quicker but I will compare and let you know.
Here is response time of 830 ms http://www.webpagetest.org/result/14101 ... 1/details/
Mine http://www.webpagetest.org/result/14101 ... 1/details/

It depends where your server is though.

A test run of yacy.mooo.com thru Erik_s URL https://www.youtube.com/watch?v=lOBiOAtDHcA
smokingwheels
 
Beiträge: 129
Registriert: Sa Aug 31, 2013 7:16 am

Re: Demo portal has got too many connections

Beitragvon Erik_S » Sa Okt 18, 2014 3:15 pm

Hello,

i use the Seed-List-API (see http://www.yacy-websuche.de/wiki/index.php/Dev:APIseedlist) and this is a single File that need only low CPU-Power on the YaCy-Peer, this means the Load-Time depends primary by the IP-Round-Trip-Time and TCP-Throughput. In my opinion the load-time of this File give me a good point for the "Connection-Speed" of the Peer. Most Peers transfer this File in lesser than 1 second.

smokingwheels hat geschrieben:You may add complexity into it
I have added a Test for Load-Time is below 3 seconds into my Tool and it seems to working correctly, this modification was 2 new calls to 'time()' and a 'if'.

smokingwheels hat geschrieben:If you could possibly test the peer load time at http://www.webpagetest.org/ just to complete the home search page for the ones that are faster than 3-4 seconds this would be good.
Sorry, but this is real complexity. Additionally webpagetest.org need some time but one of the design goals of my Random-Forwarder is to deliver really up to date IP-Addresses.

smokingwheels hat geschrieben:I increased the tabs to over 100 in one go
Do you mean you open 100 (or more) Tabs at one Moment? This is, in my opinion, not a realistic Test-Case. The internal List of IP-Addresses, that are used by the Random-Forwarder, is small because it should not contain too old IP-Addresses. I think in the real World the Users of a generic/random YaCy-Search-Page comes distributed over time.

smokingwheels hat geschrieben:Here is response time of 830 ms http://www.webpagetest.org/result/14101 ... 1/details/
Mine http://www.webpagetest.org/result/14101 ... 1/details/
This looks nice for me. My Forwarder has below 140ms at "Initial Connection" and "Time to First Byte", i think this is a good sign for my VPS-Hoster and for my HTTP-Server-Implementation (and the last of this has potential for further optimizations).

Do you have a working IPv6-Connection to the public Internet?
I will try an other way for a random Search-Page in my Tool, similar to your yacy.mooo.com. In my opinion the solution that is realized by flegno (okay it was my idea) has the disadvantage that pressing the reload-button does not take another random YaCy-Peer.

Greetings
Erik
Erik_S
 
Beiträge: 185
Registriert: Sa Aug 30, 2014 11:13 am

Re: Demo portal has got too many connections

Beitragvon smokingwheels » So Okt 19, 2014 5:09 am

Erik_s hat geschrieben:I have added a Test for Load-Time is below 3 seconds into my Tool

I did a quick test and all peers bring up search results in a reasonable amount of time now.

Erik_s hat geschrieben:Do you mean you open 100 (or more) Tabs at one Moment?

Yes but my system is very slow though.

Erik_s hat geschrieben:Do you have a working IPv6-Connection to the public Internet?

My VPS Peer used show up to have IPv6 last week but since the updates it does not show as one, Yacy status displays one but in the network page it is not there, not sure if I have to configure anything my end for it to work.
smokingwheels
 
Beiträge: 129
Registriert: Sa Aug 31, 2013 7:16 am

Re: Demo portal has got too many connections

Beitragvon Erik_S » Di Okt 21, 2014 8:52 pm

Hello,

smokingwheels hat geschrieben:I did a quick test and all peers bring up search results in a reasonable amount of time now.
Really, that is very cool. Mainly because i have updated my Tool on the Server today.

And now there is a Frame-Based Search-Page on the URL http://130.255.73.69:5353/, similar to your yacy.mooo.com. Try it.
It performs well, see http://www.webpagetest.org/result/141021_KV_118T/1/details/.
Now i need a real Domain-Name for my Server.

smokingwheels hat geschrieben:
Erik_s hat geschrieben:Do you have a working IPv6-Connection to the public Internet?
My VPS Peer used show ....
I was meaning: Do you have a working IPv6-Connection for your Work/Home-Computer respectively for your Browser?
If yes i can send you the IPv6-Address of my Server and you can test the Forwading for IPv6.
But there are only 2 up to 4 real available IPv6-YaCy-Peers at the Moment.

Greetings
Erik
Erik_S
 
Beiträge: 185
Registriert: Sa Aug 30, 2014 11:13 am

Re: Demo portal has got too many connections

Beitragvon smokingwheels » Mi Okt 22, 2014 10:45 am

Erik I have run a few tests and found the yacy.mooo.com with freedns.org sometimes takes 10 to 25 seconds to redirect.

This run of 9 was with http://random-yacy.sprechrun.de/ and does not seem to suffer.
When you go to http://www.webpagetest.org hit the advanced tab and change to 9 tests.

http://www.webpagetest.org/result/141022_MD_FQ6/

My home router does not support IPv6.

Could you increase the time constraints by 30 to 40 % and see how that goes, I still get repeating peers when calling it 10 times in a row.
smokingwheels
 
Beiträge: 129
Registriert: Sa Aug 31, 2013 7:16 am

Re: Demo portal has got too many connections

Beitragvon Erik_S » Mi Okt 22, 2014 1:40 pm

Hello,

smokingwheels hat geschrieben:I have run a few tests and found the yacy.mooo.com with freedns.org sometimes takes 10 to 25 seconds to redirect.
How is this Forwarding-Service implemented? Is the HTML-Frame-Code generated (automagically) by freedns.org or by your hands.

smokingwheels hat geschrieben:My home router does not support IPv6.
You should update, your router is 15 years in the Past.

smokingwheels hat geschrieben:Could you increase the time constraints by 30 to 40 % and see how that goes
Do you mean i should increase the 3 second time limit to a 4 seconds time limit (for selecting "fast" Peers)?

smokingwheels hat geschrieben:I still get repeating peers when calling it 10 times in a row.
If you query my Tool multiple times in a short time than this is normal. My tool has a small List with the last few positive tested Peers and it needs a time for replacing all Peers in this List with new ones. If you query multiple times inside this duration, than exist the possibility for obtaining the same Peer multiple times. In my Opinion, for typical Users this should not a Problem. If this is a real problem than i have the option for a bigger List, this results in more Peers for a random selection (makes a smaller possibility for the same Peer) but also it needs more time for replacing all Peers in this List and the Peers are outdated for a longer time (makes a higher possibility for offline Peers). I am not sure that is the ideal size of the List.

Greetings
Erik
Erik_S
 
Beiträge: 185
Registriert: Sa Aug 30, 2014 11:13 am

Re: Demo portal has got too many connections

Beitragvon smokingwheels » Do Okt 23, 2014 2:59 am

Erik_S hat geschrieben:Hello,

smokingwheels hat geschrieben:I have run a few tests and found the yacy.mooo.com with freedns.org sometimes takes 10 to 25 seconds to redirect.
How is this Forwarding-Service implemented? Is the HTML-Frame-Code generated (automagically) by freedns.org or by your hands.

smokingwheels hat geschrieben:My home router does not support IPv6.
You should update, your router is 15 years in the Past.

smokingwheels hat geschrieben:Could you increase the time constraints by 30 to 40 % and see how that goes
Do you mean i should increase the 3 second time limit to a 4 seconds time limit (for selecting "fast" Peers)?

smokingwheels hat geschrieben:I still get repeating peers when calling it 10 times in a row.
If you query my Tool multiple times in a short time than this is normal. My tool has a small List with the last few positive tested Peers and it needs a time for replacing all Peers in this List with new ones. If you query multiple times inside this duration, than exist the possibility for obtaining the same Peer multiple times. In my Opinion, for typical Users this should not a Problem. If this is a real problem than i have the option for a bigger List, this results in more Peers for a random selection (makes a smaller possibility for the same Peer) but also it needs more time for replacing all Peers in this List and the Peers are outdated for a longer time (makes a higher possibility for offline Peers). I am not sure that is the ideal size of the List.

Greetings
Erik


Yes Increase the time limit to 4 seconds.

I think when the Yacy peer to peer network has less bugs in it for overloaded systems your system might be useful, I think its a good idea though as mobile users could use it.
smokingwheels
 
Beiträge: 129
Registriert: Sa Aug 31, 2013 7:16 am

Re: Demo portal has got too many connections

Beitragvon Erik_S » Do Okt 23, 2014 11:34 am

Hello,

smokingwheels hat geschrieben:Yes Increase the time limit to 4 seconds.
Currently are little above 100 Peers classified as "fast" Peer, i can not see any reason for a weaker selection.

smokingwheels hat geschrieben:I think when the Yacy peer to peer network has less bugs in it for overloaded systems your system might be useful
Me too.

smokingwheels hat geschrieben:I think its a good idea though as mobile users could use it.
I hope that all Users can use this Service.

Greetings
Erik
Erik_S
 
Beiträge: 185
Registriert: Sa Aug 30, 2014 11:13 am

Re: Demo portal has got too many connections

Beitragvon smokingwheels » Fr Okt 24, 2014 9:03 am

Hey Erik,
Could you do a random peer list for Yacy servers with SSL enabled?
I read on a website that yacy does not support SSL. Its probably cause you have to mess around to get it to work.
My Redirect is http://swa.evils.in/
Cheers
smokingwheels
 
Beiträge: 129
Registriert: Sa Aug 31, 2013 7:16 am

Re: Demo portal has got too many connections

Beitragvon Erik_S » Fr Okt 24, 2014 9:42 am

Hello,

smokingwheels hat geschrieben:Could you do a random peer list for Yacy servers with SSL enabled?
Do you mean i should generate an additional List with an additional Query-URL only for Peers what have the SSL-Feature enabled?
Theoretical i can do this, the information about enabled SSL is available. But the SSL-Port of the Peers is not available, that means i can not offer a forwarding-mechanism for HTTPS.

smokingwheels hat geschrieben:My Redirect is http://swa.evils.in/
In my opinion the problem is the self-signed certificate that is used by YaCy.
I have tried your Link and that forwards my browser to an other Domain-Name with the HTTPS-Protocol and Port 8443. Both Chromium and Firefox are not able to use this SSL-Encryption.
But if i resolve the Domain-Name by hand (with dig) and enter the IP-Address with Protocol HTTPS and Port 8443 directly than are Chromium and Firefox able to access your YaCy-Peer, Firefox needs some extra Mouse-Clicks for accepting (but not saving) a special Exception-Rule but it works too.
As far as i see this situation, the root cause for this problem is that the certificate is only valid for the IP-Address of the YaCy-Peer and not for a Domain-Name. A SSL-Certificate can only be valid for one Host. A Domain-Name and an IP-Address are different Hosts (from the Point of View of a Browser).

Greetings
Erik
Erik_S
 
Beiträge: 185
Registriert: Sa Aug 30, 2014 11:13 am

Re: Demo portal has got too many connections

Beitragvon smokingwheels » Fr Okt 24, 2014 11:40 am

Erik_s hat geschrieben:
smokingwheels hat geschrieben:Could you do a random peer list for Yacy servers with SSL enabled?
Do you mean i should generate an additional List with an additional Query-URL only for Peers what have the SSL-Feature enabled?
Theoretical i can do this, the information about enabled SSL is available. But the SSL-Port of the Peers is not available, that means i can not offer a forwarding-mechanism for HTTPS.

You might have to assume port 8443 or 443 and manually make a list, which I could do for you.

Erik_s hat geschrieben:
smokingwheels hat geschrieben:My Redirect is http://swa.evils.in/
In my opinion the problem is the self-signed certificate that is used by YaCy.
I have tried your Link and that forwards my browser to an other Domain-Name with the HTTPS-Protocol and Port 8443. Both Chromium and Firefox are not able to use this SSL-Encryption.
But if i resolve the Domain-Name by hand (with dig) and enter the IP-Address with Protocol HTTPS and Port 8443 directly than are Chromium and Firefox able to access your YaCy-Peer, Firefox needs some extra Mouse-Clicks for accepting (but not saving) a special Exception-Rule but it works too.
As far as i see this situation, the root cause for this problem is that the certificate is only valid for the IP-Address of the YaCy-Peer and not for a Domain-Name. A SSL-Certificate can only be valid for one Host. A Domain-Name and an IP-Address are different Hosts (from the Point of View of a Browser).

Greetings
Erik

I think the certificate is self signed and that's why Firefox says its untrusted. Do you know a free way to fix that?
Chrome just gives me a sad face in a square box when I try to access it.
I will create create another certificate for my domain and see how that goes.
Cheers

I created another certificate Crashed Yacy but a remove and install fixes that in Debian and it makes no difference to the warning message.
smokingwheels
 
Beiträge: 129
Registriert: Sa Aug 31, 2013 7:16 am

Re: Demo portal has got too many connections

Beitragvon Erik_S » Fr Okt 24, 2014 3:58 pm

Hello,

smokingwheels hat geschrieben:You might have to assume port 8443 or 443 and manually make a list, which I could do for you.
Why? Sorry, but i do not understand the intention of this task. I do not know how many YaCy-Peers have SSL enabled (but i can implement a Statistic-Value for this Information in my Tool if you wish), so i can not say how useful is a List of this Peers. And guessing the Information about the used Port is not a meaningful way, in my opinion.

smokingwheels hat geschrieben:I think the certificate is self signed and that's why Firefox says its untrusted.
No, the Problem is: there exist multiple Ways to access a YaCy-Peer (one or multiple IP-Addresses and a not limited Number of Domain-Names that resolve to one or more of the IP-Addresses) and for fulfilling the SSL-Paradigm YaCy need an extra Certificate for each Way (and Browser that support this: http://en.wikipedia.org/wiki/Server_Name_Indication, most Browsers do https://sni.velox.ch/).

smokingwheels hat geschrieben:Do you know a free way to fix that?
I think RFC6698 can be a Solution. The primary intention of my Tool is to be a DNS-Server for the both YaCy-Top-Level-Domains .yacy and .yacyh, so there exist soon a possible Way to solve this Problem.

Greetings
Erik
Erik_S
 
Beiträge: 185
Registriert: Sa Aug 30, 2014 11:13 am

Re: Demo portal has got too many connections

Beitragvon smokingwheels » Fr Okt 24, 2014 10:51 pm

Erik_S hat geschrieben:Hello,

smokingwheels hat geschrieben:You might have to assume port 8443 or 443 and manually make a list, which I could do for you.
Why? Sorry, but i do not understand the intention of this task. I do not know how many YaCy-Peers have SSL enabled (but i can implement a Statistic-Value for this Information in my Tool if you wish), so i can not say how useful is a List of this Peers. And guessing the Information about the used Port is not a meaningful way, in my opinion.


More people might try Yacy out if there is a demo SSL sort of URL connection.
I did a test to my Twitter feed yesterday and found I had a log file of 66 k pretty quickly and today some 12 hours later its now 244k. Note "I does not contain any IP's"
If you go to /Network.html?page=1&maxCount=1000 you should see a little black lock next to SSL enabled peers. However I have noticed when I look at my other peer from each peer there not showing up as SSL Enabled.

Erik_S hat geschrieben:
smokingwheels hat geschrieben:I think the certificate is self signed and that's why Firefox says its untrusted.
No, the Problem is: there exist multiple Ways to access a YaCy-Peer (one or multiple IP-Addresses and a not limited Number of Domain-Names that resolve to one or more of the IP-Addresses) and for fulfilling the SSL-Paradigm YaCy need an extra Certificate for each Way (and Browser that support this: http://en.wikipedia.org/wiki/Server_Name_Indication, most Browsers do https://sni.velox.ch/).

Interesting looks like someone should add it to the DEV Wishlist, beyond me.

Erik_S hat geschrieben:
smokingwheels hat geschrieben:Do you know a free way to fix that?
I think RFC6698 can be a Solution. The primary intention of my Tool is to be a DNS-Server for the both YaCy-Top-Level-Domains .yacy and .yacyh, so there exist soon a possible Way to solve this Problem.

Greetings
Erik

Hope you get your tool to work.
Cheers
smokingwheels
 
Beiträge: 129
Registriert: Sa Aug 31, 2013 7:16 am

Re: Demo portal has got too many connections

Beitragvon Erik_S » Sa Okt 25, 2014 10:40 am

Hello,

smokingwheels hat geschrieben:More people might try Yacy out if there is a demo SSL sort of URL connection.
Yes, i have the same opinion. But i think the most important Step in this direction is the inclusion of the SSL-Port into the Seedlists, after this i will add a HTTPS-Forwarder to my tool.

smokingwheels hat geschrieben:I did a test to my Twitter feed yesterday and found I had a log file of 66 k pretty quickly
66k what? HTTPS-Accesses to your YaCy-Peer?

smokingwheels hat geschrieben:If you go to /Network.html?page=1&maxCount=1000 you should see a little black lock next to SSL enabled peers.
Yes, this information is included in the Seedlists, but the Port-Number is not.

smokingwheels hat geschrieben:looks like someone should add it to the DEV Wishlist
Yes, but i am the wrong person for this, Orbiter ignored me.

smokingwheels hat geschrieben:Hope you get your tool to work.
Thanks. After some small fixes and improvements of the current feature-set of my tool, the DNS-Server (for UDP-Port 53) is the next task on my todo-list.

Greetings
Erik
Erik_S
 
Beiträge: 185
Registriert: Sa Aug 30, 2014 11:13 am


Zurück zu English

Wer ist online?

Mitglieder in diesem Forum: 0 Mitglieder und 2 Gäste