a YaCy Search Portal with javascript: peer-search.net

Ereignisse, Vorschläge und Aktionen

a YaCy Search Portal with javascript: peer-search.net

Beitragvon Orbiter » Fr Sep 03, 2010 8:44 am

Nathan made a very interesting and YaCy-architecture-compliant search portal for YaCy on
http://peer-search.net

When entering the site, the search client is loaded into your browser and a search to a random selection of YaCy peers is done right from your browser, not from a search portal. That means that the search is still decentralized while it is possible to get an entry point from a single address. This is a little bit like a YaCy search client re-implementation for javascript.

Please try that out and give Nathan some feed-back!

What I really like is, that the search is super-fast and the switch from first to next result page can be made with arrow keys (which is even faster than the first search). What do you think? Nathan wants to improve the page bevore making more 'noise' about it.
Orbiter
 
Beiträge: 5796
Registriert: Di Jun 26, 2007 10:58 pm
Wohnort: Frankfurt am Main

Re: a YaCy Search Portal with javascript: peer-search.net

Beitragvon nstaudt » Fr Sep 03, 2010 2:30 pm

As you can tell from the current design of this site, I'm not a designer - if anyone wanted to put together a layered pdn file with a sleeker design, I would be most grateful :)
nstaudt
 
Beiträge: 73
Registriert: Fr Aug 13, 2010 10:54 am

Re: a YaCy Search Portal with javascript: peer-search.net

Beitragvon cfr34x » Sa Sep 04, 2010 1:12 pm

Hi,

I have just tested it, and here is my feedback:

What I really like is the search speed, which is very fast.
However, I found 2 problems:

  • It seems that german umlauts are not supported. I tried some random words and never got results. However, the same words gave results on my local peer.
  • With NoScript enabled, it is not really comfortable to use, because I have to allow every single peer to get any results. Is it possible to use only one script, so that I only have to allow one site?

Your design is not at all bad, I like the simplicity, but there is of course room for improvement ;) .

With best regards
cfr34x
cfr34x
 
Beiträge: 49
Registriert: Sa Jan 31, 2009 6:24 pm

Re: a YaCy Search Portal with javascript: peer-search.net

Beitragvon nstaudt » So Sep 05, 2010 9:47 am

Thanks for the feedback, I'll have a look at why umlauts aren't producing correct results tomorrow. My first guess is that the urlencoding routine that I use has an error, but that's just a suspicion.

Regarding noscript... I would be surprised if it worked at all with javascript disabled, javascript is how it does everything. I'll put in something to detect noscript and tell the user that script is required. (with javascript turned off, the search terms would have to be sent back to my webserver and that is what we're trying to avoid :)

Many Thanks,
Nathan.
nstaudt
 
Beiträge: 73
Registriert: Fr Aug 13, 2010 10:54 am

Re: a YaCy Search Portal with javascript: peer-search.net

Beitragvon cfr34x » So Sep 05, 2010 8:17 pm

nstaudt hat geschrieben:Regarding noscript... I would be surprised if it worked at all with javascript disabled, javascript is how it does everything. I'll put in something to detect noscript and tell the user that script is required. (with javascript turned off, the search terms would have to be sent back to my webserver and that is what we're trying to avoid :)

Well, I looked into this again, and it seems this is a NoScript-specific problem, so I don't think you can change anything in this case.
It is blocking the loading of yacysearch.json from the individual peers, even if JavaScript from peer-search.net is allowed.
cfr34x
 
Beiträge: 49
Registriert: Sa Jan 31, 2009 6:24 pm

Re: a YaCy Search Portal with javascript: peer-search.net

Beitragvon nstaudt » Mo Sep 06, 2010 7:45 am

ah, that makes more sense - I'll try to find out if I can detect that plugin and notify users accordingly.

Thanks :D
nstaudt
 
Beiträge: 73
Registriert: Fr Aug 13, 2010 10:54 am

Re: a YaCy Search Portal with javascript: peer-search.net

Beitragvon Quix0r » Di Sep 07, 2010 6:58 pm

Sorry, but this is the only thing I can see. Latest Firefox version:
Bild

And there are two disabled arrows on right upper side.
Quix0r
 
Beiträge: 1345
Registriert: Di Jul 31, 2007 9:22 am
Wohnort: Krefeld

Re: a YaCy Search Portal with javascript: peer-search.net

Beitragvon nstaudt » Di Sep 07, 2010 8:13 pm

yeah, sorry - should have posted, i've been breaking it today while experimenting with a number of things (and having problems with hosting environment too), will post when it's ready again. (this is why i don't want to advertise the site just yet - need to get it right first :D )

thanks for posting though ;)
nstaudt
 
Beiträge: 73
Registriert: Fr Aug 13, 2010 10:54 am

Re: a YaCy Search Portal with javascript: peer-search.net

Beitragvon nstaudt » Mi Sep 08, 2010 10:29 am

I'm moving it to a different web host - my original choice of web host turned out to be very poor, their environment didn't support the advertised features and their support staff were absolutely clueless... oh well, live and learn. :lol: Until the name server records update/propagate, there's just an iframe on the old web host, which won't matter too much, except you won't be able to enter urls like peer-search.net#searchterm
nstaudt
 
Beiträge: 73
Registriert: Fr Aug 13, 2010 10:54 am

Re: a YaCy Search Portal with javascript: peer-search.net

Beitragvon nstaudt » Mi Sep 08, 2010 11:37 am

umlauts (and indeed all UTF-8 characters) are now properly encoded.

However, I have identified an issue with the yacy JSON interface (I made a topic in the problems thread), so for now, any searches with double quotes (") won't work.
nstaudt
 
Beiträge: 73
Registriert: Fr Aug 13, 2010 10:54 am

Re: a YaCy Search Portal with javascript: peer-search.net

Beitragvon nstaudt » Do Sep 09, 2010 1:25 pm

I've updated the site again, have a look :)
nstaudt
 
Beiträge: 73
Registriert: Fr Aug 13, 2010 10:54 am

Re: a YaCy Search Portal with javascript: peer-search.net

Beitragvon Quix0r » Do Sep 09, 2010 7:47 pm

Ah, I see those JSON requests to many YaCy nodes. I was a little confused because Active Server Pages are installed on your server. Nice work. :)

If you like, you can add mine as well: http://free-search.homelinux.org:8080

This is my computer I'm typing this message, so it is only a customer/commerical ADSL connection with limited upload. But JSON should be fine.
Quix0r
 
Beiträge: 1345
Registriert: Di Jul 31, 2007 9:22 am
Wohnort: Krefeld

Re: a YaCy Search Portal with javascript: peer-search.net

Beitragvon Orbiter » Fr Sep 10, 2010 11:43 am

Nathan, the navigation in the search results using left/right arrow keys inspired me to have the same in the 'normal' YaCy search result page: this is really great and is very nice to present YaCy search speed. Added that with SVN 7134
Orbiter
 
Beiträge: 5796
Registriert: Di Jun 26, 2007 10:58 pm
Wohnort: Frankfurt am Main

Re: a YaCy Search Portal with javascript: peer-search.net

Beitragvon nstaudt » Fr Sep 10, 2010 1:32 pm

Quix0r hat geschrieben:Ah, I see those JSON requests to many YaCy nodes. I was a little confused because Active Server Pages are installed on your server. Nice work. :)

If you like, you can add mine as well: http://free-search.homelinux.org:8080

This is my computer I'm typing this message, so it is only a customer/commerical ADSL connection with limited upload. But JSON should be fine.


The list of peers that it searches is rebuilt by the peer-search.net web server every few minutes - it starts from a small list of known reliable peers and uses them to discover other available peers - so if your peer is A) online B) responds to web requests in under 1.5 seconds C) has more than a predefined list of links (currently 5,000,000) and has an uptime of more than half an hour (this is to help eliminate some "jitter" during peer updates), then it should make it onto the list!
nstaudt
 
Beiträge: 73
Registriert: Fr Aug 13, 2010 10:54 am

Re: a YaCy Search Portal with javascript: peer-search.net

Beitragvon nstaudt » Fr Sep 10, 2010 1:33 pm

Orbiter hat geschrieben:Nathan, the navigation in the search results using left/right arrow keys inspired me to have the same in the 'normal' YaCy search result page: this is really great and is very nice to present YaCy search speed. Added that with SVN 7134


Great, I'm glad you like it!! I sent you an email recently about peer-search.net - did you get it? :geek:
nstaudt
 
Beiträge: 73
Registriert: Fr Aug 13, 2010 10:54 am

Re: a YaCy Search Portal with javascript: peer-search.net

Beitragvon nstaudt » Fr Sep 10, 2010 4:15 pm

Would anybody be kind enough to offer to translate peer-search.net into German (or any other language)? I would like to make it more accessible :)
nstaudt
 
Beiträge: 73
Registriert: Fr Aug 13, 2010 10:54 am

Re: a YaCy Search Portal with javascript: peer-search.net

Beitragvon nstaudt » Di Sep 21, 2010 12:35 pm

modified the client script to always use resource=global, should see improved results.

fixed a couple of mistakes and added a mouse-over to search results for better visibility.
nstaudt
 
Beiträge: 73
Registriert: Fr Aug 13, 2010 10:54 am

Re: a YaCy Search Portal with javascript: peer-search.net

Beitragvon Quix0r » Di Sep 21, 2010 1:07 pm

Can you try to make your fixes "generic" so it can be applied on the source code?
Quix0r
 
Beiträge: 1345
Registriert: Di Jul 31, 2007 9:22 am
Wohnort: Krefeld

Re: a YaCy Search Portal with javascript: peer-search.net

Beitragvon nstaudt » Mi Sep 22, 2010 6:40 am

I'm sorry, I don't understand what you mean... I'm the one changing the source code on peer-search.net, that was just to let people know that something had changed. :)
nstaudt
 
Beiträge: 73
Registriert: Fr Aug 13, 2010 10:54 am

Re: a YaCy Search Portal with javascript: peer-search.net

Beitragvon Quix0r » Mi Sep 22, 2010 11:04 am

Ah, I thought you mean the YaCy code.
Quix0r
 
Beiträge: 1345
Registriert: Di Jul 31, 2007 9:22 am
Wohnort: Krefeld

Re: a YaCy Search Portal with javascript: peer-search.net

Beitragvon nstaudt » Mi Sep 22, 2010 3:19 pm

No worries m8 - I should have been more explicit :D

While I have your attention, would you mind heading over to peer-search.net and letting me know what you think about the results - do they seem to be what you would expect?

Thanks,
Nathan.
nstaudt
 
Beiträge: 73
Registriert: Fr Aug 13, 2010 10:54 am

Re: a YaCy Search Portal with javascript: peer-search.net

Beitragvon Quix0r » Mi Sep 22, 2010 3:37 pm

So far it looks good. :)
Quix0r
 
Beiträge: 1345
Registriert: Di Jul 31, 2007 9:22 am
Wohnort: Krefeld

Re: a YaCy Search Portal with javascript: peer-search.net

Beitragvon Juri » Mi Sep 22, 2010 9:01 pm

nstaudt hat geschrieben:modified the client script to always use resource=global, should see improved results.


It's very fast. But is it really peer-to-peer? Why every of the 15 peers queried by peer-search.net is returning a different result set (when observing the json in a http monitor)?
In a DHT a lookup initiated by any peer (at the same time) should return the same result set from the same peer (which is closest to the query hash).
Looks more like only local search results are returned, perhaps because of the 1.5 sec timeout?

Also the privacy claim on peer-search.net seems to be a bit misleading. Sure, the search request is not sent to peer-search.net, but then instead of one server there are 15 peers receiving (and potentially observing and logging) the unencrypted query.
Juri
 
Beiträge: 2
Registriert: Mi Sep 22, 2010 8:49 pm

Re: a YaCy Search Portal with javascript: peer-search.net

Beitragvon Orbiter » Mi Sep 22, 2010 9:59 pm

Juri hat geschrieben:In a DHT a lookup initiated by any peer (at the same time) should return the same result set from the same peer (which is closest to the query hash).
Looks more like only local search results are returned, perhaps because of the 1.5 sec timeout?

they must be different since every peer also uses the local index data also. the timeout is another reason which has definitely an effect but this is possibly not the main reason.

Nathan: using resource=global may be correct for some targets but not for all targets. If more that one target gets a global=true then these two peers will possibly select the same target peers for remote search which will cause that those peers must respond two times and that may cause that search time increases. That means: the more peers are asked simultaneously with resource=global, the longer the response time will be because they will block each other.

I would suggest to use only a very small number of target peers with resource=global, all others with resource=local.
Orbiter
 
Beiträge: 5796
Registriert: Di Jun 26, 2007 10:58 pm
Wohnort: Frankfurt am Main

Re: a YaCy Search Portal with javascript: peer-search.net

Beitragvon Juri » Mi Sep 22, 2010 11:11 pm

Orbiter hat geschrieben:
Juri hat geschrieben:In a DHT a lookup initiated by any peer (at the same time) should return the same result set from the same peer (which is closest to the query hash).
Looks more like only local search results are returned, perhaps because of the 1.5 sec timeout?

they must be different since every peer also uses the local index data also. the timeout is another reason which has definitely an effect but this is possibly not the main reason.

Sure, but the highest ranked results should generally come from the global search from the same peer (which is closest to the query hash),
this is what structured DHT search is all about (as opposed to an unstructured Gnutella like search).
Those highest ranked results should then be returned by all of the 15 peers (perhaps supplemented with some individual, less relevant local results).
But there seems to be no overlap at all (same link / pubdate) between the results of any of the 15 peers ( e.g. for query=peer ) ?
Juri
 
Beiträge: 2
Registriert: Mi Sep 22, 2010 8:49 pm

Re: a YaCy Search Portal with javascript: peer-search.net

Beitragvon nstaudt » Do Sep 23, 2010 6:33 am

I remove any duplicate results on the client, though I have noticed that if the same url has been indexed as both www.abc.com and abc.com, then yacy doesn't return the same hash for the result and therefore I can't easily spot the duplicate.
nstaudt
 
Beiträge: 73
Registriert: Fr Aug 13, 2010 10:54 am

Re: a YaCy Search Portal with javascript: peer-search.net

Beitragvon nstaudt » Do Sep 23, 2010 6:42 am

Juri hat geschrieben:Also the privacy claim on peer-search.net seems to be a bit misleading. Sure, the search request is not sent to peer-search.net, but then instead of one server there are 15 peers receiving (and potentially observing and logging) the unencrypted query.


Hi Juri, thanks for your feedback :)

peer-search.net hat geschrieben:Your search request is not sent to a central web server (that's right, not even to peer-search.net!!): YaCy is a peer to peer search engine.
This means that it's very difficult for your search results to be censored or used for marketing/advertising analysis!
You may also wish to consider using a Tor client to further protect yourself.


If you're worried about privacy, there's already a tool for that, you should be using Tor.

peer-search.net + yacy combine to make it difficult to censor and/or analyse search results (you would have to control a large number of peers in the network before you could offer meaningful analysis of search behaviour or alter the results - a task that becomes harder as the network grows).

I hope that answers your question: peer-search.net doesn't claim to improve your privacy, it even offers a link to a tool that does :)

Regards,
Nathan.
nstaudt
 
Beiträge: 73
Registriert: Fr Aug 13, 2010 10:54 am

Re: a YaCy Search Portal with javascript: peer-search.net

Beitragvon nstaudt » Do Sep 23, 2010 5:52 pm

I've added a blog (http://www.peer-search.net/blog) to peer-search.net - if any yacy contributors would like to make a guest entry there about their experiences of contributing to YaCy (or anything else for that matter), you are very welcome - just send me a private message on this forum with your email and I'll get in touch. :)
nstaudt
 
Beiträge: 73
Registriert: Fr Aug 13, 2010 10:54 am

Re: a YaCy Search Portal with javascript: peer-search.net

Beitragvon nstaudt » Fr Okt 01, 2010 4:23 pm

I've updated peer-search.net so that the results from different peers are interleaved and prioritised according to the number of links a peer has and whether or not the request to that peer included the global p2p index or just a local search.

Results should be more accurate now (and less affected by network latency) - please try it out and let me know how you like it :)
nstaudt
 
Beiträge: 73
Registriert: Fr Aug 13, 2010 10:54 am

Re: a YaCy Search Portal with javascript: peer-search.net

Beitragvon Copro » Mi Okt 06, 2010 1:02 am

I absolutely love the design and the speed ;-)
If you still need help with the German translation just give me a shout ... 'd love to help out ;)
Copro
 
Beiträge: 207
Registriert: Do Dez 13, 2007 3:37 am

Re: a YaCy Search Portal with javascript: peer-search.net

Beitragvon nstaudt » Mi Okt 06, 2010 6:53 am

Copro hat geschrieben:I absolutely love the design and the speed ;-)
If you still need help with the German translation just give me a shout ... 'd love to help out ;)


Many thanks! I sent you a PM about the translation :)
nstaudt
 
Beiträge: 73
Registriert: Fr Aug 13, 2010 10:54 am

Re: a YaCy Search Portal with javascript: peer-search.net

Beitragvon nstaudt » Fr Okt 08, 2010 12:56 pm

I've added a new function to my site: visitors may add domains for yacy to crawl: http://www.peer-search.net/submit.aspx :D

Currently, this just adds the domains to my peer with remoteindexing = true, but if anyone would like to add their peer so that it also receives domains to crawl, please send me a private message with your peer details (don't forget to create a user account that may create crawls - and include the user details in the PM).
nstaudt
 
Beiträge: 73
Registriert: Fr Aug 13, 2010 10:54 am

Re: a YaCy Search Portal with javascript: peer-search.net

Beitragvon Quix0r » Fr Okt 08, 2010 1:15 pm

Nice idea but I would use a seperate database where the submitted URLs are stored in a separate database for moderation and if you think the webpage is okay, it will be added to YaCy (with remoteIndexing=true again).
Quix0r
 
Beiträge: 1345
Registriert: Di Jul 31, 2007 9:22 am
Wohnort: Krefeld

Re: a YaCy Search Portal with javascript: peer-search.net

Beitragvon nstaudt » Fr Okt 08, 2010 1:37 pm

The submitted sites are stored on the webserver first and then every 5 to 10 minutes the webserver will attempt to contact my peer. (that way if my peer is offline for some reason, the domain will be submitted when it comes back online)

The crawl profile which is created is limited to the submitted domain. If one particular submission is taking too long to complete, I'll just delete it from my peer.

Ideally, I would like to have a few other senior peers accept these submitted domains so the load is spread out a bit better. What do you think of that?
nstaudt
 
Beiträge: 73
Registriert: Fr Aug 13, 2010 10:54 am

Re: a YaCy Search Portal with javascript: peer-search.net

Beitragvon nstaudt » Mo Okt 11, 2010 12:29 pm

Thanks to Copro, http://peer-search.net is now available in German (still a few small strings to fill in, but mostly there)

Many Thanks!! :D
nstaudt
 
Beiträge: 73
Registriert: Fr Aug 13, 2010 10:54 am

Re: a YaCy Search Portal with javascript: peer-search.net

Beitragvon Copro » Di Okt 12, 2010 12:13 am

Hehe - reading some of my not so brilliant translation in context gives me the giggles :o
During the translation I also first saw that I can navigate between search pages with the keyboard arrow keys <- and ->
Are those "pages" different peer results or just the normal accumulated results from different peers split over multiple pages ?
Ah, and btw. on the German About page you linked all three links (Email, Blog, Twitter) back to the About page instead of the external pages. Good work tho - still a very nifty way to use YaCy :geek:
Copro
 
Beiträge: 207
Registriert: Do Dez 13, 2007 3:37 am

Re: a YaCy Search Portal with javascript: peer-search.net

Beitragvon nstaudt » Di Okt 12, 2010 9:05 am

Thanks for having a look at it - I've fixed those links and added the additional items you translated last night.

Each page is a merged result from the different peers that have responded - so the first page is made up of the first result from each peer, the second page is the second result and so on. :)
nstaudt
 
Beiträge: 73
Registriert: Fr Aug 13, 2010 10:54 am

Re: a YaCy Search Portal with javascript: peer-search.net

Beitragvon Orbiter » Sa Sep 10, 2011 11:23 am

peer-search.net is unfortunately switched off :( :( :(
Nathan wrote to me that he cannot maintain the site due to personal and also financial circumstances. I hope he is well an I thank hin very much for his work. This javascript-side implementation was very inspiring and I hope that sometime someone will catch up with that idea.
Orbiter
 
Beiträge: 5796
Registriert: Di Jun 26, 2007 10:58 pm
Wohnort: Frankfurt am Main


Zurück zu Mitmachen

Wer ist online?

Mitglieder in diesem Forum: 0 Mitglieder und 1 Gast