A Peer To Peer Web Directory To Go With YACY

For YaCy users and applications in a professional environment.

A Peer To Peer Web Directory To Go With YACY

Beitragvon Bungeebones » Do Jul 18, 2013 4:01 pm

Hi all,

I just discovered Yacy because of a BBC article http://www.bbc.co.uk/news/technology-23318889

I've built a proof-of-concept of a combination peer-to-peer and central server web directory that would work nicely with YaCy.

To become a node they download the web directory script for free from BungeeBones.com when they add their link (also for free but better placement in competitive categories can be gotten with paid links via Bitcoin). The node displays a branded version of the web directory that is sent complete with categories, links and management from the server. It provides a free, maintenance free web directory as a feature for a web site.

I know in the early history of the web Yahoo (a web directory) was king and search engines (even Google) were contracted by Yahoo to step in when users didn't find what they were looking for (as a sort of last-ditch effort). Obviously that has changed but despite Google's monopoly on search there are still millions of web directories maintained by small sites like mine. I believe my approach makes more sense than millions of dissociated web directories . Instead of submitting my link to those "millions" one at a time I can submit it once and the server distributes it to the "millions" for me. And instead of a million maintainers inspecting the same links millions of times they are reviewed once for everyone.

I don't believe web directories have become irrelevant. Even Google's sophisticated algorithm has its problems and there is still something to be said for human reviews (which I perform on my directory). I believe Yahoo's directory would still be a player if they had only abandoned their flat fee ($300/yr) structure in favor of a more scaled fee structure (as Pay Per Click has masterfully done).

So one thing I have done is create a unique bidding process that lets web sites in congested categories establish their position across multiple matrices (such as continent, country, "state", and city) as well as category. This enables exact pricing precision with high efficiency and eliminates many a bottle neck at the category level.

The payments are paid out as commissions to the sites hosting the web directory in a manner similar to Adsense. Instead of a placing links alongside your own content you place the web directory on it's own page (off by itself and not trying to lure your visitor away with a "better deal").

The site is 95% complete. I am still working on the reporting of registrants and payments to the hosts and that should be completed very soon. I do need help in testing however which would entail installing the web directory script on web sites as hosts. I also will provide some "bounty" in the form of advertising vouchers for anyone that merely adds their web site link which will enable their link to jump ahead of previously registered links.

As a result of that same BBC link above, I earlier today visited Blekko.com. Their search engine performs "curated results and organizing content into categories". For a while I was including a Google search window on my web directory pages that would pre-load the visitor's current location into the search window. For example, if they had drilled down a few layers in the automotive category it would place in the search box their location Automotive -> Dealers -> New -> California -> Los Angeles. I removed it sometime ago but could do the same with a Yacy search window.

I am curious if Yacy is doing anything with their algorithm along the same lines? What I see possibly developing is a human reviewed network of thousands of web directories with those type of "long tail" descriptions that could greatly help an algorithm. I've built in multiple inspection points of the sites not just by me but also by those adding links. For example, I list previously entered links in the category you would be adding your link to. They are, essentially, your competitors so if you are adding as a free link you have a vested interest in their accuracy because they will be listed before you. By investing a little time in previewing them you may discover a site or two that are mis-categorized. You could elimate them by reporting them resulting in them getting moved which moves your link listing higher as a result.

I've looking forward to a lot of reading and testing with Yacy. I'm hoping I can set it up on my server and have it index all the sites submitted to the directory and then sharing that with the rest of the nodes. Through my quick preliminary read it sounds like Yacy can do that which is great if it does. I currently have over 3000 links from all around the world.
Beiträge: 1
Registriert: Do Jul 18, 2013 2:19 pm

Re: A Peer To Peer Web Directory To Go With YACY

Beitragvon David » Fr Jul 19, 2013 6:09 pm

Hi! Your project seems to be pretty interesting, although I have to say, that I'm not really a fan of "freemium services", where paying users have an advantage over the other users. Especially in web directories and search engines, the ranking should only be influenced by factors like quality, popularity etc. However, that's just my opinion. Nonetheless, I wish you good luck for your project, and if you need help or assistance using yacy, feel free to ask.
Beiträge: 170
Registriert: Di Mär 05, 2013 5:35 pm

Re: A Peer To Peer Web Directory To Go With YACY

Beitragvon lux » Sa Jul 20, 2013 7:57 am

Hi Bungeebones,

I endorse David's opinion. YaCy's concept is a noncommercial peer-to-peer network. Your proposal to combine Yacy with a central server sounds nice, but is in contradiction to YaCy being a distributed, uncensored search-engine. Also, a central node in a network always is a weak point regarding its vulnerability.

Finally, those people who want to be served by a big central search-engine should stay with Google.

Just my opinion
Beiträge: 41
Registriert: Sa Jun 01, 2013 9:49 am

Zurück zu Pro-Users

Wer ist online?

Mitglieder in diesem Forum: 0 Mitglieder und 1 Gast