how a new website is referenced into yacy-p2p/dht-cloud ?

Discussion in English language.
You can start and continue with posts in english language in all other forums as well, but if you are looking for a forum to start a discussion in english, this is the right choice.

how a new website is referenced into yacy-p2p/dht-cloud ?

Beitragvon paleolas » Mi Aug 02, 2017 7:53 pm

Hi everyone,

what if someone, creates a website.
how this new website could be referenced by yacy-p2p-cloud ?

The video "Search Result Processing and Fraud Protection" explain well the internal process of searching some website and how the rank is build.
in my limited knowledge of Yacy, when we do a resarch the engine search on (1) local cache database and (2) yacy-p2p database. It then create a rank, and after that go get the real information directly to the ranked websites then print the result.
what if the website is neither on n°1 local cache nor the n°2 yacy-p2p database ?
in other words : how a unknown website enters into the yacy-p2p database ?
in other words : what is the link between the creation of a new website and its presence into one yacy search result ?

thank you for your answers.
Beiträge: 1
Registriert: Mi Aug 02, 2017 7:24 pm

Re: how a new website is referenced into yacy-p2p/dht-cloud

Beitragvon smokingwheels » Do Aug 03, 2017 12:07 pm

Not really sure what you mean but you can have your peer in an iframe on a web site then have it crawled by other search engines by adding it to there index to be crawled. I dont really know much about SEO.
See /ConfigPortal_p.html for some examples. You need to setup some tags in the header and the file robots.txt

I have a program (somewhere) to generate hyper links on a web page from just suppling URL's.
Beiträge: 129
Registriert: Sa Aug 31, 2013 7:16 am

Re: how a new website is referenced into yacy-p2p/dht-cloud

Beitragvon luc » Do Aug 10, 2017 11:21 am

Hello paleolas,
from the moment a web resource is indexed in at least one YaCy peer reachable from other peers in the 'freeworld' network, one can consider that this resource has entered the YaCy peer-to-peer network.
But from that moment, if you pick a random YaCy peer and search for a term present in that web resource, this one will not necessarily appears immediately in the results, as the corresponding index entry has not yet been propagated to other peers.

As I see it, there are some key steps for a web resource initially unkown to the YaCy p2p network to then appear in most peers search results.

First indexing.
Here are some possible example scenarios :
- someone running a YaCy node explicitely crawls that unkown web resource. This can eventually be the owner of the resource that would like it appears in the YaCy network.
- during a crawl on a YaCy node, a resource linking to the unkown web resource is encountered, and the crawl profile permits following that link and continue crawling on it
- someone using the YaCy proxy navigation feature encounter a resource linking to the unknown resource and follows the link
- the unknown resource is indexed by a website or search engine supporting the OpenSearch specification, that last one is configured as an alternative source by a YaCy peer (in its Heuristics configuration), and someone searches on that YaCy peer for term(s) present in the unknown web resource.

For all these examples, the prerequisite is that the initially unkown web resource is not blocked by each considered YaCy peer blacklist.
One can also easily see that except when a YaCy peer owner already knows the new resource, the more resources links to it, the more there are chances for a new resource to be introduced in the YaCy p2p network.

Again, some possible scenarios :
- the new index entry is distributed when the first YaCy peer which holds it applies the scheduled background distribution algorithm and eventually transfers it to another peer
- someone searches on a YaCy peer for term(s) present in the new index entry, the peer that holds it is selected by the remote peers selection algorithm, and the new entry is duplicated on the searching peer local data. The peers selection algorithm does not necessarily selects all peers holding index entries matching the searched term(s), but likely only a subset of them. So even if the new entry contains popular terms, it will not be immediately distributed this way.

Again, during propagation, each peer custom blacklist rules apply. So in a situation where every YaCy peers blacklist blocked the new entry, it would never be propagated.
We also see here that there are more chances for the new entry to be propagated if the first peer that holds it runs continuously 24 hours a day.

Ok, so now multiple YaCy peers have a copy of the new entry in their own local index or in their own part of the globally distributed index. If you search for term(s) present in the new resource on each of them, the resource entry will probably be each time in a different position in the search results, because :
- each peer can define its own custom ranking rules
- each peer has a different local index and local parts of the globally distributed index

Maintaining in indexes.
Of course each YaCy peer has only a limited amount of dedicated disk space available. To prevent unlimited index growing size, some may be configured to regularly delete the oldest entries. In the end, if no peer at all recrawl our newly indexed entry, it may eventually completely disappear from the network.
So to maintain its presence in the YaCy p2p network, a given resource must be regularly recrawled, by at least one peer, explicitely or indirectly, with the same possible scenarios as described in the 'First indexing' step.
Again, the various custom blacklists are to consider, and the more resources links to the resource, the more there are chances for it to be regularly recrawled and thus maintained in the YaCy p2p network.

I hope I didn't forget too many important points, and it will help having a more clear view of the global process.

Have a nice day
Beiträge: 285
Registriert: Mi Aug 26, 2015 1:04 am

Zurück zu English

Wer ist online?

Mitglieder in diesem Forum: 0 Mitglieder und 2 Gäste