Index Transfer

Discussion in English language.
Forumsregeln
You can start and continue with posts in english language in all other forums as well, but if you are looking for a forum to start a discussion in english, this is the right choice.

Index Transfer

Beitragvon ktplulo » Di Jul 15, 2014 3:47 pm

Where is the former IndexTransfer_p feature (to send all the index to other nodes) now?
ktplulo
 
Beiträge: 18
Registriert: Do Mär 01, 2012 11:27 am

Re: Index Transfer

Beitragvon ktplulo » Di Jul 15, 2014 6:40 pm

commit 04ec42e1d0564feeb0de067af621ce7cc7198bf0
Author: orbiter <orbiter@6c8d7289-2bf4-0310-a012-ef5d649a1542>
Date: Sun May 10 21:00:39 2009 +0000

removed index Transfer, because it does not make sense any more to do a complete index transfer using the DHT methods. It is far easier and faster to just copy the whole index files from the source to the target peer. To do this, copy all files from DATA/INDEX/<network>/TEXT/RICACHE and DATA/INDEX/<network>/TEXT/METADATA from the source to the destination peer.
Future versions of YaCy will have different file structures than given here, because of the new index segment concept, where each peer may have several indexes (called segments) for each network definition. A copy from one peer to another will then create a new index segment.
To track changes for this feature, see forum discussion at
viewtopic.php?p=14728#p14728

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5945 6c8d7289-2bf4-0310-a012-ef5d649a1542
ktplulo
 
Beiträge: 18
Registriert: Do Mär 01, 2012 11:27 am

Re: Index Transfer

Beitragvon ktplulo » Di Jul 15, 2014 6:41 pm

I am shutting my peer down. Should I just disconnect it from freeworld?
ktplulo
 
Beiträge: 18
Registriert: Do Mär 01, 2012 11:27 am

Re: Index Transfer

Beitragvon David » Di Jul 15, 2014 6:53 pm

If you want to copy or move your index, you have to shut down yacy.
David
 
Beiträge: 170
Registriert: Di Mär 05, 2013 5:35 pm

Re: Index Transfer

Beitragvon ktplulo » Di Jul 15, 2014 6:56 pm

I'm thinking of what will happen to the crawl results of the recent days (or weeks). Have they all been sent out to the global network already? Or should I do something to force them to be sent out faster?
ktplulo
 
Beiträge: 18
Registriert: Do Mär 01, 2012 11:27 am

Re: Index Transfer

Beitragvon David » Di Jul 15, 2014 7:08 pm

If yacy is crawling, then the distribution of the index data is automatically slowed down. So, if you want to make sure that most of your index is distributed to other peers, you have to stop crawling and keep your peer running for a few more days or weeks. It also depends on how big your index is.
David
 
Beiträge: 170
Registriert: Di Mär 05, 2013 5:35 pm

Re: Index Transfer

Beitragvon David » Di Jul 15, 2014 7:16 pm

If you want to make sure that your work wasn't in vain, you could also upload your SEGMENTS-folder (yacy/DATA/INDEX/freeworld) to a website like mediafire.com and then post the download links here. Maybe someone will grab it and merge it with his own.
David
 
Beiträge: 170
Registriert: Di Mär 05, 2013 5:35 pm

Re: Index Transfer

Beitragvon ktplulo » Sa Jul 19, 2014 2:21 pm

OK, thanks, David.
Zuletzt geändert von ktplulo am Sa Jul 19, 2014 2:53 pm, insgesamt 1-mal geändert.
ktplulo
 
Beiträge: 18
Registriert: Do Mär 01, 2012 11:27 am

Re: Index Transfer

Beitragvon ktplulo » Sa Jul 19, 2014 2:36 pm

____
Zuletzt geändert von ktplulo am Sa Jul 19, 2014 3:12 pm, insgesamt 2-mal geändert.
ktplulo
 
Beiträge: 18
Registriert: Do Mär 01, 2012 11:27 am

Re: Index Transfer

Beitragvon ktplulo » Sa Jul 19, 2014 2:52 pm

Apparently the "Dump and Restore of Solr Index" feature overwrites the whole index: viewtopic.php?f=5&t=4615&p=27690

Merging two indices needs 3rd-party (or less easy) tools: viewtopic.php?f=5&t=4857&p=28763
ktplulo
 
Beiträge: 18
Registriert: Do Mär 01, 2012 11:27 am

Re: Index Transfer

Beitragvon ktplulo » Sa Jul 19, 2014 3:10 pm

Would it make sense, and is it possible, to use DHT Words from one network (freeworld) in another one (webportal mode)?
ktplulo
 
Beiträge: 18
Registriert: Do Mär 01, 2012 11:27 am

Re: Index Transfer

Beitragvon ktplulo » Sa Jul 19, 2014 3:12 pm

If I revert my profile to an old copy, will the DHT Words be distributed in the same order (mostly wasting time), or is it randomized?
ktplulo
 
Beiträge: 18
Registriert: Do Mär 01, 2012 11:27 am

Re: Index Transfer

Beitragvon David » So Jul 20, 2014 1:30 pm

ktplulo hat geschrieben:Merging two indices needs 3rd-party (or less easy) tools: viewtopic.php?f=5&t=4857&p=28763

Oh, ok. But you can merge the RWI (reverse word index), which gets created in parallel to the solr-index, by simply copying the files to the folder:
/DATA/INDEX/freeworld/SEGMENTS/default/
Just make sure to shut down yacy first.

ktplulo hat geschrieben:Would it make sense, and is it possible, to use DHT Words from one network (freeworld) in another one (webportal mode)?

As far as I know, if you switch from the p2p-mode to the webportal mode, the RWI gets deactivated, since this index is mainly used for the distribution and receiving of index data from other peers, and in the webportal mode you don't communicate with other peers.

ktplulo hat geschrieben:If I revert my profile to an old copy, will the DHT Words be distributed in the same order (mostly wasting time), or is it randomized?

I think it happens in a random fashion.
David
 
Beiträge: 170
Registriert: Di Mär 05, 2013 5:35 pm

Re: Index Transfer

Beitragvon Orbiter » So Jul 20, 2014 10:49 pm

some month ago I implemented an import method which stretches out to the complete Solr Schema within the so-called surrogate input process. That means that this process can now read in complete Solr XML exports if there would be any process that produces such output... I will implement that if there is time left which is not at this time. With such an full Solr-to-XML export it would then be possible to merge indexes again in the same way as it was possible in the past before the usage of Solr.
Orbiter
 
Beiträge: 5798
Registriert: Di Jun 26, 2007 10:58 pm
Wohnort: Frankfurt am Main


Zurück zu English

Wer ist online?

Mitglieder in diesem Forum: 0 Mitglieder und 2 Gäste

cron