Config pls New Cloud Group + Migration of Robinson Servers

Discussion in English language.
Forumsregeln
You can start and continue with posts in english language in all other forums as well, but if you are looking for a forum to start a discussion in english, this is the right choice.

Config pls New Cloud Group + Migration of Robinson Servers

Beitragvon xioc752 » Di Dez 30, 2014 2:11 pm

Hello.
Please we are setting up a private group of new YaCy servers which will be in full P2P with DHT. We need your ' seasoned ' advice, please. Thanks.

1. We need the Current Best Practices Instructions on prefered configuration for this new group of cloud servers, starting with 3 new servers that are ' UP ' and functioning now in stand alone Robinson mode.
We will continue to add more servers to the private group.
The servers in the new group are currently identical to each other.
The servers can currently read each other's Solr files, but they cannot write to each other.
This means the the new cloud servers were set up, one by one, in Robinson mode with cross reading privileges between servers.
Now they need to be changed to the fully shared and standard YaCy environment for private groups of servers working together.
We wish these to be in the full DHT P2P mode as a private network.

We appreciate this requires more space and we have cloud computing and storage capacity assigned for this - in and across the clouds.

+
2. We have 2 prior Robinson servers that need to be migrated to this new group, described above.
They each have somewhat less than 20 GB of DATA Sets, each.
This data must be retained and moved to the new P2P DHT environment safely.

We understand that the function of the environment is such that as this data is migrated, duplicates will be eliminated as part of the standard indexing process.
Is this correct?

How do we safely change their Robinson configurations to become part of the new private DHT P2P group.
Essential is that they not lose the data they have collected and indexed.

3. As noted: Our ultimate goal is to migrate the data the earlier two Robinson servers have independently collected and move it to the pool of data among the others - in the framework of the new collective, described above.
The result is that there will no longer be any Robinson servers (-0-).
All servers will be in a secured private P2P group with DHT.
All 5 servers (initially) will become part of the private pool of servers in full P2P with DHT - but as a stand alone private network of cloud servers with appropriate security features protecting it from the outside.

The environment will be fed data, and read by standard approaches to a YaCy server. We are assuming that this distributed computational environment will create a kind of load sharing for crawling and search results generation.
Traffic is still light.

Thank you for your careful, precise and detailed instructions and guidance. No detail is too small, please. Please Assume that we do not know what may seem obvious to you. Thanks for your patience! :)
PMs are very OK, too. Many Thanks, once again!
xioc752
 
Beiträge: 68
Registriert: Mo Jul 28, 2014 5:01 pm

Re: Config pls New Cloud Group + Migration of Robinson Serve

Beitragvon smokingwheels » Sa Jan 17, 2015 5:27 am

What have you tried so far?
Export > Import ?

I hope your plan has HDD's no SSD's or you will run out of space.
smokingwheels
 
Beiträge: 136
Registriert: Sa Aug 31, 2013 7:16 am

Re: Config pls New Cloud Group + Migration of Robinson Serve

Beitragvon xioc752 » Sa Jan 17, 2015 4:07 pm

HI and thanks for your reply.

All the servers are cloud based.
Hence, there is always a limitation on the internal size available for the ubuntu + YaCy dual Solr cores.

Our concept is to use individual servers with their own internal dual Solr cores, and connect them together in P2P + DHT to achieve maximum search and indexing speeds - shared across multiple individual clouds.
This will also facilitate failback and recover, later.

Our problem now is that we have multiple servers that should be sharing but are in fact acting like Robinson servers that read each other.
Hence, our migration from cross-reading Robinson servers to a full P2P + DHT group is incomplete.
Our network is the original
defaults/yacy.network.allip.unit

Likely this needs to be changed, but how to do it and not lose our many GB of data?
This is the only 'selectable' when starting our kind of environment...which we started in cross-reading Robinson servers.
We suspect that the network ID needs to be changed, but we do not know how to do it safely.
Many thanks!
xioc752
 
Beiträge: 68
Registriert: Mo Jul 28, 2014 5:01 pm

Re: Config pls New Cloud Group + Migration of Robinson Serve

Beitragvon Orbiter » Mo Feb 02, 2015 3:40 pm

you can move your index from intranet/webportal (whatever you use) to allip if you just rename the folder in DATA/INDEX/
Do:
- switch to allip
- shutdown
- open DATA/INDEX/
you will notice subdirectories freeworld, intranet, webportal, allip (or a subset of that):
- You must rename the folder containing your previous index to allip
- start
Orbiter
 
Beiträge: 5792
Registriert: Di Jun 26, 2007 10:58 pm
Wohnort: Frankfurt am Main

Re: Config pls New Cloud Group + Migration of Robinson Serve

Beitragvon xioc752 » Mo Feb 02, 2015 5:45 pm

HI and Thank you for your reply!

In fact, all our servers are already using the ALLIP labels.
Yet, they are a collection of physically separate Robinson servers that read each other but do not write to each other.

They also do not have a separate network name.
We used the name presented in the generic setup.
All the servers are identical.
We assume this a security risk that needs to be corrected.
We cannot find a sample file + instructions for creating a new network name using existing data sets across multiple and existing - stand-alone - Robinson servers.

We need to find instructions please how to change this 'early' environment to full p2p with DHT, please.
Many thanks
xioc752
 
Beiträge: 68
Registriert: Mo Jul 28, 2014 5:01 pm


Zurück zu English

Wer ist online?

Mitglieder in diesem Forum: 0 Mitglieder und 2 Gäste

cron