YaCy in BOINC environment

Hier finden YaCy User Hilfe wenn was nicht funktioniert oder anders funktioniert als man dachte. Bei offensichtlichen Fehlern diese bitte gleich in die Bugs (http://bugs.yacy.net) eintragen.
Forumsregeln
In diesem Forum geht es um Benutzungsprobleme und Anfragen für Hilfe. Wird dabei ein Bug identifiziert, wird der thread zur Bearbeitung in die Bug-Sektion verschoben. Wer hier also einen Thread eingestellt hat und ihn vermisst, wird ihn sicherlich in der Bug-Sektion wiederfinden.

YaCy in BOINC environment

Beitragvon krzyszp » So Jan 12, 2014 2:53 pm

Hi,

I'm currently involved in few BOINC based projects and would like to use YaCy as client application for one of it, but there is few problems.
I would like to create virtual machine with (let's say) 5GB of space and distribute it to min. 500-800 users and more than 1000 computers. Unfortunately, YaCy require port forwarded to it.
I know, that I can use proxy for it - but how to do it?
Also, 5GB is not a large space for index - so the question is if it will be useful for YaCy network?
Other solution possible is to fork YaCy as BOINC client application, but I don't have enough knowledge to do it...

Anyway. It will be very helpfully to create English section on forum, as most users don't speak German...
krzyszp
 
Beiträge: 21
Registriert: So Jan 12, 2014 2:40 pm

Re: YaCy in BOINC environment

Beitragvon Yududi » So Jan 12, 2014 6:24 pm

Is this question related to YaCy or to BOINC?
I just know how distributed computing works but dont know how the tool BOINC was created by their developers.
Yududi
 
Beiträge: 64
Registriert: Di Dez 10, 2013 12:30 pm

Re: YaCy in BOINC environment

Beitragvon krzyszp » So Jan 12, 2014 6:48 pm

Is related to both ;)

I would like just to get answer from YaCy developers how to solve lack of static IP's on client computers (through proxy? other way?) and get feedback if this have any sense to create loads of peers with limited space on their hdd's.
krzyszp
 
Beiträge: 21
Registriert: So Jan 12, 2014 2:40 pm

Re: YaCy in BOINC environment

Beitragvon Orbiter » So Jan 12, 2014 11:27 pm

Hi krzyszp,
The usage of BOINC is a very good idea in principle, but there is a difference in between what BOINC is (as far as I know) an what YaCy does: BOINC is distributed computation and YaCy is about distributed storage. From your post I learn that BOINC has storage limitations for their projects? If yes, then we would need to plan what is the right way to handle that situation, it would be very similar to the YaCyPi project where there is just limited space on a SD card.

The other problem you address is the port forwarding task the user has to do. We have a uPNP component in YaCy but I believe in most cases it does not work. There should be a development team around that problem to solve it: is there anybody who has experience with uPNP?
Orbiter
 
Beiträge: 5792
Registriert: Di Jun 26, 2007 10:58 pm
Wohnort: Frankfurt am Main

Re: YaCy in BOINC environment

Beitragvon krzyszp » So Jan 12, 2014 11:50 pm

BOINC haven't storage limitations itself but as BOINC project maintainer I would like to keep users save from unexpected computers behaviour (e.g. fill system drive with only YaCy data). Also because YaCy isn't simple application and need to be installed I found that simplest way is to provide Virtualbox machine with YaCy preinstalled and with space limit (5GB in this example, but of course I'm opened for discussion).
There is two main problems IMHO:
1. Opening 8090 (or other) port in network - also, some users can setup few computers in their network, so YaCy should find other instances in local network. UPNP doesn't work in my home network, so I found this service as unreliable. Possible solution will be make YaCy independent from port forwarding and truly P2P - but I don't have knowledge how to do this.
2. Reliability and stability. Unfortunately, on virtual machine created by me, YaCy not always start correctly... Is any chance that YaCy developers will try to play with Virtualbox to create reliable YaCy server?

Keep in mind, that in this circumstances we not require any front or back-end for it - all parameters can be hard coded in virtual machine or (better) will be downloaded from our (my) server and YaCy will be prepared only as remote crawler. I hope this can simplify developers work...
krzyszp
 
Beiträge: 21
Registriert: So Jan 12, 2014 2:40 pm

Re: YaCy in BOINC environment

Beitragvon Orbiter » Mo Jan 13, 2014 12:19 am

krzyszp hat geschrieben:BOINC haven't storage limitations itself but as BOINC project maintainer I would like to keep users save from unexpected computers behaviour (e.g. fill system drive with only YaCy data). Also because YaCy isn't simple application and need to be installed I found that simplest way is to provide Virtualbox machine with YaCy preinstalled and with space limit (5GB in this example, but of course I'm opened for discussion).

Thats a good way to handle that.

krzyszp hat geschrieben:There is two main problems IMHO:
1. Opening 8090 (or other) port in network - also, some users can setup few computers in their network, so YaCy should find other instances in local network. UPNP doesn't work in my home network, so I found this service as unreliable. Possible solution will be make YaCy independent from port forwarding and truly P2P - but I don't have knowledge how to do this.

"truly P2P" is always a 'hack' to tunnel 'somewhere'. We have a concept to use other YaCy peers as gateways for this (the Mentor and Mentee concept) but there was no time yet to implement that. However, this would also rely on 'true' senior peers.

krzyszp hat geschrieben:2. Reliability and stability. Unfortunately, on virtual machine created by me, YaCy not always start correctly... Is any chance that YaCy developers will try to play with Virtualbox to create reliable YaCy server?

I tried that for some time, but the main problem is here not stability but the size of the VM image; I did not succeed in making images below 500MB. I know that this is not a big problem to download that, but it is a big problem for me to upload this to a download location ... :x

krzyszp hat geschrieben:Keep in mind, that in this circumstances we not require any front or back-end for it - all parameters can be hard coded in virtual machine or (better) will be downloaded from our (my) server and YaCy will be prepared only as remote crawler. I hope this can simplify developers work...

Yes, it's a good concept.
Orbiter
 
Beiträge: 5792
Registriert: Di Jun 26, 2007 10:58 pm
Wohnort: Frankfurt am Main

Re: YaCy in BOINC environment

Beitragvon krzyszp » Mo Jan 13, 2014 12:30 am

So, then we are on point where we approve, that (maybe) good solution will be a tunnel to my 'main' YaCy server?
Also, as I seed - target vm will be 5GB file at the start and will be same for all clients.

I will really appreciate if YaCy developers helps to create this vm image. Also I have space on my ftp to keep it (always you can use torrent to upload it, as it is not a commercial/closed software).
krzyszp
 
Beiträge: 21
Registriert: So Jan 12, 2014 2:40 pm


Zurück zu Fragen und Antworten

Wer ist online?

Mitglieder in diesem Forum: Google [Bot] und 1 Gast

cron