Using YaCy with centralized storage

Forum for developers

Using YaCy with centralized storage

Beitragvon mr_aliagha » So Mai 11, 2014 10:52 am

Dear YaCy developers,
Hi,
I recently got familiar with YaCy. From what I saw, I think YaCy is strong distributed crawler. But I want to make some changes to YaCy based on my demands. As far as I found out YaCy uses a embedded storage in each peers. Since I going to run a distributed local search portal using YaCy I need to have a central database (Relational or NOSQL ones) for all local peers. Could you please give me some hints about how that is possible with YaCy?
Regards.
mr_aliagha
 
Beiträge: 6
Registriert: Mo Mai 05, 2014 9:49 pm

Re: Using YaCy with centralized storage

Beitragvon sixcooler » So Mai 11, 2014 11:51 am

Hello mr_aliagha,

I think what you whant could be done by using an external Solr as central DB and attach your YaCy-Peers to that Solr.
Have a look at your /IndexFederated_p.html - there ist also a wiki-page http://www.yacy-websearch.net/wiki/index.php/Dev:Solr describing a solr-setup.

cu, sixcooler.
sixcooler
 
Beiträge: 494
Registriert: Do Aug 14, 2008 5:22 pm

Re: Using YaCy with centralized storage

Beitragvon mr_aliagha » So Mai 11, 2014 11:58 am

Thank you very much for you reply. But how can I use external storage for the pages that are fetched? I mean before do any parsing?
Regards.
mr_aliagha
 
Beiträge: 6
Registriert: Mo Mai 05, 2014 9:49 pm

Re: Using YaCy with centralized storage

Beitragvon sixcooler » So Mai 11, 2014 12:33 pm

Hello mr_aliagha,

sorry - I don't understand what your're trying to do.
YaCy doesn't store pages that are fetched bevore they are parsed.
Why should there be a storage for that?

Cu, sixcooler.
sixcooler
 
Beiträge: 494
Registriert: Do Aug 14, 2008 5:22 pm

Re: Using YaCy with centralized storage

Beitragvon mr_aliagha » So Mai 11, 2014 1:47 pm

So here is what I thought about YaCy and integration with external Solr:
YaCy uses its in-memory data-structure(probably DHT?) to fetch web pages and it can uses external Solr with Hbase database to store indexed contents. Now my question is how indexed contents could be retrieve from Hbase (for search) that I am going to use for storing Solr indexed contents?
Regards.
mr_aliagha
 
Beiträge: 6
Registriert: Mo Mai 05, 2014 9:49 pm

Re: Using YaCy with centralized storage

Beitragvon sixcooler » So Mai 11, 2014 10:04 pm

Hello mr_aliagha,

I'm verry sorry, but I don't understand anything of your question.

@all: is anybody out there to help us here please?

Could you perhaps describe the setup of your peers and what you're missing for that?

Cu, sixcooler.
sixcooler
 
Beiträge: 494
Registriert: Do Aug 14, 2008 5:22 pm

Re: Using YaCy with centralized storage

Beitragvon mr_aliagha » Di Mai 13, 2014 10:28 pm

Hi,
I meant suppose we have 4 different task in YaCy: crawling, indexing, storing and retrieving. Assume we want to use YaCy for crawling, index and storing. But we want to use our own portal for retrieving. My question would be is there anyway API or client available for cached page results and indexes from YaCy? Can we use any kind of database connection for access to stored caches? If yes how?
Regards.
mr_aliagha
 
Beiträge: 6
Registriert: Mo Mai 05, 2014 9:49 pm

Re: Using YaCy with centralized storage

Beitragvon sixcooler » Di Mai 13, 2014 11:06 pm

Hello mr_aliagha,

there're many APIs - see: http://www.yacy-websuche.de/wiki/index.php/Dev:API
If you're using a central external Solr (for more than one instance of YaCy) you can also use the Solr-API for your search-portal.

cu, sixcooler.
sixcooler
 
Beiträge: 494
Registriert: Do Aug 14, 2008 5:22 pm


Zurück zu YaCy Coding & Architecture

Wer ist online?

Mitglieder in diesem Forum: 0 Mitglieder und 1 Gast

cron