Yacy database schema?

Forum for developers

Yacy database schema?

Beitragvon robinpaulson » Fr Jul 11, 2014 9:36 am

Hi,
I'm building a piece of software (GPL of course) which builds on certain elements of Yacy. In particular I am interested in the database, which I gather is a NOSQL variant. Is the database schema (or whatever it is called in NoSQL-land) published anywhere I can see it? I've searched the forums and wiki, but nothing has turned up, I wonder if "schema" is the right word to use, or if that concept even translates from relational to NoSQL databases?

Cheers,

Robin
robinpaulson
 
Beiträge: 1
Registriert: Fr Jul 11, 2014 9:30 am

Re: Yacy database schema?

Beitragvon Orbiter » Fr Jul 11, 2014 1:07 pm

Hi robinpaulson, interesting, what kind of software are you developing?

During the long development time of YaCy we created, used and discarded a lot of self-made data structures. Mainly used are now:
- Solr, for local search indexes and as metadata store of the RWI (see below)
- a self-made reverse word index (RWI) data structure for index fragments that are stored in the distributed hash table in the peer network.
- a self-made schema-less data structure 'MapHeap' which you can use to throw in any map; it will be retrieved using a primary key
- a self-made schema-less data structure BEncodedHeap which does the same as MapHeap but stores the Maps using bencoding, which is more space-efficient. (bencoding is the same encoding that the torrent files use :) ). This should be your preferred format over MapHeap; use MapHeap if you want to read the dumps with your own tools, that should be easy.

What do you want to store in your database?
Orbiter
 
Beiträge: 5798
Registriert: Di Jun 26, 2007 10:58 pm
Wohnort: Frankfurt am Main


Zurück zu YaCy Coding & Architecture

Wer ist online?

Mitglieder in diesem Forum: 0 Mitglieder und 3 Gäste