Hi robinpaulson, interesting, what kind of software are you developing?
During the long development time of YaCy we created, used and discarded a lot of self-made data structures. Mainly used are now:
- Solr, for local search indexes and as metadata store of the RWI (see below)
- a self-made reverse word index (RWI) data structure for index fragments that are stored in the distributed hash table in the peer network
- a self-made schema-less data structure 'MapHeap' which you can use to throw in any map
; it will be retrieved using a primary key
- a self-made schema-less data structure BEncodedHeap
which does the same as MapHeap but stores the Maps using bencoding
, which is more space-efficient. (bencoding is the same encoding that the torrent files use
). This should be your preferred format over MapHeap; use MapHeap if you want to read the dumps with your own tools, that should be easy.
What do you want to store in your database?