YaCy heuristics : extend with RSS-Bridge

Discussion in English language.
Forumsregeln
You can start and continue with posts in english language in all other forums as well, but if you are looking for a forum to start a discussion in english, this is the right choice.

YaCy heuristics : extend with RSS-Bridge

Beitragvon luc » Fr Mai 20, 2016 11:15 am

Hi, maybe you know YaCy Heuristics feature (/ConfigHeuristics_p.html) : it allows you to feed YaCy search results from external websites supporting OpenSearch.
This feature looks interesting but if you tried to play with it you may have soon realized it has quite a limited use : YaCy only support OpenSearch results in RSS or Atom feed formats but most websites offering OpenSearch only return results in HTML.

Example from Twitter opensearch.xml :
Code: Alles auswählen
<Url type="text/html" method="get" template="https://twitter.com/search?q={searchTerms}"/>


Unfortunately, OpenSearch results in HTML are not standardized, so if you want a program to be able to use it, you have to write a custom parser for each website. So gentle websites should also provide OpenSearch results in RSS or Atom if they want visitors to easily plug any program they want on their results (WordPress.com is a good example. See their OpenSearch description)... But that not the case for many wesites I checked.

Here come to rescue a project like RSS-Bridge. Volunteers contributors are providing mappings from web APIs to RSS or Atom feeds. All you need is to install a PHP enabled web server and launch your RSS-Bridge instance.

So with YaCy, resumed architecture will be : user search -> YaCy server (heuristics enabled) -> RSS-Bridge -> External website

Here are the required setup steps :
1. Install a web server supporting PHP : for example Apache with the PHP module enabled
2. Get latest sources from https://github.com/sebsauvage/rss-bridge
3. Extract it to a suitable directory : for example /var/www/html for a Debian Apache
4. Eventually set needed permissions : for example chown www-data ... for a Debian Apache
5. Check rss-bridge is working at http://your_host/rss-bridge/, and verify search is working for the bridge you are interested in
6. Eventually modify rss-bridge/whitelist.txt file to enable a specific bridge
7. Go to your YaCy /ConfigHeuristics_p.html page : tick "opensearch load external search result list from active systems below" checkbox
8. Add one or more bridge URLs : any title and comment you wish, and URL copied from a rss-bridge search result. Example (searching yacy on Twitter Bridge) :
Code: Alles auswählen
http://your_host/rss-bridge/?action=display&bridge=TwitterBridge&q=yacy&format=AtomFormat
.
9. Important : replace your search term with {searchTerms} to make a valid OpenSearch URL or it won't work. Example :
Code: Alles auswählen
http://your_host/rss-bridge/?action=display&bridge=TwitterBridge&q={searchTerms}&format=AtomFormat

10. That's it! Now heuristics results from the bridge may appear in your 20 first YaCy search results (with H favicon left from some search result entries).
luc
 
Beiträge: 231
Registriert: Mi Aug 26, 2015 1:04 am

Zurück zu English

Wer ist online?

Mitglieder in diesem Forum: 0 Mitglieder und 1 Gast

cron