Orbiter hat geschrieben:since twitter decided to switch off RSS feeds it is not easy any more to integrate tweets in YaCy search results. We would need a twitter scraper which may be possible to set specific crawl filter rules. Someone must invest some work to find out what to do exactly to crawl Twitter accounts in a nice way.
(to everyone): please invest some time to find a solution.
i looked upon this, just to figure how to scrape twitter. It is possible to scrape twitter via their api, but they have limitations regarding twitter api (terms of service + query restrictions).
There are few do-able solutions:
1)Scrape Twitter streaming JSON API and receive all tweets*, parse it with suitable already existing applications which can turn it to RSS, which yacy can read directly.
2)Same as above but use twitter json/api compatible java library and integrate it to yacy.*
*You receive only ~1% of the tweets per token with streaming api, also theres restrictions what you can do with the data.Its quite easy to do the choice one, but i don't know how it will look in search results.
I dont have enough knowledge in programming to do option 2 at this time but flip side, this would be excellent programming experience + learning opportunity.
Of course it would be nice in some cases but also extremely creepy to make social search into yacy which would build social profile for every people based on crawling, like showing profile pictures, all social media accounts and other information like friends etc.