How to search all of craigslist?

Discussion in English language.
Forumsregeln
You can start and continue with posts in english language in all other forums as well, but if you are looking for a forum to start a discussion in english, this is the right choice.

How to search all of craigslist?

Beitragvon kato » Sa Jul 23, 2016 3:51 pm

I want to search search multiple craigslist.org sites at once and almost real time. The problem is, they won't allow that. They let you search one sub-page per query. Many sites use google to offer customized craigslist searches. However, google doesn't offer up to date results. So those pages have poorly results. There are many scripts out there but most of them are out of date. Plus the craigslist.org administrators ban everybody who intent to index their site.
I wish to make a personal search engine similar to the:
https://web.archive.org/web/20100317031 ... sPal-works
Unfortunately the service was forced to shut down:
https://web.archive.org/web/20130416171 ... gspal.com/

I know, I could make an RSS feed reader, but that's very inflexible. So the question is, is YaCy suitable for the job or shall I write a script instead? Than is possible to avoid getting banned?
kato
 
Beiträge: 1
Registriert: Sa Jul 23, 2016 2:32 pm

Re: How to search all of craigslist?

Beitragvon luc » Di Jul 26, 2016 8:30 pm

Hi, if craigslist ban anyone intending to crawl their sites, I am not sure YaCy would currently help you...
However I suppose YaCy network could theorically do it, using "global crawl" feature (see "Do Remote Indexing" on the bottom of the CrawlStartExpert.html page on a running YaCy peer), but I fear currently there are not so much YaCy peers accepting remote crawls, and also maybe not enough peers running at the same time...

YaCy heuristics feature (/ConfigHeuristics_p.html) could also help, but it is designed to only configure some OpenSearch URLs which will fill P2P search results with some more items, and won't be the only results source.

But maybe a Meta Search engine such as Searx could help you more...
luc
 
Beiträge: 293
Registriert: Mi Aug 26, 2015 1:04 am


Zurück zu English

Wer ist online?

Mitglieder in diesem Forum: YaCy [Bot] und 2 Gäste

cron