Filter specific urls from scarping

Discussion in English language.
Forumsregeln
You can start and continue with posts in english language in all other forums as well, but if you are looking for a forum to start a discussion in english, this is the right choice.

Filter specific urls from scarping

Beitragvon wrecked » Sa Jul 12, 2014 7:32 pm

Hi, I noticed than when scraping en.wikipedia.org and stackoverflow.com many requests were for signup and login pages, with a re-direct to the original site. Is it possible to black-list or white-list specific URLs?
wrecked
 
Beiträge: 4
Registriert: Di Jul 08, 2014 8:38 pm

Re: Filter specific urls from scarping

Beitragvon David » So Jul 13, 2014 11:36 am

Hello,

I think the most login and signup pages on the internet contain an "?" in the URL, and the yacy crawler ignores such pages by default (somewhere in the advanced crawler settings, you can tell yacy to index them anyways).

wrecked hat geschrieben:Is it possible to black-list or white-list specific URLs?

Blacklists can be created in the menu "Filter & Blacklists".
http://localhost:8090/Blacklist_p.html
David
 
Beiträge: 170
Registriert: Di Mär 05, 2013 5:35 pm

Re: Filter specific urls from scarping

Beitragvon wrecked » So Jul 13, 2014 9:56 pm

For example I started crawling on "http://askubuntu.com/questions", and on path I selected the "load only files in a sub-path of given url" option. I also added "askubuntu.com/users/*" to blacklists, but crawling monitor still displayed user URLs like "http://askubuntu.com/users/163406/user1945827" in the list of scraped URLs.

It also reported crawling URLs like "http://askubuntu.com/questions?page=2&sort=newest", so I guess that "?" filter is not being applied.

At askubuntu.com you can choose to list 15, 30 or 50 questions / page. I wonder if Yacy is smart enough to avoid scraping this list tree times instead of once. On the other hand questions themselves have been scraped already, so it isn't too bad to go over this list multiple times. Correct me if I'm wrong...
wrecked
 
Beiträge: 4
Registriert: Di Jul 08, 2014 8:38 pm


Zurück zu English

Wer ist online?

Mitglieder in diesem Forum: 0 Mitglieder und 1 Gast

cron