Removing index entries when a page unavailable temporarily

Discussion in English language.
Forumsregeln
You can start and continue with posts in english language in all other forums as well, but if you are looking for a forum to start a discussion in english, this is the right choice.

Removing index entries when a page unavailable temporarily

Beitragvon ktplulo » Mi Aug 20, 2014 1:25 pm

Problem:
"If verification fails, delete index reference" is enabled, or anything else from viewtopic.php?f=23&t=5314&p=30466#p30466 is being done. Data about a page is to be deleted. But the page is actually not available not because it was updated, but because the server is down, or broken, or being blocked (censored) by mistake or otherwise temporarily. It is likely to go up soon, or the block (censorship) is going to be lifted.

Proposal:
A list of strings to detect such cases, and ways of detection (for example, delete only after the URL has been accessed at least 3 times with an interval of at least 3 days, but at most 30 days in sum).

Problem with the proposal:
1. It can be used to increase DoS spam.
Partial solution:
1. Only use for cached pages?

Another potential problem:
2. It can be seen from outside that somebody is searching repeatedly for something intentionally censored.
ktplulo
 
Beiträge: 18
Registriert: Do Mär 01, 2012 11:27 am

Re: Removing index entries when a page unavailable temporari

Beitragvon David » Fr Aug 22, 2014 6:30 pm

Maybe the expression "failed verification" is a bit misleading. As far as I know, if a website is offline or overloaded, it doesn't count as failed verification. A verification fails, if a page has changed or was updated since it was crawled, and the search term simply isn't there anymore, but I might be wrong.

ktplulo hat geschrieben:"If verification fails, delete index reference" is enabled, or anything else from viewtopic.php?f=23&t=5314&p=30466#p30466 is being done. Data about a page is to be deleted.

I think if a failed verification occurs, only the URL(s) get removed from the index. The words stay.
David
 
Beiträge: 170
Registriert: Di Mär 05, 2013 5:35 pm


Zurück zu English

Wer ist online?

Mitglieder in diesem Forum: Bing [Bot] und 2 Gäste