Is there any progress bar (or any other - similar kind of indicator, for crawling process? I started crawling a website and after ~10h yacy is still crawling. Can someone predict, when the process will be completed?
I only joined these forums recently myself, so I am not certain how much help I can be to you. It would seem to me however that in order for the crawler to produce a progress bar, it would have to know how long the process should take. Since that determination cannot be made until the site has been crawled, I would think that a crawler progress bar would be impossible. Just a thought though.
Hello, I also agree with jerk, YaCy can not predict itself the end of a crawl task because it depends on the links found in the crawled pages... unknown before crawling. But maybe an approximation could be calculated when re-crawling a known website...
By the way, I believe that currently the best estimation will come from your knowledge of the website content, the "Crawling Depth" and eventual URLs filters you used, and the monitored "Crawler PPM" value on the first hours.
If your crawl appears to be too long for you, maybe some of your crawl parameters should be re-adjusted.