I've just noticed that after starting crawling there are large files being downloaded (these can be observed under "Crawler Monitor" -> "Processing Monitor"/"Loader") which seem to exceed file size limits. I've checked limits under "System Administration" -> "Advanced Settings" -> "Crawler Settings" and these are:
- HTTP Crawler Settings: 10485760
- FTP Crawler Settings: 10485760
- SMB Crawler Settings: 100000000
- Local File Crawler Settings: 100000000
For example, I've observed following file being downloaded: http://www.swi-prolog.org/download/stab ... d.mpkg.zip
HTTP response for this file returns Content-Length:"12026535" which is bigger than Yacy HTTP file size limit (10485760).
I have also seen 400Mb+ files being downloaded.
Could you please clarify whether this is a bug (I will create ticket in bugtracker then) or I am just missing something. Could you please also explain what is a Loader in Yacy terminology (is that just an additional queue for large binary files that runs in parallel with crawler?)