Loader and file size limits

Discussion in English language.
Forumsregeln
You can start and continue with posts in english language in all other forums as well, but if you are looking for a forum to start a discussion in english, this is the right choice.

Loader and file size limits

Beitragvon solaris » Sa Dez 27, 2014 8:49 am

Hi guys

I've just noticed that after starting crawling there are large files being downloaded (these can be observed under "Crawler Monitor" -> "Processing Monitor"/"Loader") which seem to exceed file size limits. I've checked limits under "System Administration" -> "Advanced Settings" -> "Crawler Settings" and these are:
  • HTTP Crawler Settings: 10485760
  • FTP Crawler Settings: 10485760
  • SMB Crawler Settings: 100000000
  • Local File Crawler Settings: 100000000

For example, I've observed following file being downloaded: http://www.swi-prolog.org/download/stab ... d.mpkg.zip
HTTP response for this file returns Content-Length:"12026535" which is bigger than Yacy HTTP file size limit (10485760).
I have also seen 400Mb+ files being downloaded.

Could you please clarify whether this is a bug (I will create ticket in bugtracker then) or I am just missing something. Could you please also explain what is a Loader in Yacy terminology (is that just an additional queue for large binary files that runs in parallel with crawler?)
solaris
 
Beiträge: 2
Registriert: Sa Dez 27, 2014 8:32 am

Re: Loader and file size limits

Beitragvon drixter » Mo Jan 05, 2015 10:14 pm

Hi,

I observe the same, have you submit the bug?

Regards,
drixter
 
Beiträge: 2
Registriert: Do Jan 01, 2015 12:02 pm

Re: Loader and file size limits

Beitragvon solaris » Di Jan 13, 2015 6:12 pm

drixter hat geschrieben:Hi,

I observe the same, have you submit the bug?

Regards,


Not yet
solaris
 
Beiträge: 2
Registriert: Sa Dez 27, 2014 8:32 am

Re: Loader and file size limits

Beitragvon drixter » Di Jan 20, 2015 7:21 pm

Hi,

I just made a bug using my and your data.

http://mantis.tokeek.de/view.php?id=526

Regards,
drixter
 
Beiträge: 2
Registriert: Do Jan 01, 2015 12:02 pm


Zurück zu English

Wer ist online?

Mitglieder in diesem Forum: 0 Mitglieder und 2 Gäste