This is in general a very good idea, but
smokingwheels hat geschrieben:I ideally need to schedule each new crawl 3.6 seconds apart as not to overload any of the systems.
(which is also not a bad idea) does not work. The reason is, that the scheduler for the API actions does not work this way. Here is how it works:
- there is a "cleanup"-process which runs every 10 minutes (you can change this in /PerformanceQueues_p.html, see "Delay between busy loops" column for the "Cleanup" row)
- as part of the cleanup-process (which does a lot, i.e. cleaning caches, running postprocessing etc.) the API table (see also: /Tables_p.html?table=api ) is checked for processes that are due which are then startet all at once
That means, even if you configure 3.6 seconds distance between such starts in the schedule, a set of then would be startet at once.
Outside of the automatism that you want to establish I suggest the following two options to solve the scheduling-problem that comes from the current architecture.
- either move the API-scheduler process out from the cleanup-process into it's own busy thread so you can change the frequency you want to
- or add a delay option in the scheduled process start so you can cause that the api calls are not made too fast after each other
While the first option is much more work I would suggest that this is the better option. Additionally, the second option could be established independently from the first one.