Why short jobs are better than long jobs
There are lots of reasons why short jobs have advantage over long jobs and, in particular, why really long jobs should be broken up into smaller jobs if possible:
- better queue utilisation: long jobs will generally queue longer and be suspended longer.
- protect yourself against system and node failures. It can be very frustrating waiting patiently for a week-long job to see it fail in the last day or hour. This is also useful for smaller time windows leading up to system or node downtimes.
- small jobs with defined execution time can be more easily placed in available execution slots.