Job pipeline using slurm dependencies

Some times we need to launch a list of jobs that execute in sequence, one after another. In those cases we will use the --depency sbatch option, check the manual page for more details, we will only present a simple example.

Simple example

Suppose we need to submit the script and then that should run after the first one:

[user@cirrus01 ~]$ sbatch
Submitted batch job 1843928

[user@cirrus01 ~]$ sbatch --dependency=after:1843928
Submitted batch job 1843921

[user@cirrus01 ~]$ squeue
1843928       hpc user  R 0:11     1 hpc046
1843921       hpc user PD 0:00     1 hpc047

In this case the second job will run even if the first job fails for some reason. The pending job will execute when the first finish his execution.

Tipical example

On a real case we may need the ensure that a good termination of the first job, for example, the first job may produce some output file needed as input for the second job:

[user@cirrus01 ~]$ sbatch
Submitted batch job 1843922

[user@cirrus01 ~]$ sbatch --dependency=afterok:1843922
Submitted batch job 1843923

The afterok parameter states that the second job would start only if the previous job terminate with no errors.

Complex cases

Check the sbatch manual page for more details:

 [user@cirrus01 ~]$ man sbatch

search for the -d, --dependency=<dependency_list> options explanation.