Parallel Jobs on a Cluster#
Cleaning one month of TRACE takes about two minutes. The full history is more than 270 months. Run one after another, that is nine hours; run at the same time on 270 cores, it is still two minutes. A cluster has the cores. This page is about how you ask for them.
There are two common ways, and you should be able to read both:
GNU Parallel runs many commands at once inside one job.
A SLURM job array submits many jobs at once, one per task.
We build up to the TRACE job in six small steps. The scripts are in the
slurm folder of the in-class examples.
Each one runs in under a minute and costs almost nothing, so submit them
yourself.
When Can Work Run in Parallel?#
Tasks can run at the same time when they are independent: no task needs another task’s output, and no two tasks write to the same file.
Stage 0 of the TRACE pipeline is independent by month. February’s raw trades are in one file, and cleaning them produces one file. March has its own raw file and its own output. Nothing in March’s cleaning reads February. Work like this is called embarrassingly parallel, because there is nothing clever to do: split it up and run the pieces.
Stage 1 is different. It follows each bond through time to compute returns and to track its rating, so it needs every month together. It runs as one task, after stage 0 has finished.
The Work: a Task That Takes Five Seconds#
To see the scheduling without waiting on real data, the examples use a task that sleeps for five seconds, squares a number, and says which machine it ran on:
# slow_square.sh
N="$1"
sleep 5
echo "$(date +%T) $(hostname) ${N} squared is $((N * N))"
We want to run it for the numbers 1 through 6.
Step 1: One After Another#
#!/bin/bash
#SBATCH --job-name=01-serial
#SBATCH --account=finm32800
#SBATCH --partition=caslake
#SBATCH --cpus-per-task=1
#SBATCH --mem=1G
#SBATCH --time=00:02:00
#SBATCH --output=%x_%j.out
for N in 1 2 3 4 5 6; do
bash slow_square.sh ${N}
done
Submit it with sbatch 01_serial.sbatch and read the output file when it is
done:
02:00:50 midway3-0156.rcc.local 1 squared is 1
02:00:55 midway3-0156.rcc.local 2 squared is 4
02:01:00 midway3-0156.rcc.local 3 squared is 9
02:01:05 midway3-0156.rcc.local 4 squared is 16
02:01:10 midway3-0156.rcc.local 5 squared is 25
02:01:15 midway3-0156.rcc.local 6 squared is 36
One machine, one core, six tasks five seconds apart: 30 seconds.
Step 2: GNU Parallel#
GNU Parallel is a command-line program that takes a command and a list of inputs, and runs the command once per input, several at a time. It is not part of SLURM. It runs on a laptop the same way it runs on a cluster.
The smallest example, which you can type at any prompt where it is installed
(on Midway3, after module load parallel):
parallel echo "Hello, {}" ::: Alice Bob Carol
Hello, Alice
Hello, Bob
Hello, Carol
:::starts the list of inputs.{}is where each input goes in the command.The three
echocommands run at the same time, so the lines can come out in any order.
The options you will meet most often:
Option |
What it does |
|---|---|
|
Run at most four commands at a time |
|
Print the commands that would run, and run nothing |
|
Read the inputs from a file, one per line |
|
Split each input line into |
|
Write one line per finished command: start, run time, exit code |
|
Skip the commands that the job log says already ran |
|
Skip those that succeeded, and rerun those that failed |
--dry-run is the one to learn first. Use it every time before the real run:
parallel --dry-run -j 2 python clean_month.py {} ::: 2020-02 2020-03 2020-04
python clean_month.py 2020-02
python clean_month.py 2020-03
python clean_month.py 2020-04
Without --dry-run, this runs those three commands, two at a time. The third
starts when either of the first two finishes.
Now the six squares, as a batch job. The job asks for six cores on one machine, and GNU Parallel uses them:
#!/bin/bash
#SBATCH --job-name=02-gnu-parallel
#SBATCH --account=finm32800
#SBATCH --partition=caslake
#SBATCH --cpus-per-task=6
#SBATCH --mem=1G
#SBATCH --time=00:02:00
#SBATCH --output=%x_%j.out
export LANG=en_US.UTF-8 LC_ALL=en_US.UTF-8
module load parallel
parallel -j ${SLURM_CPUS_PER_TASK} bash slow_square.sh {} ::: 1 2 3 4 5 6
02:00:50 midway3-0099.rcc.local 6 squared is 36
02:00:50 midway3-0099.rcc.local 4 squared is 16
02:00:50 midway3-0099.rcc.local 2 squared is 4
02:00:50 midway3-0099.rcc.local 3 squared is 9
02:00:50 midway3-0099.rcc.local 1 squared is 1
02:00:50 midway3-0099.rcc.local 5 squared is 25
One machine, the same second on every line, and no particular order: 5 seconds.
Two things in the script are worth a note. SLURM_CPUS_PER_TASK is set by
SLURM to the number you asked for, so the -j can never disagree with the
#SBATCH line. And the export line is there because a Mac sends its own
language setting over SSH, the compute nodes do not have it, and GNU Parallel
(a Perl program) prints a long warning for every task without it.
Step 3: a Job Array#
A job array asks SLURM to do the splitting. One line does it:
#!/bin/bash
#SBATCH --job-name=03-array
#SBATCH --account=finm32800
#SBATCH --partition=caslake
#SBATCH --array=1-6
#SBATCH --cpus-per-task=1
#SBATCH --mem=1G
#SBATCH --time=00:02:00
#SBATCH --output=%x_%A_%a.out
bash slow_square.sh ${SLURM_ARRAY_TASK_ID}
--array=1-6 tells SLURM to run this script six times, as six separate jobs.
Each copy is identical except for one environment variable,
SLURM_ARRAY_TASK_ID, which is 1 in the first copy, 2 in the second, and so
on. The script uses that number to decide what its own piece of the work is.
Every other #SBATCH line describes one task, not the whole array: each
of the six gets one core, 1 GB, and two minutes.
In the output file name, %A is the array’s job ID and %a is the task’s
number, so each task writes its own file:
== 03-array_63296473_1.out
02:00:50 midway3-0156.rcc.local 1 squared is 1
== 03-array_63296473_2.out
02:00:50 midway3-0214.rcc.local 2 squared is 4
== 03-array_63296473_3.out
02:00:50 midway3-0136.rcc.local 3 squared is 9
== 03-array_63296473_4.out
02:00:50 midway3-0097.rcc.local 4 squared is 16
== 03-array_63296473_5.out
02:00:57 midway3-0203.rcc.local 5 squared is 25
== 03-array_63296473_6.out
02:01:10 midway3-0203.rcc.local 6 squared is 36
Look at the machines and the times. The six tasks ran on five different machines, and tasks 5 and 6 started a few seconds after the others. Each task is its own job in the queue, and each started when the scheduler found it a free core.
sacct shows one row per task:
$ sacct -X -j 63296473 --format=JobID%16,JobName,State,Elapsed,NodeList
JobID JobName State Elapsed NodeList
---------------- ---------- ---------- ---------- ---------------
63296473_1 03-array COMPLETED 00:00:06 midway3-0156
63296473_2 03-array COMPLETED 00:00:06 midway3-0214
63296473_3 03-array COMPLETED 00:00:06 midway3-0136
63296473_4 03-array COMPLETED 00:00:06 midway3-0097
63296473_5 03-array COMPLETED 00:00:16 midway3-0203
63296473_6 03-array COMPLETED 00:00:29 midway3-0203
A few more forms of --array:
Line |
Meaning |
|---|---|
|
Tasks 1 through 6 |
|
Ten tasks; the task ID is the year |
|
Only tasks 3 and 5 |
|
A hundred tasks, at most ten running at a time |
The %10 matters on a shared allocation. Without it, a large array takes
every core the scheduler will give it.
Step 4: GNU Parallel Across Machines#
Step 2 kept all six tasks on one machine. A job can also ask for six slots and let SLURM put them wherever there is room:
#SBATCH --ntasks=6
#SBATCH --cpus-per-task=1
parallel -j ${SLURM_NTASKS} \
srun --exclusive -N1 -n1 bash slow_square.sh {} ::: 1 2 3 4 5 6
GNU Parallel still decides which task runs next. srun is SLURM’s command
for starting a program on the resources a job already holds; here each srun
takes one free slot of the job’s allocation and runs one task there. In our
run, five tasks landed on midway3-0012 and one on midway3-0029.
This is still one job: one line in squeue, one job ID, one output file.
Steps 5 and 6: When One Task Fails#
In real work one month has a bad row and its task crashes. The examples use
flaky_square.sh, which fails for the number 3 until a file named fixed
exists.
With a job array, sacct tells you which task failed:
JobID State ExitCode
---------------- ---------- --------
63296475_1 COMPLETED 0:0
63296475_2 COMPLETED 0:0
63296475_3 FAILED 1:0
63296475_4 COMPLETED 0:0
63296475_5 COMPLETED 0:0
63296475_6 COMPLETED 0:0
SLURM does not rerun task 3, and it keeps no list of what is left to do. You fix the problem and submit the same script again, for that task only:
touch fixed # "fix the bug"
sbatch --array=3 05_array_with_a_failure.sbatch
An option given on the command line overrides the #SBATCH line of the same
name. So this submission is an array with one task, number 3, and the script
does what it always does for task 3.
With GNU Parallel, the scheduler knows only that the one job failed. The
detail is in the job log, which has one line per command. Four of its nine
columns are shown here; Exitval is the command’s exit code, and anything
but 0 is a failure:
Seq JobRuntime Exitval Command
1 5.000 0 bash flaky_square.sh 1
2 5.000 0 bash flaky_square.sh 2
4 5.000 0 bash flaky_square.sh 4
5 5.000 0 bash flaky_square.sh 5
6 5.000 0 bash flaky_square.sh 6
3 5.000 1 bash flaky_square.sh 3
Because the script passes --joblog joblog.txt --resume-failed, you fix the
problem and submit the same script, unchanged. GNU Parallel reads the log,
skips the five that succeeded, and reruns the one that failed.
Idempotent Tasks#
There is a third way, and it works with either tool. Write each task so that it checks for its own output first and exits at once when the output is already there. A task like that is called idempotent: running it twice leaves the same result as running it once.
Then you never work out which tasks failed. You submit everything again. The finished tasks each take a second, and only the missing ones do any work. The TRACE tasks below are written this way.
The Two Side by Side#
Job array |
GNU Parallel in one job |
|
|---|---|---|
What the scheduler sees |
N jobs, each queued and billed on its own |
One job; you divide its allocation yourself |
When the work starts |
Each task as soon as a slot is free for it |
When the whole allocation is free |
Where a failure shows up |
|
The job log |
Rerunning a failure |
|
Submit again with |
Output |
One file per task |
One file for the job |
Works without a scheduler |
No |
Yes, on any Linux or Mac |
Good for |
Tens to hundreds of tasks, minutes to hours each |
Hundreds or thousands of short tasks |
The last row is the one that decides most cases. Every job costs the scheduler some work to queue, start, and record. For a thousand tasks of three seconds each, that overhead is larger than the work, and cluster administrators ask you not to submit them as an array. One job that runs GNU Parallel is the right tool there. For six tasks of two minutes, or two hundred tasks of an hour, the array is simpler and tells you more.
Neither is a pipeline. If task 4 needs the output of task 3, the tasks are
not independent, and the tool is doit, or a second job submitted with
sbatch --dependency=afterok:<job id>, which starts only after the first has
finished without an error.
The TRACE Job, Both Ways#
The real job cleans February, March, and April of 2020 for two TRACE feeds,
Enhanced and Rule 144A: six tasks. The scripts are in the
slurm folder of the TRACE case study.
Both versions read the same list, slurm/tasks.txt, one task per line:
enhanced 2020 2
enhanced 2020 3
enhanced 2020 4
144a 2020 2
144a 2020 3
144a 2020 4
and both call the same script, slurm/clean-one-month.sh <feed> <year> <month>,
which cleans one month and skips it if the output already exists.
As a job array (slurm/clean-trace-array.sbatch), task N takes line N of
the list:
#SBATCH --array=1-6
#SBATCH --cpus-per-task=2
#SBATCH --mem=8G
#SBATCH --time=00:30:00
#SBATCH --output=%x_%A_%a.out
TASK=$(sed -n "${SLURM_ARRAY_TASK_ID}p" slurm/tasks.txt)
bash slurm/clean-one-month.sh ${TASK}
sed -n "3p" file prints line 3 of a file. This is the usual way to hand an
array anything more complicated than a number: put the inputs in a file, one
per line, and let the task ID pick the line.
$ sacct -X -j 63296433 --format=User,JobID%16,JobName,State,Elapsed,NodeList
User JobID JobName State Elapsed NodeList
--------- ---------------- ---------- ---------- ---------- ---------------
jbejarano 63296433_1 trace-arr+ COMPLETED 00:01:41 midway3-0090
jbejarano 63296433_2 trace-arr+ COMPLETED 00:01:57 midway3-0105
jbejarano 63296433_3 trace-arr+ COMPLETED 00:01:55 midway3-0012
jbejarano 63296433_4 trace-arr+ COMPLETED 00:00:35 midway3-0099
jbejarano 63296433_5 trace-arr+ COMPLETED 00:00:36 midway3-0185
jbejarano 63296433_6 trace-arr+ COMPLETED 00:00:35 midway3-0185
Six tasks on five machines, done in under two minutes. One after another, the same work takes more than seven.
With GNU Parallel (slurm/clean-trace-parallel.sbatch), one job asks for
six slots and hands the six lines out:
#SBATCH --ntasks=6
#SBATCH --cpus-per-task=2
#SBATCH --mem-per-cpu=4G
module load parallel
parallel -j "${SLURM_NTASKS}" --colsep ' ' \
--joblog trace-parallel_joblog.txt --resume \
srun --exclusive -N1 -n1 bash slurm/clean-one-month.sh {1} {2} {3} \
:::: slurm/tasks.txt
--colsep ' ' splits each line of the list at the spaces, so {1} is the
feed, {2} the year, and {3} the month.
To clean ten years, neither script changes. The list gets longer, and for
the array --array=1-6 becomes --array=1-240%20.
HW 5 asks you to submit the array version yourself.