Lazyqueue: opportunistic, preemptible work

What it is#

lazyqueue runs work at the lowest possible priority, on the same 20 nodes as cpuqueue — it isn't separate hardware, it's a way to use capacity that would otherwise sit idle.

When it's appropriate#

Work that can tolerate being stopped and started again from the beginning: opportunistic batches, exploratory runs, and anything where losing progress and restarting costs you time but not correctness. It is not appropriate for anything with a deadline, and it is not a good fit for work that can't safely repeat itself — see Checkpointing and restart safety below.

What preemption actually does — verified configuration#

lazyqueue sits at a lower partition priority tier than cpuqueue (tier 0 vs. tier 1), and the cluster's preemption mechanism (preempt/partition_prio) uses exactly that: when a cpuqueue job needs resources currently held by a lazyqueue job on the same node, the lazyqueue job is preempted to free them.

The lazy QoS is configured with a 5-minute grace time and PreemptMode=requeue. In practice:

1. A higher-priority job needs the node your lazyqueue job is using.
2. Your job gets a 5-minute warning before it's actually preempted.
3. When that grace period ends, your job is stopped and requeued — put back in the pending queue rather than cancelled outright.
4. It waits its turn again, and may restart on the same node or a different one once resources are free.

Do you need to add --requeue? No — verify before you assume otherwise#

This is worth being precise about, because it's easy to conflate two different settings.

• PreemptMode=requeue is a QoS/partition-level setting. It controls what Slurm does to a preempted job — requeue it, rather than cancel or suspend it. This is already configured on the lazy QoS; you don't set it yourself.
• **A job's own requeue *eligibility* is a separate, job-level property, controlled by sbatch's --requeue / --no-requeue flags. On this cluster, the default (JobRequeue=1) already makes every job requeue-eligible without you specifying anything**.

So for ordinary use, you do not need to add --requeue — the behaviour you want is already the default. What you should actively avoid is --no-requeue: if you pass it, your job opts out of requeue eligibility, and there's nothing left for the QoS's PreemptMode=requeue to act on when you're preempted — defeating the point of using lazyqueue at all.

Submitting a lazyqueue job#

#!/bin/bash
#SBATCH --job-name=lazy-example
#SBATCH --partition=lazyqueue
#SBATCH --qos=lazy
#SBATCH --nodes=1
#SBATCH --cpus-per-task=2
#SBATCH --mem-per-cpu=2G
#SBATCH --time=04:00:00

echo "Started on $(hostname) at $(date)"
# your restartable work here

--qos=lazy is mandatory, exactly like every other job on this cluster. Nothing else needs to be added for the standard preempt-and-requeue behaviour described above.

Checkpointing and restart safety#

Being requeued at the Slurm level is not the same as your application resuming from where it left off. Unless your program explicitly checkpoints its own progress, a requeued job restarts your batch script — and whatever it runs — from the very beginning.

This page doesn't teach checkpointing itself — that depends entirely on your software. The practical takeaway is: pick work for lazyqueue where restarting from scratch is merely wasteful, not wrong.

Utilization is not a reason to choose lazyqueue#

Whether the cluster currently looks busy or idle isn't the deciding factor — lazyqueue and cpuqueue share the same hardware regardless. Choose lazyqueue based on whether your work tolerates preemption and restart, not based on what current utilization happens to show at the moment you submit.

Related#

• Partitions, QoS, and fairshare
• Submitting jobs
• Why did my job fail? — including what a requeue looks like in sacct