Check job efficiency

Your job finished. Did it need everything it asked for?

This page answers that for a completed job: how much of the allocated CPU capacity the program actually used, how much memory it really needed, and how its runtime compared with the time limit you set. Completed jobs are the right place to look, because Slurm only has the full accounting record once the job is over.

The workflow#

1. Find the job ID.
2. Get a quick summary with seff.
3. Look at the detail with sacct.
4. Compare CPU used against CPU allocated.
5. Compare peak memory against memory requested.
6. Compare elapsed time against the time limit.
7. Adjust the next job — if the pattern holds across runs.

Set the job ID once and reuse it:

JOBID=12345678

Step 1 — Find the job ID#

If you no longer have it, list your recent jobs:

sacct -X -u $USER --starttime today --format=JobID,JobName%20,State,Elapsed,AllocCPUS

Use --starttime 2026-09-01 for an older window. -X shows one row per job instead of one row per job step, which is what you want when you are just hunting for the ID.

Step 2 — The quick summary#

seff $JOBID

seff is installed on Mjolnir for all users and reads the same accounting records as sacct. It prints CPU and memory use as percentages of what was allocated, so it is the fastest way to see whether a request was roughly the right size.

Example output (illustrative — not from a real Mjolnir job):

Job ID: 12345678
Cluster: cluster
User/Group: abc123/users
State: COMPLETED (exit code 0)
Nodes: 1
Cores per node: 12
CPU Utilized: 1-01:06:21
CPU Efficiency: 2.40% of 43-13:09:48 core-walltime
Job Wall-clock time: 3-15:05:49
Memory Utilized: 22.17 GB
Memory Efficiency: 22.17% of 100.00 GB

Two things to know before you act on those percentages:

• Use it on finished jobs. On a job that is still running, seff reports the accounting so far, which usually looks far worse than the final figure. For a running job, see While the job is still running below.
• A low percentage is a question, not a verdict. It tells you the job did not use everything it reserved. It does not tell you why. The sections below cover the reasons that matter.

Step 3 — The detail#

seff gives you the headline. sacct gives you the numbers behind it, which is what you need when the headline is surprising.

CPU and time:

sacct -j $JOBID --format=JobID%18,State%12,AllocCPUS,TotalCPU,CPUTime,Elapsed,Timelimit%12

Memory:

sacct -j $JOBID --units=G --format=JobID%18,ReqTRES%36,MaxRSS

--units=G prints memory in GB instead of Slurm's default kilobytes, which saves you converting a number like 23247278K by hand.

Why there are two rows#

A normal batch job produces two rows: the job itself, and a .batch step underneath it.

             JobID        State  AllocCPUS   TotalCPU    CPUTime    Elapsed
------------------ ------------ ---------- ---------- ---------- ----------
          12345678    COMPLETED         12 1-01:06:21 43-13:09:48 3-15:05:49
    12345678.batch    COMPLETED         12 1-01:06:21 43-13:09:48 3-15:05:49

Memory is recorded per step, so MaxRSS appears on the .batch row and is blank on the job row. An empty MaxRSS next to the job ID is not evidence that the job used no memory — it means you are reading the wrong row. Jobs that use srun get extra numbered rows such as 12345678.0, one per step; read the highest MaxRSS among them.

CPU: how much of the allocation was used#

The comparison is TotalCPU against CPUTime:

• CPUTime is the CPU time the job had — AllocCPUS × Elapsed.
• TotalCPU is the CPU time the job used.

Both are printed in the same D-HH:MM:SS form, so you can compare them by eye without any arithmetic. seff prints the same ratio as a percentage.

When they are close, the program kept most of the allocated CPU capacity busy.

When TotalCPU is much smaller than CPUTime, the job reserved more CPU capacity than it consumed. That is worth investigating, but it does not mean the program is written badly or that the request was necessarily wrong. Common reasons:

• The program is I/O-bound — it spends much of its time waiting on storage rather than computing.
• The work has serial phases: setup, reading input, writing results, or a final merge step that only one CPU can do.
• Threads spend time synchronising or waiting for each other.
• The application does not scale to the number of CPUs you gave it, however many you request.
• The application was not told to use them. Many tools default to one thread unless you pass something like -t, -p, --threads, or --cores, and that value has to match your --cpus-per-task.
• CPU demand varies during the run, so a single average hides a busy phase and an idle one.

The most common fixable cause is the second-to-last one. Before changing the CPU count, check that the program was actually asked to use the CPUs it was given.

A TotalCPU of essentially zero on a job that ran for a long time usually means something else again — the job was stuck or waiting, not merely over-provisioned. Start with Why did my job fail?.

Memory: peak use against the request#

Read the requested total from ReqTRES (the mem= value) and the peak from MaxRSS on the .batch row.

ReqTRES is the recommended field because it reports memory as a single total for the job, already worked out from whichever flag you used. sacct also has a ReqMem field, and you will see it on other pages and in older guidance, but on Mjolnir it carries a suffix you have to interpret yourself:

Reading 5Gc as "5 GB" on a job that was allocated ten CPUs understates the request tenfold. ReqTRES avoids the problem entirely, which is why the command above uses it.

Two further limits worth knowing:

• Memory is sampled periodically rather than watched continuously, so a spike lasting only a few seconds may not appear in MaxRSS.
• MaxRSS is the peak for a single step, not a sum across nodes. For a multi-node job, read it as the largest per-node peak.

sacct also offers MaxVMSize. It reports *virtual* memory, which for many programs is far larger than anything they actually touch — a job peaking at 22 GB can easily show over 1 TB of MaxVMSize. It is not a useful basis for a memory request, and it is deliberately left out of the commands here.

Interpreting the gap. If a job requests 100 GB and peaks at 12 GB, the request looks much larger than the demand, and a smaller request is worth considering. If MaxRSS sits close to the request, leave the request alone — memory is a hard limit, and a job that exceeds it is killed (Why did my job fail?).

There is no Mjolnir policy setting a required safety margin, so no fixed percentage is prescribed here. Base the new figure on peaks you have actually observed across representative inputs, and leave enough room that a slightly larger input does not kill the job.

If you set no memory flag at all, cpuqueue gives each job 2 GB per CPU by default.

Walltime: runtime against the limit#

Compare Elapsed with Timelimit.

Unused walltime is not consumed CPU time, and it is not charged. It still matters, because the limit is what the scheduler plans around: Slurm backfills short jobs into gaps ahead of larger queued ones when they fit, and an honest limit gives your job more gaps it can fit into (Partitions, QoS, and fairshare).

A job that finishes in 12 minutes after asking for 7 days is good evidence that the next request can be tighter. But one run does not establish your maximum runtime. Before tightening:

• Check several runs, not just the fastest one.
• Consider how much your inputs vary in size — a larger input may take considerably longer.
• Remember that a job which hits its limit is killed and loses unwritten work.

Leave headroom above the longest run you have actually seen, rather than trimming to the last successful one.

Worked examples#

The figures below are illustrative. They show the shape of the output and how to read it.

A — More CPUs than the program used#

Reading it: the job had about 32 CPU-hours available and consumed about 5. Roughly one sixth of the allocation was used.

What to do: check whether the program was told to use 32 threads, and whether its threading option matches --cpus-per-task. If it was, test whether it actually scales — run the same input at 8 and 16 CPUs and compare Elapsed. It does not follow that 5 CPUs is the right number: the work may be doing genuinely parallel bursts between serial phases, in which case cutting to 5 would slow the parallel part without helping the serial one.

B — More memory than the job needed#

Reading it: the request is roughly eight times the observed peak.

What to do: confirm the peak is representative by checking a few runs with different inputs. If they all sit near 12 GB, a substantially smaller request is reasonable — with enough room above the observed peak to absorb a larger input.

C — More time than the job used#

Reading it: the job asked for 24 hours and ran for 35 minutes.

What to do: if your other runs of this workflow also finish well within an hour, a much shorter --time is realistic and makes the job easier to backfill. If runtime depends heavily on input size, size the limit for your larger inputs rather than this one.

D — Reasonably matched#

Reading it: about two-thirds of the CPU time was used, memory peaked at roughly 70% of the request, and the job finished comfortably inside its limit with margin to spare.

What to do: nothing. This is what a well-sized request looks like. CPU efficiency below 100% is normal — startup, input reading, and output writing are rarely parallel — and the memory and time margins are doing their job. Chasing a higher percentage here would risk killed jobs for no real gain.

While the job is still running#

sstat reads live counters for a job that has not finished yet:

sstat -a -j $JOBID --format=JobID,MaxRSS,AveRSS,AveCPU

The -a matters: sstat -j 12345678 on its own returns an empty table, because sstat addresses steps rather than jobs. -a selects all steps; sstat -j 12345678.batch names the batch step directly.

This is useful for spotting a job climbing towards its memory limit before it is killed. For comparing a request against what was used, wait for the job to finish and use sacct — the completed record is more complete and easier to compare between runs.

Why right-sizing is worth the effort#

• Fairshare is charged on what you request, not what you consume. A job holding 48 CPUs and using 4 is charged for 48, which lowers the priority of your later jobs (Partitions, QoS, and fairshare).
• Smaller requests are easier for the scheduler to place, and a realistic time limit gives a job more chances to be backfilled into a gap. Neither is a promise about queue time — that depends on fairshare and on what else is queued — but both improve the odds.
• Memory is a consumable resource on Mjolnir: memory you reserve and do not use is unavailable to everyone else for the whole run, even while the CPUs on that node are free.
• The normal QoS allows at most 48 CPUs per job and per node. Requests that fit comfortably inside the limits are simpler to schedule than ones sitting at the ceiling.

Related#

• Monitoring jobs — checking state while a job is queued, running, or done
• Submitting jobs — where the resource flags are set
• Why did my job fail? — out-of-memory kills and timeouts
• Why is my job pending? — when a large request is the reason
• Partitions, QoS, and fairshare — limits, priority, and backfill
• Job arrays — checking efficiency across many similar tasks
• Getting help — when the numbers do not add up