DGX Spark (GB10) · vLLM

Why vLLM pegs four CPU cores at 100% on the DGX Spark — and the one-line fix

Reworked in English from the Korean original, which has the full experiment log.

If you serve LLMs with vLLM across two or more GB10 machines — NVIDIA DGX Spark, ASUS Ascent GX10, or another OEM variant — check per-core CPU usage during decode. Chances are three or four performance cores are at 100% (about 333% CPU in total), even though the GPU is doing the work. They are not computing; they are spinning in a wait loop. On GB10, where the CPU and GPU share one package and one heatsink, this is also a large part of why these machines run hot enough to shut down. (Default single-unit setups are not affected — see the scope section below.)

Cause. vLLM's inter-process queue spins for up to 1 second (busy_loop_s) before it lets a thread sleep. During decode, messages arrive every few milliseconds — the grace period never expires, so the sleep path is never taken and the cores spin at 3.9GHz continuously.

Fix. One default — 1s → 2ms — cuts vLLM's CPU from 333% to 89% and the SoC by 11°C. Throughput, latency, and GPU utilization are unchanged.

Something didn't add up

I also serve large models on a Mac Studio (M3 Ultra, 512GB). There, LLM inference barely touches the CPU — the GPU does the work and the CPU stays near idle. Against that baseline, two things about the GB10 stood out.

First, the CPU load. Overall utilization read 19.3%, but per-core, four cores sat at 100% while decoding a single request — all of them 3.9GHz performance cores.

Fig 1. Per-core utilization while decoding one request. Four cores pegged, all performance cores; three share one cluster — and that cluster's thermal sensor runs hottest.

Second, the temperatures. The hottest zone on the die wasn't the GPU (72.7°C average under load) but a CPU performance-core cluster (86.5°C average, 96°C peak vs the GPU's 76°C). On a machine that hard-kills itself at 104.8°C, with owner reports of shutdowns from around 87°C, that matters.

The busy threads were vLLM's EngineCore and Worker_TP: always runnable, no kernel wait channel, 27–35% of their time in system calls. Capping the CPU clock (3.9 → 2.8 → 2.0GHz) did not slow decode, so these cores were not a bottleneck. They were waiting at full clock.

py-spy points at one line

A stack dump under load:

Thread (active): "MainThread"
    sched_yield  (vllm/distributed/utils.py:48)
    wait         (…/shm_broadcast.py:196)   ← SpinCondition.wait
    acquire_read (…/shm_broadcast.py:698)
    dequeue      (…/shm_broadcast.py:779)
    worker_busy_loop

The engine process and GPU workers exchange work over a shared-memory ring buffer; the receiving side waits on this SpinCondition:

def wait(self, timeout_ms=None):
    ...
    if current_time <= self.last_read + self.busy_loop_s:   # default: 1 second
        sched_yield()                        # spin: yield and come right back
    else:
        self.poller.poll(timeout=timeout_ms) # sleep: woken via zmq socket

It's a hybrid wait: spin for busy_loop_s (default 1 second) after the last message, then sleep. Everything needed for sleeping — the notify socket, the poller — is already built. But during decode, messages arrive every few milliseconds, so the 1-second grace period never expires. The sleep path exists in the code and is simply never taken.

Physical confirmation

A stack trace names a suspect; before patching I wanted physical evidence. If the spinning performance cores are the real heat source, capping only their clock should produce a temperature drop centered on them, fading with distance across the die. Performance cores 3.9 → 2.8GHz, same boot, alternating runs:

Fig 2. Temperature change by sensor when only the P-core clock is capped: performance cores −14°C, uncore −3.7°C, GPU −1.7°C, NVMe ≈ 0. The second node shows the same shape.

If total system load had simply dropped, every sensor would fall evenly. Instead the drop decays with distance from the performance cores — a point source got weaker. Code analysis and physical measurement independently point at the same spot.

The one-line fix

busy_loop_s: float = 1 → 0.002. Only the waiting policy changes, so model output is identical.

	default (1s spin)	2ms
vLLM total CPU under load	333.6%	88.7%
SoC temperature (same node, same load)	77°C	66°C
Decode c=1 / c=8 (tok/s)	52–63 / 154–173	58.0 / 157.8
GPU utilization	95%	96%

Same load (4 concurrent requests, GPU ~96%). Throughput and time-to-first-token stay within restart-to-restart noise; only the CPU time and the heat disappear.

Why performance doesn't change. A decode step is tens of milliseconds, and most of it is GPU time with no CPU-side messages. With a 2ms grace the thread sleeps through nearly every step, and the writer wakes it via the notify socket in tens of microseconds — once or twice per step, under 1% of step time. Messages that arrive within 2ms of each other still hit the spin path, so the burst-latency behavior the original design aimed for is preserved. The change doesn't alter how vLLM waits; it moves a threshold so the sleep path that already exists actually gets used.

Why the temperature drops. A spinning core executes the check-yield-return loop continuously at full dynamic power, the same as if it were doing real work. A sleeping thread comes off the run queue, and an idle core clock-gates to nearly zero power. The 14–20°C difference between the two is just that power difference.

The remaining 89% is real work — scheduling, sampling, and a separate CUDA stream-sync spin inside PyTorch that this change doesn't touch.

Who this affects

The wasted cores: any deployment using vLLM's multiprocessing executor — the default whenever the world size is 2 or more (TP≥2), including ordinary multi-GPU x86 servers. The value is hardcoded in upstream main, and neither call site overrides it. On a 128-core server with a dedicated CPU cooler, 3–4 spinning cores is about 3% and goes unnoticed, which is presumably why the default has survived. Correction (2026-08-17): a default single-GPU setup (TP=1) selects the in-process uni executor, so this spin path does not exist there — a single Spark on default settings is not affected. An earlier version of this article overstated the scope.
As an overheating problem: every GB10 device. Twenty CPU cores and the GPU share one package and one small-chassis heatsink. Under this load the GPU rail draws ~24W, and four P-cores at 3.9GHz is the same order of magnitude — the spin accounts for roughly half the thermal budget, millimeters from the GPU. It is also 4 of the 20 CPU cores.
Model choice doesn't matter. Any model whose decode messages arrive more often than once per second — in practice, all of them — keeps the spin alive. Each tensor-parallel rank spins on its own.

If you have a GB10

This applies to multi-unit setups (TP≥2), or an explicit --distributed-executor-backend=mp. On a default single Spark there is nothing to fix — step 1 tells you either way.

# 1) Confirm on your box: a few cores at 100% during decode
mpstat -P ALL 3 1
# Thermal zones all report as "acpitz"; the firmware path says which is which
# (TSOC = SoC, TS0P/TS1P = perf-core clusters, TGPU, TUNC = uncore)
for z in /sys/class/thermal/thermal_zone*; do
  echo "$(cat $z/device/firmware_node/path): $(($(cat $z/temp)/1000))°C"
done

# 2) The fix — one line inside the serving container
docker exec $CT sed -i 's/busy_loop_s: float = [0-9.]*/busy_loop_s: float = 0.002/' \
  /usr/local/lib/python3.12/dist-packages/vllm/distributed/device_communicators/shm_broadcast.py
docker restart $CT          # on every node if you run TP

No CPU clock caps are required — the cause itself is removed.

Community tooling. drowzeys/vllm-gb10-spin-wait-fix packages this fix as an automated image-patching script — it builds a -spinfix derived image and verifies it, which survives container recreation better than patching a live container. Credit to that repo for correctly flagging the TP=1 limitation as well.

Appendix: two more things that help

The spin fix removes the cause, but if you want more thermal headroom on a GB10, two other things measurably helped under the same load:

A bottom intake fan. A single fan (a Noctua, in my case) blowing into the underside intake dropped the SoC about 12°C versus an identical node without one — per-sensor gaps of 8–22°C. Pure hardware, no performance impact, works regardless of the software fix.
Capping the P-core clock, 3.9 → 2.8GHz. About −9°C on the SoC, with interactive speed unchanged and roughly −5% at 8 concurrent requests. Before the code fix this was the main mitigation; with the fix in place its remaining contribution is a few degrees. I keep it as a safety net for containers that come up without the patch.

All three combined — spin fix, fan, clock cap — took the worst-case SoC peak from 97°C to 73°C on the same workload: from 8°C below the hard-shutdown threshold to 32°C of margin.

Setup: 2× ASUS Ascent GX10 (GB10, 128GB LPDDR5X each), ConnectX-7 200GbE direct link, vLLM TP=2 (0.25.2-line community build), DeepSeek V4 Flash 0731 original FP8, reasoning enabled. Temperatures logged every 15s on both nodes; performance judged by non-overlapping min–max ranges across restarts. The full Korean write-up adds a fan A/B experiment, the failed remedies, and a clock-cap appendix.