DGX Spark round 2: two 120B MoEs and the number that lied

The first number out of our second DGX Spark bake-off was 10.83 tokens per second — from a 120B model that llama.cpp's official benches clock at 58.7 on this exact hardware. When your measurement disagrees with a published anchor by 5×, one of you is wrong. This time it was us — and finding out why turned into the most useful result of the whole exercise, because the same bug had quietly corrupted the benchmarks we'd already published.

Here's round 2 of our DGX Spark bake-off: the two 100B-class models we never ran, the first independent numbers anywhere for one of them, and the silent failure that every local-AI benchmark is one systemd reload away from.

Round 2: the two models we never ran

Our first bake-off settled the production question — a 35B mixture-of-experts brain at ~100 tok/s under vLLM. But the DGX Spark has 128 GB of unified memory, and we'd never tested what the big end of that budget buys:

• gpt-oss:120b — 65 GB, MXFP4 MoE, ~5.1B active parameters. Published official number on this hardware: 58.7 tok/s.
• Laguna-S-2.1 (q4_K_M) — 75 GB, 118B total / 8B active MoE. The vendor claims it runs on a single Spark. As of this week, no independent DGX Spark numbers existed anywhere. Ours appear to be the first.

Ground rules, set before running: single fixed prompt, cold loads, the production brain stopped for each run (neither contender co-fits with its ~50 GB reservation), everything restored and verified afterward, and a pre-registered decision rule — the incumbent stays unless a challenger shows a clear quality win at ≥30 tok/s. Deciding the bar before you see the numbers is cheap insurance against talking yourself into your own hype.

The number that lied

10.83 tok/s made no sense, so before blaming the model we asked a more basic question: where did this actually run? One command answered it — Ollama's process list showed the model loaded 100% on CPU. The GPU-side check confirmed it: inside the container, nvidia-smi returned Failed to initialize NVML: Unknown Error, while the host saw the GPU fine.

The failure mode is documented in the NVIDIA container ecosystem: a routine systemd reload can revoke a long-running container's GPU device access. Our serving container had been up 46 hours and had lost the device at least 19 hours earlier (per its own logs); a freshly restarted sibling container on the same host still had its GPU — that asymmetry is the diagnostic tell. The insidious part: nothing crashed. The runtime fell back to CPU, kept serving every request ~3× slower, and kept reporting a healthy pool of free GPU memory from a stale startup probe.

One container restart fixed it. Then came the uncomfortable arithmetic: our published July 24 benchmarks included two Ollama-lane numbers measured through this same container. Re-measured on verified GPU: 24.6 became 61.6 tok/s, 25 became 77.4 tok/s — and our headline "4× serving-stack gap" deflated to ~1.6×. We've corrected the original post and said so in place, because a benchmark post that quietly edits its own numbers is worse than a wrong one.

The real numbers

With placement verified (every row below re-checked as 100% GPU):

Model	Size	Measured decode	Context
gpt-oss:120b	65 GB	42.5 tok/s (real 3,100-token generation)	vLLM stopped, cold load
Laguna-S-2.1 q4_K_M	75 GB	22.6 tok/s (first independent DGX Spark numbers)	vLLM stopped, cold load

Two honest footnotes. The published 58.7 for gpt-oss is a best-case 32-token-generation figure; 42.5 over a real 3,100-token answer is consistent with it, not a contradiction — published numbers measure their conditions, not yours. And the Laguna verdict on the vendor's single-Spark claim: true — it fits, loads, and runs entirely on GPU — but at 22.6 tok/s it missed our 30 tok/s interactive bar, and it needed 9,300 tokens (~7 minutes) to finish a reasoning task the 120B dispatched in 2,700.

Quality: the 120B knowledge showed up exactly once

We ran three fixed tasks against both contenders and the production brain — a briefing over planted notes with four known signals, a scheduling problem with one correct answer, and a strict-JSON tool plan — then had nine blind judges score the anonymized outputs, with an adversarial pass re-verifying every cited failure (including actually parsing the JSON).

Aggregate: brain 74, gpt-oss 60, Laguna 45.5 (of 90). But the texture is the interesting part:

• gpt-oss won the briefing task unanimously — the only model with zero factual errors, and it caught a planted calendar conflict our production brain missed. That's where 120B-class knowledge actually showed.
• The same model shipped a tool plan with one invalid escape character that made the entire JSON array unparseable. Zero of the calls would have executed. For an assistant that lives on tool calls, that's disqualifying, whatever the briefing prose looks like.
• Laguna fabricated a calendar conflict, inverted a fact, got a checkable number wrong while showing no work, and silently dropped half the tool plan.

Verdict, per the pre-registered rule: the incumbent stays. Laguna failed both bars; gpt-oss passed speed but showed no clear quality win. It did earn a niche: at 42 tok/s with the best briefing prose of the three, it's now our on-disk candidate for timer-driven batch work where JSON never matters.

One metric we'll carry forward: tokens-to-answer. On the same reasoning task, gpt-oss needed 2,700 tokens, Laguna 9,300, and our brain — a heavy thinker — 29,800. By raw tok/s the brain is more than twice as fast as gpt-oss; by wall-clock to a finished answer, gpt-oss won. Speed per token is not speed per answer.

What we'd tell anyone benchmarking local AI

1. Verify device placement before recording any number. Silent CPU fallback produces plausible-looking results, and the runtime's own memory dashboard can lie to you. One process-list check per run.
2. Use published benchmarks as anchors, not targets. They flagged our bug (10.83 vs 58.7) and validated our fix (61.6 vs an official 61.06 for a sibling model). Both jobs need you to compare like conditions.
3. Bench each model at its production settings. Forcing our brain to the contenders' low-temperature config sent its reasoning into a loop that never answered; at production defaults it answered everything.
4. Pre-register the decision rule. Ours turned a potentially motivated debate into a mechanical verdict.
5. A failure that degrades silently needs a monitor, not a memory. Ours shipped the same day: a five-minute timer that checks every GPU container, auto-restarts on revocation, and feeds alert rules — because the monitoring container itself turned out to have been GPU-blind through the whole window.

The honest scorecard, as always: one machine, single runs, one prompt set, one afternoon. Round 2 cost us nothing but electricity and returned a corrected public record, the first independent Laguna numbers, and a monitoring gap we'd have hit eventually in production. Cheap.

Weighing local AI for your business?

If you're sizing local AI hardware — or deciding whether your benchmark numbers can be trusted enough to build on — this is exactly what we work through in a free 45-minute AI Opportunity Briefing. Bring your numbers; we'll bring the placement checks.