The zero that wasn't a bug: DGX Spark code benchmark results

The overnight run finished at 04:58. The Slack message that announced it said "0/6 legs scored". Six legs had scored. The notifier's pattern had anchored on the start of the line; the harness pads its output with spaces. The numbers were fine all along. That is the whole post in miniature: the table is the easy part, and everything interesting is in what you do before you trust it.

Yesterday's post opened a coding lane on our DGX Spark and promised the screening table. Here it is, plus the one result that looked like a bug and turned out to live in the model, not the harness.

The two lanes

Both run on the same 128 GB box, side by side, and neither left the building.

• The brain: Qwen3.6-35B-A3B, a 35B mixture-of-experts generalist (3B active), NVFP4 under vLLM — it already runs our chief-of-staff agent.
• The coding specialist: poolside's Laguna XS 2.1, a 33B model built for code, Q4_K_M under Ollama, with yesterday's guard rails (64K context, one model, one request at a time).

Suites at screening size: HumanEval 100 problems, BFCL 100 samples (a fixed-seed shuffle across its eleven Python categories; nine drew samples at n = 100), MBPP 50 problems × 5 attempts at temperature 0.5. Inspect AI ran everything; MLflow kept the scores. One run each — these are screening numbers, and the caveats section is not optional.

The table

The generalist brain won HumanEval and BFCL outright and fought MBPP to a draw — pass@1 0.956 to Laguna's 0.96 is one generation in 250; pass@5 0.98 to 0.96 is one problem in 50. The wall-clock column is the part that changes a decision: the brain finished its three legs in under an hour; the specialist took four hours thirty-eight minutes for the same work — the whole five-and-a-half-hour night was mostly waiting for it.

Two things hide inside that.

The specialist thinks out loud, a lot. On HumanEval, Laguna produced 597,000 output tokens for 100 problems — about 6,000 per problem, most of it reasoning. The brain produced 227,000. That is a 2.6× token gap before serving even enters the picture.

The serving stack multiplies it. Ollama on this lane is deliberately single-request (that was the guard rail), so the harness's two concurrent connections queue. vLLM ran the brain's leg with four connections in flight. The 5.6× overall wall-clock gap (8.8× on HumanEval; on BFCL's short answers Laguna was actually the faster lane, 4 minutes to 5) is roughly "2.6× more tokens, served one at a time" — a deployment comparison as much as a model comparison. Laguna on an engine that serves requests in parallel should close some of it; we have not run that yet.

MBPP pass@5 hides a small contrast. The brain's five attempts recover a problem (0.956 → 0.98); Laguna misses the same two problems all five times, so its pass@5 never moves. Consistently right — and consistently wrong.

The zero that looked like a bug

BFCL's per-category breakdown is why yesterday's post refused to print a headline number alone. Here it is (per-category n ranges from one to forty-one, most under a dozen — read direction, not decimals):

Laguna scored zero on every sample in the three parallel categories — ten samples where the correct answer is several tool calls in one turn. It beat the brain on irrelevance and tied on live_irrelevance — the categories where the right answer is to decline to call a tool. The zero is not noise — ten for ten, reproduced on demand below. The small edges elsewhere are; but the direction is consistent with a model that prefers one call, or none.

A zero is where a benchmark is most likely measuring the harness, not the model, so we chased it down the stack before writing a word about it:

1. The per-sample logs. On all ten parallel samples Laguna returned exactly one tool call — always the right first one — while its prose announced the whole set: "I'll calculate the factorials for 5, 7, and 9 …" The brain returned the full set on the same samples (and got two of the ten wrong on argument values, which is why it scored 0.75 and 0.80 rather than 1.00).
2. Ollama's parser. Ollama handles this model with a built-in parser rather than a chat template. We read its source at the exact version we run: it loops — after each closing tool-call tag it scans for the next — and injects no stop sequence. Three calls would have come through.
3. The raw output. We rendered the prompt by hand in the model's native format and read the unparsed completion at temperature 0: one tool-call block, then end-of-sequence, ~45 tokens total. Same with reasoning on; same with a system prompt that demanded every call in one turn — it wrote "in parallel" and stopped after one.
4. The loop. Fed the result of the first call back, it called the second, then the third, then answered correctly. Three tool turns instead of one.

So: the model, not the parser. Under this quantization and Ollama's renderer, Laguna XS 2.1 emits one tool call per turn, even when told not to — whether poolside's own serving behaves differently is a question we have not asked it. In an agent loop that is a style: it gets there, one round trip per call, and its tool-call turns are terse — about 45 output tokens in our probe. (Its afternoon went elsewhere: ~6,000 output tokens of reasoning per HumanEval problem, served one request at a time.) On BFCL's parallel categories that style scores zero by construction. Both statements are true, and a headline "0.74" would have told you neither.

What changes

• The brain stays the default coding lane. It won two suites, tied the third, and is 5.6× faster in our deployment. It was already the default on a hunch; now there is a table behind it.
• The specialist is not fired, it is re-queued. Laguna's one-call style and its reasoning budget matter far more than a 7-point HumanEval gap on a near-saturated suite. The fair next test is Laguna on an engine that serves requests in parallel, so the serving excuse goes away — that run decides whether the lane stays.
• Per-category breakdowns stay mandatory. One of the two lanes would have shipped with a hidden zero behind a respectable headline.
• Notifications get the same review as benchmarks. The "0/6" header was a one-line fix — and it was the first thing anyone read at breakfast, saying the night had failed.

Caveats, because screening numbers are screening numbers

• One run each, screening size. Enough to rank two lanes and find a zero; not enough to claim a 0.92-versus-0.99 gap is stable to the second decimal. HumanEval in particular is close to saturated for current models.
• BFCL categories are small. Seven of the nine have fewer than a dozen samples. The zero is a real zero (ten for ten, with a mechanism we reproduced on demand); the small differences elsewhere are noise.
• Different quantizations, different engines. Q4_K_M on Ollama against NVFP4 on vLLM. The score comparison is what you would actually deploy on this box today; the speed comparison is as much about the engines as the models.
• MBPP and HumanEval execute model-written code inside a hardened container that still has LAN reach — the accepted, tracked risk from yesterday. Nothing misbehaved; that is an observation, not a guarantee.

What we'd tell a client

• Run candidates through your suites before believing a vendor page. A coding specialist lost to a generalist here, on its home turf.
• A benchmark zero is a question, not an answer. Ours took four steps and one morning to turn into a fact about the model — and the fact was more useful than the score.
• Measure wall-clock in the deployment you would actually run. Tokens and concurrency decided this table as much as accuracy did.

If you want help deciding what should run on your own hardware — and how you would know it works — book a free 45-minute AI Opportunity Briefing at /#contact.