The benchmark was green and the agents were broken

Our function-calling benchmark scored the model 0.84 — ten points clear of the coding specialist it was screened against. The same model, on the same box, failed every task in our new coding-agent harness. Zero for three, twice in a row, each attempt dying on the first step.

Both numbers were correct. The model was fine. A serving optimization — speculative decoding, in our vLLM stack — was corrupting its output on the way out, and a benchmark suite we had spent two weeks hardening could not see it at all.

The contradiction

Yesterday our 35B generalist "brain" won the code screening on our DGX Spark: 0.99 on HumanEval, 0.84 on BFCL's function-calling suite. The next step was an agentic harness — the same model driving a real coding agent through small git repositories we built as fixtures: read the failing test or the spec, edit the files, run the tests, finish.

The specialist model sailed through, three for three. The brain — the model that had just won the screening — collapsed. One step, zero tool calls, under 150 tokens, done. It would emit a sentence or two and stop, like a contractor who describes the renovation and then drives away.

The chase

Everything plausible got ruled out first, because everything plausible was wrong:

Not the client. The same agent binary and config succeeded from one machine and failed from another; then captured, byte-compared requests showed the two differing only in environment details (working directory, platform string, one tool description).
Not the prompt. Replaying the successful machine's exact request against the server failed too.
Not sampling luck. We swept temperature from the model's default down to zero — fully greedy decoding — and it still emitted mangled tool-call syntax: fragments of its own XML dialect with pieces missing.

The tell came from a probe that costs nothing and takes a minute: send the identical request four times at temperature zero with a fixed seed and hash the outputs. Greedy decoding with a pinned seed is supposed to be boring — the same answer, four times. We got four different answers. One of them contained a raw internal control token — the kind of marker a serving stack should never let escape into user-visible text.

That is not a model with opinions. That is a pipe writing garbage.

The culprit

Our vLLM launch flags included multi-token-prediction (MTP) speculative decoding — a standard optimization where a small draft mechanism proposes several tokens ahead and the full model verifies them. It is designed to be lossless — the full model checks every proposed token, so the output distribution is provably unchanged; quality should not move at all. On our stack, it did. With it enabled, zero of twenty tool-calling requests produced parseable tool calls. We removed one flag and restarted:

Tool-calling requests: 0/20 parseable → 9/9 parseable
Agent harness: 0/3 → 3/3, twice
The benchmark that started this story: 0.84 before, 0.84 after — identical

That last line is the one worth staring at. The corruption showed up on long agentic requests — a nine-thousand-character system prompt plus ten full tool schemas — and barely touched the short single-turn prompts that benchmarks are made of. Our suite wasn't weak; it was structurally blind to this failure class. Every agent-facing client of that model had been degraded for as long as the flag had been on, and every dashboard said things had never been better.

The cost of the fix is real: speculative decoding was buying meaningful decode speed (we measured ~78% draft acceptance). We pay it. A slower correct model beats a faster one that drops tokens from its own function calls.

What we can and cannot claim

Honesty section, because a finding like this invites overreach:

This is our stack: a specific vLLM version in an ARM container, an FP4-quantized 35B mixture-of-experts model, FP8 KV cache, MTP speculative decoding, on a DGX Spark. We bisected to the speculative-decoding flag and stopped; the true culprit could be any interaction underneath it. We found no matching public report and plan to file one upstream rather than declaring the technique broken.
Removing it did not buy byte-level determinism — batching numerics still vary — but output became structurally correct in every probe and every agent run since.
Speculative decoding is not the villain in general. Unverified serving optimizations are. The lesson would read the same for a quantization, a kernel backend, or a caching layer.

The rematch, as a bonus

The same investigation pushed our specialist model onto a current vLLM of its own — it had been serving one request at a time behind our own deliberate guard rail from an earlier incident, and the pinned vLLM version carries a known parser bug that silently drops its tool calls — so yesterday's "specialist is slow" verdict got its rematch: quality unchanged (HumanEval 0.91 vs 0.92, BFCL 0.76 vs 0.74), but the suite the specialist had reasoned its way through in 2 hours 8 minutes finished — together with the full function-calling suite — in 6.1 minutes. Same model in a different quantization, more than twentyfold wall-clock, and quality parity says the gap is a serving story. And its zero on parallel function calls held for a third consecutive deployment: that one really is the model.

What we'd tell a client

Benchmarks validate models. They do not validate serving stacks. Anything between the weights and the wire — speculative decoding, quantization, parsers, caches — can corrupt output in ways a green suite never surfaces.
The determinism probe is free; run it on every serving change. Temperature zero, fixed seed, four identical requests, hash the outputs. Different hashes with structural damage means stop trusting the stack, not the model. It is now a standing gate in our runbook: no engine change ships without it plus a full agent-harness pass.
Agentic evaluation is not optional. Our agent harness surfaced on its first runs what two weeks of conventional benchmarking never did, because it exercises the one thing benchmarks rarely do: long prompts, real tool schemas, multi-step consequences.

If you run models on your own hardware — or pay someone who does — and your evaluation story ends at a benchmark table, this failure mode is invisible to you today. For help building an evaluation practice that would have caught it, book a free 45-minute AI Opportunity Briefing at /#contact.