Our DGX Spark bake-off: 100 tok/s from a 35B MoE, and the dense-70B trap

We just brought a local AI hub online on an NVIDIA DGX Spark, and the model bake-off produced numbers worth sharing — because the intuition almost everyone brings to local LLM decisions ("bigger model, better assistant") nearly had us commit to a build our own records estimated at 4–5 tokens per second. On the exact same hardware, LMSYS has measured the dense-70B class as low as 2.7 tok/s. The model we deployed instead runs at ~100 tokens per second. Here's the physics, the DGX Spark benchmarks we measured ourselves, and the three research traps we hit on the way — including one this post's own draft failed.

The physics: your bottleneck is bandwidth, not memory

The DGX Spark has 128 GB of unified memory — enough to hold a dense 70B model comfortably. What it doesn't have is the memory bandwidth to run one interactively: ~273 GB/s, shared between CPU and GPU.

Token generation for a dense model streams essentially all of its weights per token, so decode speed has a hard ceiling of bandwidth ÷ model size in bytes:

A dense 70B at FP8 (8-bit weights ≈ 1 byte per parameter, so ~70 GB) ceilings under 4 tok/s. LMSYS measured Llama-3.1-70B FP8 at 2.7 tok/s decode on a DGX Spark — patient-adult reading speed, unusable as an interactive assistant. The 4-bit build we'd actually planned projects to ~4–5 tok/s: better, still a batch tool, not a colleague.
A mixture-of-experts (MoE) model routes each token through a small subset of "expert" weights, so it only streams a fraction of itself. Qwen3.6-35B-A3B carries 35B total parameters but ~3B active per token — big-model knowledge, small-model memory traffic.

The rule that falls out: on bandwidth-bound hardware, active parameters set your speed and total parameters set your quality ceiling — MoE lets you buy both. This applies to the whole unified-memory class (DGX Spark, Mac Studio, and friends), not just our box.

What we measured

Our numbers, from our own machine on delivery day. Methodology, stated plainly: single user, one benchmark set, one afternoon. The brain figure is two 500-token generations through the OpenAI-compatible API, wall-clock including prompt processing — which understates pure decode speed. The Ollama figures are single --verbose runs taken while the vLLM lane stayed resident holding its memory reservation (a real-world condition, not a clean-room one).

Correction (2026-07-26): the two Ollama rows below originally read 24.6 and 25 tok/s. Those were CPU numbers — our Ollama container had silently lost GPU access, a known NVIDIA-container failure mode we've since diagnosed and fixed. The table now shows GPU-verified values, and the serving-gap section below has been corrected from ~4× to ~1.6×. Everything else in the post stands.

Model	Serving	Measured
Qwen3.6-35B-A3B (MoE), NVFP4 — NVIDIA's 4-bit format for its latest GPUs	vLLM	97–100 tok/s
Qwen3.6-35B — same model, 4-bit GGUF (the compressed format Ollama runs)	Ollama	61.6 tok/s (corrected; originally 24.6 — CPU-degraded)
Nemotron-3-Nano 30B (MoE, 3.5B active, 1M-token context)	Ollama	77.4 tok/s (corrected; originally 25 — CPU-degraded)

Published reference points, both measured on DGX Spark hardware by others: dense 70B FP8 at 2.7 tok/s (LMSYS, above) and gpt-oss-120B — a 120B MoE — at 58.7 tok/s in llama.cpp's official benches. Those two came from different serving engines, so treat the comparison as directional — but a 120B running ~20× faster than a 70B is not a rounding error. It's the active-parameter rule in action: only ~5B of that 120B works per token.

And the headline gulf, with its attribution shown: our measured ~100 tok/s against LMSYS's published 2.7 for the FP8 dense-70B is a 36× difference across different quantizations and engines. The honest like-for-like — against the 4-bit 70B build we almost shipped — is roughly 20–25×. Either number ends the argument.

The serving-stack gap — corrected from 4× to ~1.6×

This section originally reported the same model running ~4× apart across serving stacks: NVFP4 under vLLM with NVIDIA's tuned configuration, versus 4-bit GGUF under Ollama. That comparison was contaminated — the Ollama side was silently benching on CPU. On GPU-verified runs the honest gap is ~1.6× (97–100 vs 61.6 tok/s).

What survives the correction: the engine tier is still a real decision. A 1.6× decode difference on identical weights — quantization format plus serving stack — is worth having an opinion about. What doesn't survive: our claim that serving choice is "worth as much as a model-size tier." At 1.6×, it's worth about half of one. The fair caveats still apply: the two 4-bit formats aren't bit-identical, the Ollama run shared the box with vLLM's memory reservation, and this is a single-set observation.

We kept both lanes deliberately: vLLM serves the brain at full speed; Ollama serves fallback models, embeddings, and hot-swapped experiments. Everything speaks the same OpenAI-compatible API, so swapping engines is a URL change, not a re-architecture.

Three research traps — the third one was ours

Trap 1: model names are not model availability. Our original plan named a model tag that simply doesn't exist — the family is real; that specific published artifact never was. Nothing fails until first boot. Every model identifier in our configs now gets verified against the actual registry page before it's committed.

Trap 2: benchmark numbers travel without their hardware. A widely-quoted table we nearly relied on contained numbers measured on a different NVIDIA device entirely, sitting unlabeled in a DGX Spark thread (yes, the same discussion we linked above — its benches files are authoritative; its comment tables are a mixed bag). The real Spark numbers were 40–100% higher. Every performance figure in our planning docs now carries the hardware it was measured on.

Trap 3: your own headline is where overclaiming hides. We run an adversarial pre-publish verification pass on everything we ship — including this post. It caught the draft attributing that 2.7 tok/s figure to the exact build we'd planned (records say 4–5), and a title framing a cross-source comparison as an in-house head-to-head. Both are the kind of small inflation that quietly torches credibility with the exact readers we want. The discipline that caught them is the same one from traps 1 and 2: every load-bearing claim gets checked against a primary record before it ships — the approach we've written about in shipping LLM features that survive production.

What we'd tell anyone buying local AI hardware

Do the bandwidth math before the memory math. Capacity tells you what fits; bandwidth tells you what's usable. Divide bandwidth by model bytes for a ceiling before you spend anything.
Default to MoE on unified-memory boxes. Dense models much above ~30B are batch-tier on this hardware class, not chat-tier.
Benchmark on your machine before you commit. Our bake-off took one afternoon and one command per model. It settled what three weeks of research could only narrow.
Pin your serving engine. The next-version container of our engine shipped two weeks before our deploy with vendor-documented regressions on this exact hardware ("use the previous version"). Pinned versions plus explicit upgrade decisions beat :latest every time.

The honest scorecard

One machine, one afternoon, single-user load, wall-clock methodology, 4-bit formats that differ across lanes, and Ollama numbers taken with the other engine resident. Your numbers will differ — call it ±20%. The conclusion shouldn't: on bandwidth-bound local hardware, a well-served MoE in the 30–120B-total range is the interactive sweet spot, and a dense 70B is a trap that looks like an upgrade.

Thinking about local AI for your business?

If you're weighing local AI infrastructure — what to buy, what it can actually run, or whether local makes sense for your workload at all — this is exactly the kind of question we work through in a free 45-minute AI Opportunity Briefing. Bring your use case; we'll bring the bandwidth math.