DGX Spark code benchmarks, a BFCL bug caught early, and Ox Alpha

Before our new code benchmark lane on the DGX Spark ran its first sample, review found that a limited run of BFCL — Berkeley's function-calling benchmark — would have drawn only the "do not call a tool" prompts. A model that never calls a tool would have scored 100 %, and a normal model would have printed a plausible number that measured the wrong thing, right next to HumanEval and MBPP. That is the kind of number that ends up in a slide deck. It never existed, because we put the benchmark change through an adversarial review before it shipped. This is the day it came from.

This is the build-in-public log of one day: opening a coding lane on our DGX Spark — a coding agent and an IDE that run on the models the box already serves — adding a code benchmark lane so model choices are measured instead of vibed, and taking Ox Alpha, a free, anonymous, 1-million-token-context coding model that appeared on OpenRouter two days earlier, for a first spin through the agent.

Why a coding lane

Since July the Spark has run a 35B mixture-of-experts "brain" (Qwen3.6-35B-A3B, NVFP4 under vLLM, ~100 tokens/s) for our chief-of-staff agent. Our own coding still went to a frontier API. The question was simple: can the box we already own do the coding too, and how would we know?

"How would we know" is the governance half. A model is not a coding model because a vendor page says SWE-bench 79. It is a coding model when it completes tasks in your harness, and the only way to compare candidates honestly is a benchmark you run yourself, the same way, every time.

Day one: the agent, the IDE, the lanes

We evaluated the open-source options with a 15-agent research sweep — seven researchers, seven adversarial verifiers re-fetching every citation, a completeness critic. 159 claims survived, 12 were refuted. The picks:

• OpenCode as the terminal agent: MIT-licensed, installable from Homebrew, plain-JSON config we can version, and it runs inside an editor over the Agent Client Protocol.
• Zed as the IDE: its native agent panel speaks to any OpenAI-compatible server, so the Spark's vLLM and Ollama endpoints are just provider blocks in a settings file.
• Three lanes: the brain on vLLM, a co-resident 33B coding specialist (poolside's Laguna XS 2.1) on Ollama, and OpenRouter as an explicit external lane.

First real task on the brain, through OpenCode over the LAN: read a file, fix a bug, tests pass — 9 seconds. The coding specialist took a new module plus tests in 73 seconds. Claude Code pointed at the Spark through our LiteLLM router did an edit-and-test turn in 16 seconds. None of that left the building.

The outage we caused on the way

Honest caveat, because it is the most useful thing in this post. A smoke test asked Ollama to load a 75 GB model next to the 36 GB brain. Ollama on a unified-memory box classes the whole 128 GB pool as VRAM (its own startup log says total_vram=119.7 GiB) and, by default, gives every model a 262,144-token context window — and it never refuses an overcommit. Weights plus a quarter-million-token KV cache put the machine into a memory-pressure storm: SSH hung, every port went dark, and about an hour later it came back from a hard reset — no shutdown sequence in the journal, so a power cycle or hardware watchdog, not a clean reboot.

The fix was three environment variables (OLLAMA_CONTEXT_LENGTH=65536, OLLAMA_MAX_LOADED_MODELS=1, OLLAMA_NUM_PARALLEL=1) and a rule: models that cannot co-reside with the brain are never on-demand targets. If you run Ollama on a DGX Spark, read its startup log line vram-based default context … default_num_ctx=262144 before you trust any "fits in memory" arithmetic. The weights are only part of the load: the KV cache is allocated on top of them, and nothing in Ollama stops the sum from exceeding the box.

The code benchmark lane

Our bench stack already ran GSM8K, IFEval and friends against any lane through Inspect AI and logged to MLflow. The Inspect evals package also ships the code suites; the runner just didn't allow them. Ten lines to whitelist humaneval, mbpp and bfcl, right?

The adversarial review — 39 agents, three lenses, two skeptics per finding — confirmed 17 findings. Three would have bitten: one a confident wrong number, one a right number for the wrong reason, one no number at all:

1. --limit on an id-sorted dataset is a filter, not a sample. BFCL's samples sort with irrelevance_* first — prompts where the correct answer is to not call a tool. Any limited run would have scored only refusal. Fix: a fixed-seed shuffle, and the per-category breakdown logged next to the headline.
2. pass@k lives in a different place than you think. Inspect reports epoch reducers as separate score entries, not metric names; our pass_at_1 lookup was dead code that happened to return the numerically identical mean. It was right by accident. Now it is right on purpose, and the run is tagged with which view the headline came from.
3. The dataset loader shells out to git. Our slim benchmark image had no git; the suite would have died before sample one.

A fourth finding changed how we run it: the "local sandbox" that executes model-written code is your process — your environment variables, your network. We cap the container's processes, memory and capabilities, strip secrets the run doesn't need, and keep executing suites off lanes that hold a key. It still runs as root inside that container with LAN reach — an accepted, tracked risk on a single-tenant box until the suites move to a network-isolated sandbox.

What ran, and what didn't

The plumbing ran end to end on both lanes — HumanEval on Laguna XS, BFCL and MBPP on the brain — at n = 10 to 40, and every run logged a headline, its reducer, and the per-category breakdown to MLflow. Those are plumbing checks, not results, and this post does not quote them. The real screening — HumanEval and BFCL at 100 samples, MBPP at 50, on both lanes — is scheduled to run overnight as this goes out. The table, with the per-category BFCL spread, gets its own follow-up post, because a single headline number is exactly the kind of thing this post is arguing against.

Ox Alpha was not part of that. The executing suites stay on lanes we control; the external lane got the agent trial below, not the benchmark.

Ox Alpha: free, fast, anonymous — and read the terms

Two days before all this, OpenRouter listed stealth/ox-alpha ("Ox Alpha" — letter O, not zero, whatever the community spelling suggests): a reasoning model "designed for coding, sustained agentic work", 1,048,576-token context, tool calling, $0. It answered a tool call in 5.6 seconds and refactored a small test suite through OpenCode in 35 seconds with every test passing.

Here is the governance part. The model page says prompts and completions "are retained by the provider and are not used for training; all other use is governed by the Stealth Model Terms." Those terms have you grant OpenRouter an irrevocable, perpetual license that includes handing what you send to the anonymous provider "to train, evaluate, and improve" the model. We are not lawyers; we read that contradiction conservatively and treat the terms as the controlling document. So Ox Alpha is a lane for public and throwaway code only — never client work — and that rule is written into the config, the runbook and an architecture decision record, not left to memory. OpenCode's launch post on August 20 called it free "for the next week" — call it August 27; OpenRouter's page gives no end date, the terms allow removal "with or without notice", and the previous stealth model, Owl Alpha, is simply gone from the listing. An experiment, never a dependency.

We are still glad it exists: OpenRouter also lists free hosted versions of two models we run locally — Laguna XS 2.1 and Nemotron 3.5 Lightning; the brain has no free twin — which should give us a clean local-vs-hosted delta once we run it, with the caveat that the hosted side is a different quantization.

What we'd tell a client

• A coding lane on your own hardware is a configuration project first. The serving side was already there; the work was config, guard rails, and measurement.
• Benchmarks lie in quiet ways — sort order, metric naming, a missing binary. Review the benchmark like code, because it is.
• Free hosted models are fine for the right code. "Right" is decided by the provider's terms, and it is your job to read them.

The full plan, research corpus and decisions live in the repo we opened for this work; the reusable lessons are in our ops notes. If you want help deciding what should run on your own hardware and what should never leave it, book a free 45-minute AI Opportunity Briefing at /#contact.