Ox Alpha, unmasked: one BFCL run on GLM-5.3-Flash

At 8:40 a.m. Pacific on August 26 the coding agent on my Mac got an HTTP 404. The error message was a thank-you note:

Thank you for participating in the Stealth Ox Alpha testing period. This model was ZAI's GLM-5.3 Flash. Use it now: https://openrouter.ai/z-ai/glm-5.3-flash

Ox Alpha was GLM-5.3-Flash, and we never ran a benchmark on it. I never read that error either: it sat in the agent's database for 24 days — while the same news sat on Z.ai's and OpenRouter's public feeds — and our August notes still read "Identity is unconfirmed." This post closes the loop our August 22 post opened: what Ox Alpha was, what our records got wrong, and the benchmark that post's draft title promised until review caught it.

What it was

On August 26 at 14:12 UTC, Z.ai's launch post said it: GLM-5.3-Flash had been previewed as Ox Alpha. OpenRouter confirmed two minutes later. Per Z.ai's write-up and the published config: a 320-billion-parameter mixture-of-experts, 18 billion active, a one-million-token context, text, image and video in.

What matters most to us: the Flash weights are on Hugging Face under a plain MIT license — we read the LICENSE file, not the badge. (Only Flash; the flagship GLM-5.3 carries a different license.)

Others had already called it. aitrackerbot matched its token counts to GLM-5.3 about an hour after the listing went up, and labelled the finding unconfirmed. On August 22 a researcher posting as Chetaslua reported a Java stack trace, returned through OpenCode's gateway, whose class path mirrors the documented API route of Zhipu, the company behind Z.ai. Joseph W. Elstner's tokenizer study reports 95 of 95 probes matching the released GLM-5 vocabulary.

The last anonymous night

Our agent, OpenCode, keeps every session in a local SQLite database. It is the best primary record we have — the model id no longer exists — and it contradicted our notes four times.

The night before the reveal we pointed Ox Alpha at a throwaway project: build a blog site from an empty folder. 108 model turns in 89 minutes, from ten prompts of ours; 154 tool calls. Thirty-seven minutes in it committed 42 files and 2,876 lines — a 20-page site that built clean when we rebuilt it on September 19. It read 5.3 million cached prompt tokens to write 71 thousand.

From 12:31 a.m. four requests failed on rate limits — first a bare provider 429, then three times "temporarily rate-limited upstream" — two of them mid-run. Not our account's limit: these were the last hours of what OpenRouter called its biggest model ever by traffic — over 20 trillion tokens in six days. The next request, at 8:40 a.m., was the 404 above. The listing had been switched off between 13:30 and 13:50 UTC — about 6:40 a.m. Pacific, minutes before the real name went up.

Our tool-call tally was off too: a per-session count credited another model's work to Ox Alpha. Count tool calls by the model that made each one, not by session.

The benchmark we finally ran

Ox Alpha was never benchmarked. Part of that is a rule: our harness refuses code-executing suites (HumanEval, MBPP) on a hosted model, because the model's code would run beside a real API key. That does not explain BFCL, Berkeley's function-calling benchmark: the single-turn categories we run execute nothing. The scorer compares the emitted call, function and argument values, with the expected one — or, on 34 of our 100, checks that the model correctly called nothing. On August 23 we ran exactly that against two other hosted models, decided the same day to drop Ox Alpha when its window closed, and never gave it a bench lane. That one is on us.

Strictly, it still hasn't been benchmarked: late on September 19, Pacific time, we tested the released z-ai/glm-5.3-flash, and nobody outside Z.ai can say it is the same checkpoint that served as Ox Alpha. Same pinned harness versions, same seed, and the identical 100 samples our local models had already answered.

A hosted open-weight model is not one thing. 28 providers serve these weights on OpenRouter today, four tagged 4-bit, and OpenRouter's default routing weights providers by the inverse square of price: a host at half the price is four times as likely to get an unpinned request. We have not measured whether the 4-bit hosts score differently; we pinned so we knew what we had measured — Z.ai's own endpoint, fp8 (the precision the weights shipped in), fallbacks off. It served 100 of 100.

The table is not a ranking. Every model answered the same 100 problems, so we can compare pair by pair. At low effort, GLM-5.3-Flash and the brain were both right on 79 and both wrong on 13; of the eight they split, three went to GLM and five to the brain. At default effort — the like-for-like setting — the split was 2 to 5. Against Nemotron, 3 to 6. An exact McNemar test on 3-against-5 gives p ≈ 0.73. We could not tell them apart, and this design could not have: with eight disagreements, only an 8–0 split would count as significant.

• Reasoning effort is a benchmark parameter. The model reasons on every request and defaults to maximum. Default effort spent 26 times the reasoning tokens of low (12,895 against 489 across 100 calls) and cost about half as much again. The score did not move: 81 against 82, with seven problems flipping, four one way and three the other. These short problems barely trigger reasoning, so this says nothing about long agent sessions. State the effort next to any number you publish.
• The whole exercise cost under four cents — 200 benchmark calls plus a few demo and editor runs: $0.038 on the key's usage counter as we read it; the token arithmetic agrees.

And the caveats, which are the point. One suite, 100 samples, single-turn: it does not measure whether a model can code or sustain an agent session. A model with 18 billion active parameters landing level with one with 3 billion says more about the test than the models — on the 90 non-parallel problems, all nine runs we have (headline scores 0.74 to 0.85) land between 72 and 77. It is not a hard ceiling: only 6 of the 100 were missed by every run. The sample is simply too coarse. Z.ai reports different benchmarks (Terminal-Bench 2.1, DeepSWE) and no BFCL figure to check ours against, and ours is not comparable with the public BFCL leaderboard. The hosted runs happened from a Mac, through a different Inspect adapter, four weeks after the local ones, once per setting; the raw logs sit in our private repo, not in our MLflow history.

In the editor

OpenRouter reaches our editors five ways: OpenCode in a terminal; Zed's built-in provider; OpenCode inside Zed; VS Code's bring-your-own-key endpoint; and the OpenCode extension in VS Code. We have verified one of the five, once — OpenCode in a terminal, one task, one run: a bug fixed in about 30 seconds, five tool calls, 3 of 3 tests passing, graded by a script rather than by eye. OpenRouter's usage counter moved by $0.00248 — exactly Z.ai's list price for those tokens, and twice what the cheapest host would have charged. That bill is our evidence the pin held; the trace does not name the host. The other four paths are declared, not yet run by hand. We will say so until they are.

The rule that goes with it: a hosted model gets no shell. Our demo config denies it bash; it edits a scratch copy and we run the tests.

A fair trade

In August we read OpenRouter's stealth terms conservatively, and we would again. Three documents gave three answers: the blanket terms then in force let OpenRouter pass your content to the anonymous provider for training; the listing said prompts were retained and not used for training; OpenCode's route advertised zero retention. On September 14 OpenRouter rewrote the terms so the listing governs — the right fix.

We think the trade was fair for anyone who read it the way we did. The price of six free days was your prompts, which under the blanket terms could go to the then-anonymous provider for training, whatever the listing said. We sent public and throwaway code only; someone who trusted the listing and sent client code made a different trade. We are not lawyers.

Then Z.ai published the Flash weights under MIT — no usage restrictions, no separate commercial license. We can't run them in any way we'd trust — at least 160 GB at 4 bits against our box's 128. But open weights are why 28 providers now serve it, the cheapest under ten cents per million input tokens against Z.ai's own $0.15; why we could choose a precision; and why this model cannot be withdrawn the way its stealth listing was.

The listing is gone. The model isn't.

What we'd tell a client

• A stealth model has a soft end date. OpenCode said about a week; six days in, the only notice that reached us was the 404.
• Read every document that governs your data, and plan as if the one that protects you least is the one that applies.
• Behind a router, pin the provider and the precision — then check the bill.
• Your tool's own database outranks your notes about it.
• Compare models on the same samples, pair by pair, and publish the reasoning effort.

We've built this into a talk (not yet scheduled); the replay build is at route-it.rawrtech.ai. If you want help deciding which models your code should ever be sent to, book a free 45-minute AI Opportunity Briefing at /#contact.