A cluster beside the DGX Spark: cheap resilience, and the JetPack 7 trap

Earlier this summer we brought a local AI hub online on an NVIDIA DGX Spark and benchmarked it in public: a 35B mixture-of-experts model serving at ~100 tokens per second, with a live failover demo we later took to a stage. It worked. It was also a single point of failure. The box served the models, watched itself, and held the only copy of its own state — and a box that monitors itself has a blind spot exactly where it matters, while a box holding its only backup isn't backed up at all.

So we built a second, much smaller machine to stand next to it: a four-node Turing Pi 2 cluster — two Rockchip RK3588 boards, one NVIDIA Jetson Orin NX, and the carrier's built-in management controller — drawing about 25 watts at idle for the whole thing. It exists to give the Spark the three things a single box can't give itself: eyes from outside, a copy off the box, and senses it can borrow without spending its own memory. All three sit behind the same front door.

Here's the build, the honest numbers, and the trap we hit flashing a Jetson in 2026 — where "GPU node" turned out to mean CPU, and why we shipped it anyway.

The idea: an independence plane, not a bigger box

The instinct when a rig feels fragile is to make it beefier. That's the wrong axis. The Spark isn't short on compute; it's short on independence. Three jobs specifically get worse the more you centralize them on one machine:

• Monitoring. If the box running Prometheus is the box that fell over, your dashboard went down with the thing it was meant to warn you about.
• Backups. A snapshot on the same disk as the original is a convenience, not a backup. It has to live on other hardware.
• Side-cars. Speech-to-text, embeddings, a reranker — each is small, but each one you co-locate is memory the main model no longer has. On a unified-memory box, every byte a side-car takes is a byte off your context window.

None of those needs a fast machine. They need a different machine. That's what the cluster is: an independence plane, on the cheapest ARM hardware that will hold Docker.

Capability one: eyes from outside

The first node — an RK3588 with 32 GB — runs its own Prometheus, Alertmanager, and Grafana, and scrapes the Spark across the LAN. It's a full, second observability stack whose only job is to watch the main box from a machine that shares none of its failure modes.

The payoff is the kind you only appreciate mid-incident: when we kill a service on the Spark, the alert fires on the neighbor — and the neighbor is still standing to fire it. Self-monitoring answers "is the process up?" right until the moment it can't answer at all. Monitoring from the next machine over answers it exactly when you need it.

Capability two: a copy off the box

The second RK3588 runs a Kopia backup server — encrypted, content-addressed snapshots, taken nightly, with every other node and the Spark as a client pushing to it. The important word is off: the Spark's stacks and application state now live on separate hardware, on a separate disk that the Spark itself cannot accidentally wipe.

We didn't take "the backup ran" as proof. A backup you haven't restored is a hope, so we restored one to a scratch directory and diffed it. The state survives the box now — verified, not assumed.

Capability three: borrowed senses

The third node is the interesting one: a Jetson Orin NX, 16 GB, capped at 25 watts. It runs the senses the Spark shouldn't have to host itself — a Whisper speech-to-text and text-to-speech server, and an embeddings-plus-reranker server, both speaking the OpenAI API.

The trick that makes them feel native is the front door. Every model request in our estate already goes through one LiteLLM endpoint that speaks the Anthropic format and routes each named lane to the right engine. Adding the Jetson was four new lanes — senses-embed, senses-rerank, senses-stt, senses-tts — whose only difference from the local lanes is that their address points at another machine across the LAN. A client asks LiteLLM to transcribe some audio; LiteLLM hands it to the Jetson; the client never knows a second computer was involved. The Spark gained speech and retrieval and spent none of its 128 GB doing it.

The trap: JetPack 7 is a container reset

Here's the part that cost the most time, and the reason this post exists.

We flashed the Orin on JetPack 7.2.1 — brand new, released weeks earlier. Almost everything we knew about running GPU containers on the previous Jetson generation was wrong:

• The l4t-base / l4t-jetpack images everyone builds from are frozen at the last generation. There is no JetPack 7 tag. The correct base is now NVIDIA's unified Arm CUDA image, a different lineage entirely — a guide that hands you the old tag hands you a 404.
• docker run --runtime nvidia, the incantation that gives a container the GPU, injects nothing on JetPack 7 unless the container also asks for the devices by name. The runtime is necessary and no longer sufficient.
• And the one that decided the whole design: there is no native ARM64 CUDA build yet for any of the inference servers we wanted. The Whisper server's engine is CPU-only on Arm; the embeddings server's GPU image is built for a different chip generation. The ecosystem simply hasn't shipped images for hardware this new.

So the "GPU node" runs its models on its CPU. And the honest result: it's fine. Whisper base.en transcribes roughly three times faster than real time on the Orin's six Arm cores, the embeddings and reranker answer in milliseconds, and the whole senses stack sits in a few gigabytes of RAM. The GPU gets used the day upstream publishes the images — at which point it's a package install, not a rebuild. Shipping the CPU version now cost us nothing and unblocked everything.

The builder lesson, plainly: on brand-new accelerator hardware, the silicon arrives a season before the software does. Deploy on the CPU, keep the GPU path one flag away, and don't let a benchmark you can't run yet block a capability you can ship today.

The takeaway

Resilience didn't come from a bigger machine. It came from a few hundred dollars of Arm boards doing the three jobs the main box can't do for itself — watching it, backing it up, and lending it senses — all behind a front door that doesn't care which machine answers.

If you're running local AI on one good box, that's the move: not a second good box, but a cheap, independent plane beside it. The main box gets to be the main box. Everything that makes it trustworthy runs somewhere else.