I Built a Family AI to Keep Our Data Home. Then I Broke It With Bad Math.

I don't need a vendor's permission to run my own AI. That's the whole premise — not a subscription, not a shared server somewhere, not someone else's retention policy deciding what happens to my family's files. I own the hardware. I own the boundary. Nobody gets to tell me what's on it or where it goes, as long as I'm not doing anything I shouldn't be — and I'm not. Everyone in the house gets a private folder, a shared one, and access to a model running in the next room.

There's a contradiction sitting right in the middle of that, and I'm not going to smooth it over: I built this whole thing specifically so our data doesn't leave the house, and I still pay Anthropic by the token when a problem is genuinely too hard for the local model. Sovereignty doesn't mean never trusting anyone else. It means I decide when the exception is worth it. Nobody decides that for me.

The actual building — turning "I want this" into a working system — happened with Claude, session by session, over a short-time, mabe a week?. Worth saying plainly, because it matters for what comes later in this piece, not because a tool needs credit for doing what I told it to do.

The setup

Two boxes, always on. One does the thinking — three NVIDIA GPUs crammed into a Linux machine. The other runs the front door, and this is the part I didn't plan, just happened to have lying around: a 2012 Intel Mac Mini, fourteen years old, doing absolutely nothing in a closet until I put Linux on it and made it the router for the whole family — the chat app everyone logs into, the tool servers, the decision about when to escalate to the cloud. You don't need new hardware to build something new. You need hardware that still boots.

Inference box: one 8GB card (3070 Ti), two 12GB cards (3060s), 94GB system RAM. Front-end box: 2012 Intel Mac Mini, repurposed with Linux — OpenWebUI, custom tool servers, always on.

Two decisions made this actually usable instead of a toy. I skipped Ollama's hard limit of one request at a time and ran llama-server directly — same binary Ollama uses under the hood, just without the training wheels. And the front end talks to a stable model alias, not a filename, so swapping in a better model — which has happened twice now — needs zero reconfiguration on the chat side.

Escalating to Claude costs real money, tracked against a small balance so nobody spends it blind. That balance shows up on a dashboard I built — which turned out to matter a lot more than I expected.

The dashboard did the actual work

Before settling on the model I'm running now, I trialed several. Different sizes, different architectures — you can read a spec sheet all day and it won't tell you how a model actually behaves on hardware this mismatched.

What told me was a live GPU tile on that dashboard — per-card VRAM, utilization, temperature, updating in real time while a model sat under real load. Watching which card filled up first, whether it degraded or just died, how much room was actually left once the cache warmed up — that's not in a README.

Every VRAM number in the rest of this piece came from that tile. I checked it against nvidia-smi directly during the incident below and it held up. That's the only reason I trusted it enough to make decisions off it.

Mixed cards: the part nobody warns you about

One 8GB card, two 12GB cards, one model split across all three with llama.cpp's --tensor-split flag. Forum wisdom says: give the small card a smaller slice. True. Not the whole truth.

A split ratio is layers, not memory. --tensor-split 1,3,3 reads like "GPU0 gets one-seventh of the VRAM load." It's actually one-seventh of the layers — and layers aren't created equal.

64 total layers: 48 cheap linear-attention, 16 full-attention with much heavier state. 5120 hidden dim, 24 query heads, 4 KV heads.

Two cards holding "equal" 3/3 shares can end up holding a very different mix of the expensive 16. Found that out the hard way:

Free VRAM at steady state — 851MB / 799MB / 113MB across the three cards.

Two cards, identical split, headroom differing by roughly 7x. Don't trust the ratio. Trust the meter — after real traffic, not right after a restart.

Some things don't split at all. The vision encoder — 1.84GB — doesn't participate in --tensor-split. It loads entirely onto whichever card is "device 0." Mine happened to be the 8GB card. Obviously.

GPU0 usage with the vision encoder removed: 7001MB → 4983MB, a real ~2GB drop. The two 12GB cards barely moved (11,121→11,113MB, 11,807→11,799MB) — noise.

(The first reading right after restart showed all three cards with roughly 5GB more free each. That was an empty cache, not the fix — it refilled within minutes. A lesson inside the lesson: don't trust numbers taken right after you touch anything.)

Vision is off now, permanently.

The outage, in full, mistakes included

The tightest card wasn't the 8GB one. It was one of the 12GB cards, sitting at that 113MB — under 1% — free. The plan: nudge a sliver of layer-share off the tight card onto the 8GB card, which had some room.

First move: 1,3,3 to 2,3,2.

1/7 (14.3%) → 2/7 (28.6%) share for the 8GB card. Result: cudaMalloc failed trying to allocate a 1.46GB compute buffer — more than the card's 851MB headroom could cover. Server degraded gracefully. Landed at roughly 92% / 93% / 64% VRAM used.

No outage yet. Just a very clear no.

Then I made the actual mistake. Looking for something safer, I moved to 3,4,3 — reasoning that a smaller first number meant a smaller share.

It doesn't work like that. Ratios normalize against their own total, not some fixed denominator.

2 of 2+3+2=7 is 28.6%. 3 of 3+4+3=10 is 30%. Not smaller. Larger.

I'd talked myself into backing off while doing the opposite, on a card that had already failed once at a lower number. This one didn't degrade. It took the whole thing down. nvidia-smi afterward: three idle cards, nothing running.

Recovery was fast, because I'd backed up the working config before touching anything — restored it, checked it matched the pre-incident baseline, done in minutes.

Restored free VRAM: ~851 / 799 / 113MB. Matched the original reading almost to the megabyte.

Here's the actual lesson, and it's not really about GPUs: there's no such thing as a safer-sounding change on a system running close to zero margin. Test it somewhere that isn't live. Every time. A second AI was cross-checking my reasoning in real time while I did this and still didn't catch the normalization error until after it had already taken the service down. A second opinion is not the same as re-doing the arithmetic yourself.

What that margin actually costs

Not abstract. Tight VRAM has a direct, felt cost: the system now runs 2 concurrent conversations at 114,688 tokens each, instead of 4 conversations at 24,576 tokens each.

229,376 total context ÷ 2 slots = 114,688 tokens/person. Previous config: 4 slots × 24,576 = 24,576/person. Roughly 4.7x more room per conversation, half the concurrency. Model's trained ceiling: 262,144.

I'd take that trade again. Just worth knowing it's a trade, not a free upgrade.

The lever I found and didn't touch

Quantizing the K and V cache at different precisions would claw back headroom on the tight card. I nearly did it. Then I found an open llama.cpp issue: this exact setup silently falls back from GPU to CPU on some builds — no error, no log line, just a catastrophic slowdown, unless the binary was compiled with a specific flag you can't check for at runtime.

Reported regression: 25–45x on prefill. One confirmed case: 30.6 tok/s vs 1340 tok/s — a ~44x collapse, GPU allocated, compute quietly running on CPU.¹

Still on the "maybe later, benchmarked in isolation first" list. A free memory optimization that silently costs you an order of magnitude of speed is not free.

The smaller stuff that ate more time than it should have

ls, aliased to something modern that emits color codes even when piped — broke a restore script twice. Once by mangling a path mid-copy with no error surfaced. Once by wrecking the sort a backup-picker relied on. command ls fixes it. Took longer than I'd like to admit to find.

No alerting. A dashboard nobody's watching at 3am isn't monitoring — it's decoration. The outage above would have gone completely unnoticed by anyone not staring at a terminal at the exact moment it happened.

Backup coverage that looked complete and wasn't. The box actually running the model had no backup job at all. A couple of things that can't be regenerated — an encryption key, some config — were quietly absent even from the machine that was being backed up.

A router I'd never actually checked. Firewalled the API port at the host level, confirmed it — and never once verified whether the router itself was forwarding anything to the outside world regardless. Two different questions. I'd only answered one.

NVIDIA shipped something (PAIR) that routes requests across machines on a home network.² Tempting, wrong tool — it distributes independent requests, it doesn't pool VRAM or split one model across boxes. Which was my actual problem.

What's actually working

Easy for a piece like this to read as one long list of things on fire. It isn't. This runs daily, for multiple people, and has for months.

The stable alias meant two model upgrades cost zero reconfiguration. The escalation path means the local box only has to be good at most things, with a visible, dollar-figured off-ramp for the rest. Treating anything fetched from the open web as data, never instructions, in the one tool I built that way from day one — turned out to be the right default everywhere, not just there. And backing up the working config before touching anything live is the single habit that turned a real outage into five minutes instead of an afternoon rebuilding from memory.

One more, and I'm not going to dress it up: a good chunk of this got built talking to Claude across many long sessions, and Claude forgets things too. Long conversations get compacted to fit a context window, and detail doesn't always survive that intact — an exact reading, a config already ruled out, why something got decided a particular way. More than once an audit rediscovered something already written down, not because it was new, because the note hadn't been re-read yet. Not a knock on the tool. Just a fact about how long AI conversations work — and the actual reason the living doc behind this whole project isn't decoration. It's the fix. Point a fresh session at the doc before trusting anything from memory. Every time.

---

Notes

1. Documented in llama.cpp issue #24485, referencing a confirmed case in issue #20866.
2. NVIDIA's Personal AI Router (PAIR) — routes independent requests across machines, doesn't shard a model's weights across them. Different problem, different tool.