Benjamin Meisl · Writing · 5.2

Anatomy of a $2k CI bill

July 2026 · incident postmortem

In July 2026 my CI spend for six days came to roughly $2,000. The first alert was the invoice. The failure had five separate causes, and every one of them is a thing other people's fleets do too.

The billing math that made it possible

My managed CI vendor, Namespace, bills in unit-minutes: the larger of vCPU count or half the RAM, times wall minutes, times a platform multiplier. macOS carries a 10× multiplier. The shape my fleet had standardized on — twelve cores, 28 GB — therefore billed at 140 unit-minutes for every minute of work, against a monthly included pool of 250,000. Thirty wall-clock hours of that one shape consumes the entire pool. Nobody had done that arithmetic, including me.

The blindspot: 18 jobs, one wall clock

The direct killer was a mutation-testing lane: a sixteen-shard test matrix, eighteen jobs per dispatch, every job on the big macOS shape. In a run list, one dispatch reads as an hour or so of wall time, which sounds ordinary. Billing counts job-minutes, so a sharded run costs up to eighteen times what the wall clock shows, and a full sweep at design runtime works out to roughly the entire monthly pool.

Fourteen of those dispatches went out in four days. Most were debugging iterations: an artifact-transfer step in the lane was broken, and the natural way to test a fix was to run the lane. The measured total across the fourteen came to 5,001 job-minutes, about 700,000 unit-minutes, nearly three times the monthly pool — and only that low because most of the attempts aborted partway. A one-shard dry-run mode would have produced the same debugging signal for about a dollar. It did not exist, so nobody used it.

Five causes, all required

No single mistake gets you to $2,000 in six days. Five had to compound.

A posture without a ceiling. Six days before the burn, I had explicitly adopted a speed-over-cost stance and made the priciest shape the fleet default. Nothing wrong with buying speed, except the directive shipped with no spend limit and no per-dispatch cost visibility.

An alert that was designed and never built. The plan from that same day included a weekly usage alert at 70% of pool, and its sizing notes predicted this exact failure mode. It was sequenced behind other work. Namespace has no native budget alerting, so the invoice was the first number anyone saw.

Debugging by dispatch on burst-priced hardware. Iterating on a broken pipeline by re-running the full fleet is a habit from free-tier CI. On metered macOS the same habit has a unit price.

A metrics blindspot. Run-level wall time, the number every casual audit produces, under-reports sharded runs by the shard count. My accounting was diligent and measuring the wrong axis.

Config-override traps. Repository variables silently overrode workflow fallbacks, so shapes I believed I had downsized were still pinned large at runtime in two repos. Every audit of the workflow files was an audit of the wrong source of truth.

What the money actually bought

The fourteen sweeps produced one fix for the artifact-transfer bug and two usable mutation-score baselines. The premium shape on merge gates bought roughly a minute of speedup on runs that already finished in four. The duplicated nightly lane bought nothing; it already ran free on hardware I own. Call it $150 of value on $2,000 of spend.

The fix, same day

Remediation went out the day the invoice landed. Fleet shapes halved. The duplicated nightly moved back to my own machine. The mutation lane was disabled outright, pending conversion to owned hardware, with re-enablement gated on explicit human approval. Machine-enforced guards went in the same afternoon: an agent that tries to dispatch the mutation lane without an approval marker is denied before the call executes, and edits that introduce the big shape draw a warning.

Then the accounting. I built the estimator the plan had specified, job-minutes times shape rate from the provider's own API, and it reconciled the invoice within estimation error: 1.23 million unit-minutes for the six-day window across the whole fleet, 490% of pool, with the mutation lane responsible for about 800,000 of it. The estimator now runs daily and alerts on a 70% trend line. It cost one afternoon to build. I designed it six days before the incident and built it six days after.

The placement policy that survived

Self-hosted goes first, whenever the job tolerates serialization and the hardware has headroom. Managed capacity runs at the modest shape by default. Anything burst-shaped — mutation sweeps, wide matrices — needs a human sign-off per dispatch, enforced by hook rather than by memory.