How a Tiny Team Ships Like a Big One
Dollar budgets, scope guards, and an agent's right to say no
Trust in autonomous agents came to us as a stack of constraints - six plain layers, and the two invoices that taught us to build them.
One of our engineers once spent $500 in a single day testing against a model API. Months later, a misconfigured gateway burned about $2,000 a month for several weeks before anyone noticed - and the reason nobody noticed is that our costs were estimated rather than metered. Those two invoices taught us more about running agents in production than any paper we've read.
When people ask how we let agents fix production code - the SRE fleet, the QA agent, the feature work - they usually expect an answer about model quality or prompt engineering. What we actually lean on is a stack of constraints. Each layer is simple to the point of being dumb; together they're what lets the fleet run all day without anyone watching it. This post walks the stack from the bottom up.
The stack, bottom to top
1. The IAM floor. Every agent runs on ECS inside our VPC, under a role scoped to what its job needs: invoke Claude on Bedrock, read logs, read the database - read-only wherever possible. Before any cleverness, the blast radius is bounded by plain old cloud permissions. If an agent goes fully sideways, the damage is capped by a policy document.
2. Declared scope, enforced twice. Each run declares the paths it may write. A pre-tool-use hook denies any write outside that scope the moment it's attempted, and the entire branch is re-checked before push. (For a while we layered a global deny-list of protected files on top; as covered in the SRE fleet post, we retired it once the review loop matured.)
3. Dollar ceilings. Every run has a hard budget. We started at five dollars and raised the default to eight once the data showed that the budget acts as a ceiling, not a target - easy runs finish for pennies regardless, so a higher cap only buys more attempts on the hard tail that would otherwise escalate too early. Bigger implementation work gets $10, $24, or $35 depending on the approved spec's estimated size, with turn ceilings that scale the same way (60, 100, 140). The budget is AND-gated with the turn cap, so a run stops at whichever limit it hits first. Above the individual run there's a fleet-level governor too: a daily spend ceiling and a per-item ceiling, plus a cap on how many fixes each model tier may start in a day.
4. Model routing. Severity and sensitivity decide the model. P0/P1 fixes and anything touching security-sensitive paths route to Claude Opus 5; routine P2/P3 fixes run on Sonnet 5 at roughly 40% of the cost. Security-sensitive work also gets a security label, review by the security owners, and exclusion from any model pilots we're running. Spend ends up following the risk curve.
5. The right to say no. This is the layer we'd defend hardest if forced to choose. Refusal, non-convergence, and budget exhaustion all terminate the same way: a needs-human label and a written narrative of where the agent got stuck and why. Never a silent, half-finished, plausible-looking change. We built it this way after realizing that our worst-case scenario wasn't an agent failing - it was an agent failing and papering over it. Across the fleet, that one label is the universal "stop, a person is needed" signal, and agents reach for it the moment a task turns out to be a design decision in disguise.
6. Kill switches and the merge gate. Every agent lane can be switched off with a single repository variable - no deploy, no ceremony. And underneath everything, branch protection: nothing reaches main without a human approving it.
Real numbers
- $8 default per-run ceiling (raised from $5), scaling to $10/$24/$35 for spec-sized implementation work.
- ~2.5× cost difference between the Opus and Sonnet tiers - which is why routing by severity matters.
- 90 turns on the fix lane, 60/100/140 by size on the implement lane, AND-gated with the dollar budget: a run stops at whichever it hits first.
- $75/day fleet ceiling and $40 per work item, on top of the per-run budgets.
- $500 in one day, and ~$2,000/month for weeks: what unmetered spend cost us before this stack existed.
Where the humans sit
Humans set the constraints; agents operate inside them. Every needs-human escalation lands in front of a person with a narrative attached, not a stack trace and a shrug. Humans review everything on the security lane, decide every design question the agents decline, and approve every merge. The stack didn't remove judgment from the system - it moved all of it to the places where a person actually gets to exercise it.
Steal this
Meter, don't estimate. A per-run cost ledger was the single highest-ROI guardrail we built - both of our expensive surprises happened because spend was invisible, not because anyone made a bad decision. Then give every agent the same three things ours get: a declared scope, a dollar ceiling, and an explicit escape hatch. Ours reach for the escape hatch every week, and every time they do, it's a problem we're glad landed on a person instead of in a diff.
Every one of these constraints ends at the same place: a merged commit. What happens after the merge is deliberately the most boring machinery in the company - the deploy rails, next up in this track. Next week, the product track picks the right model for each job.
This post is part of How a Tiny Team Ships Like a Big One, a series on how six builders run a production AI company. Building at Aithon - if this is how you want to work, talk to us.