How a Tiny Team Ships Like a Big One

We pointed the agents at ourselves

A fleet of subagents audited our own AI stack - every LLM call site, ten research dimensions, and an adversarial pass that caught two errors in our own brief.

6 min readagentsclaudellmopsevalsaudit

Nine posts into this series, a fair question: if the agents are so useful, why do they only work on our product? So we ran the uncomfortable experiment. We asked the fleet to audit the way we ourselves build with AI - every model call, every prompt, every dollar.

It found four parallel LLM stacks where we thought we had one. It found that not a single call site used prompt caching. And - the part that actually changed how I think about this - it found errors in our own brief. We handed the agents our assumptions, and they came back with citations proving two of them wrong.

The mirror problem

Our AI stack grew the way every startup's does: organically, under deadline, one integration at a time. Each service's corner looked fine to the team standing in it. Auditing the whole thing properly - every call site, every model choice, current pricing, current best practice - is weeks of senior-engineer work, which is exactly why it never happens. And when humans do audit their own decisions, we grade generously.

So we structured the audit the same way we'd structured the SRE fleet: narrow agents, explicit phases, and an adversary.

Four phases, one skeptic

  1. Phase 0 - inventory. Eight explore agents fanned out across the codebase in parallel - 299 tool uses between them - and produced a 2,209-line structured audit of every LLM call site, each claim anchored to a file and line number. Not "we mostly use model X," but this call, this file, this line, this model, this prompt shape.
  2. Phase 1 - research. Eleven dimensions in parallel - gateway architecture, observability, evals, cost, security, caching, and so on - each agent researching current practice and current pricing against what Phase 0 found.
  3. Phase 2 - the adversarial pass. A skeptic agent per dimension, with one job: refute the load-bearing claims, including ours. This is the phase most people skip and the phase that paid the bill.
  4. Phase 3 - synthesis into a blueprint: one readable summary on top, one deep document per dimension underneath.
The self-audit workflow: fan-out, research, adversarial verify, synthesize · click to enlarge

The skeptic earned its keep

The adversarial pass produced eleven authoritative corrections - several to the human-authored vision document the whole exercise was grounded in. Three I'll own publicly:

  1. Our pricing math was wrong by 4×. The brief compared model costs using a year-old price for one provider's model. The skeptic pulled current pricing, recomputed, and a cost gap we'd called 40× was actually 10× - which changed the conclusion about which model belongs on our high-volume extraction tier.
  2. Our call-site inventory was miscounted. We'd said roughly two dozen cascade call sites; the audited number was exactly 16 - we had conflated two different call paths. Migration scope shrank accordingly.
  3. Our proposed eval gate was statistically underpowered. We'd specced "50–100 labeled examples, fail a model swap if any scorer drops 3 points." The skeptic showed that detecting a 3-point drop independently at that sample size is roughly a 10× underestimate - and proposed the fix: a paired comparison on the same examples, gated against each scorer's measured noise floor, plus a seven-day single-tenant canary and a feature-flag rollback measured in minutes.

The part I keep coming back to is who wrote those errors: we did. The agents corrected the humans, with citations - the inverse of the relationship everyone worries about, and now my strongest argument for adversarial verification as a standing phase in any agent workflow.

What the mirror showed

The findings themselves were humbling in the useful way:

  1. Four parallel, uncoordinated LLM stacks - including three TypeScript copies of the same agents and two Python copies of the same insight logic, drifting independently.
  2. No prompt caching anywhere - dynamic values injected into system prompts broke cacheability at roughly 90% of call sites.
  3. Costs estimated, not metered. Our cost tracking assumed flat token counts. That's how the ~$2k-a-month spike you met in the budgets post ran for weeks before anyone noticed.

The blueprint's answer is one idea applied everywhere: one contract, one config. Every call site passes a single call object - tier, service, agent, purpose, tenant, request ID - and every model choice lives in one routing config file, where changing a tier is a one-line PR that must arrive with eval evidence. Cost gets metered per call by the gateway, not estimated. The build-out is underway; it's next quarter's story.

Where the humans sit

Exactly where they sit everywhere else in this series. The agents produced evidence - the inventory, the corrections, the blueprint. Humans had locked the strategic decisions up front in a vision document, and where the evidence contradicted it, humans adjudicated the conflict - usually by conceding to the better math. Nothing in the audit changed a line of production code by itself. The blueprint lands the way everything lands here: as PRs, through gates, merged by people.

Steal this

Before you buy an LLM-gateway product or trust your own architecture instincts, spend one day having agents build the inventory: every model call in your codebase, at file:line, with model and prompt shape. Then spend the second day on the part nobody does - a skeptic pass over your own assumptions, instructed to refute you. The inventory costs a day, the skeptic is nearly free, and being 4× wrong on pricing is neither.

Ten posts ago the build track started with a production bug that fixed itself in twelve minutes; since then you've seen the rails, the reviewers, the budgets, the specs, the hands - and now the mirror. Looking back, the through-line was never really the models: toil goes to the agents, decisions stay with people, and every gate we called non-negotiable still holds - zero agent changes have ever reached production without a human merge.

Next on the product track: on Thursday, what a bank-grade AI-governance review actually asks a six-person company; then the year in our own GitHub numbers; and a standup dashboard with a filler-word leaderboard. See you Thursday.

This post is part of the build track of How a Tiny Team Ships Like a Big One, a series on how six builders run a production AI company. Building at Aithon - if this is how you want to work, talk to us.