How a Tiny Team Ships Like a Big One
The agent that files bugs before they happen
Our daily QA agent reads code instead of logs - and fixed a P1 crash that had zero production occurrences. On prevention, precision, and why a silent bot is a feature.
One morning in early July, our daily QA run filed a bug against an endpoint that had never failed. Zero occurrences in production, no alert, no customer report. The crash it described was hypothetical but specific: if a user ever opened a stale bookmark to a record that had since been deleted, the page would return a hard 500. Exactly the kind of link a customer clicks from an old email, months later, when nobody remembers the code.
By that afternoon the fix - a clean 404 and a regression test - had passed review and a human had merged it. The bug went from "will eventually page someone" to "can't happen" without ever showing up in a log.
Last week I wrote about our SRE fleet, which turns production errors into merged fixes in about twelve minutes. This post is about the agent that has been quietly making that loop less necessary.
Why we went looking upstream
For a while, the fleet's reactive speed felt like the whole answer. Then we sat with what those twelve minutes actually contain: the exception has already fired, the error page may already have been seen, and the clock we were so proud of only starts after a customer could be hurt. Once the reactive side was handled, the question that stayed open was how much of our bug surface we could remove before first occurrence.
What we built
The QA reviewer runs daily, on ECS inside our VPC, and reads recent diffs instead of logs. No production error is required for it to act. It walks the code the way a skeptical senior engineer would - what happens if this reference is stale, if this list is empty, if this external call returns garbage - and when it can verify a real failure path, it writes the evidence into an issue.
Before turning it on, we were bracing for a failure mode everyone who has tried an "AI code auditor" has met: the firehose of maybe-bugs that costs more attention than it saves. So every candidate finding goes through a second, adversarial pass we call the reject-guard, whose only job is to kill the finding. Only survivors get filed. In practice that means on a clean day the agent says nothing at all, and it took us a couple of those days to stop checking whether it was broken. It wasn't. It just had nothing worth saying, which turned out to be the exact property that got the team to actually read its reports.
From there, a QA finding joins the same pipeline as a production error: labeled issue, fixer, failing test first, review bot, re-fix if the review demands it, human merge. Prevention goes through the full process like everything else.
Two saves that made the case
The stale-bookmark P1 opened this post, and the detail worth repeating is that there was nothing for a monitoring system to see. No log line, no metric, no anomaly. The bug existed only as a property of the code, and code is what this agent reads.
The second save came within 24 hours of a feature merging. The next morning's run flagged that the new code was re-sending already-processed records to the LLM on every 15-minute sweep - unbounded spend, and the same pattern behind a cost incident we had eaten a month earlier. Then came the part that impressed us more than the catch itself. The auto-fixer looked at the finding, recognized that the remedy was a design decision about how the system should remember completed work, declined to write the patch, and escalated to the author with concrete options instead. The author shipped the real fix, tombstone records, the same day.
Real numbers
From the first ten days of running agents against our own code:
- 37 genuine bugs surfaced, most before any customer impact.
- About 1 in 6 findings was noise - and the reject-guard exists to keep pushing that down.
- One P1 fixed with zero production occurrences, purely from reading code.
- Under 24 hours from a risky pattern being merged to the QA agent flagging it.
That was the opening window. The agent has kept running since, and the shape of the numbers has held up better than the first ten days suggested. As of this week it has filed 47 findings in total, 35 of which are already closed by a merged fix. Three were closed as not-a-bug, and all three came in the first ten days: of the 24 findings filed since, not one has been rejected as noise. The reject-guard turned out to be the part of this system worth the most tuning.
Where the humans sit
Same seat as everywhere in the fleet: a finding is only an issue until its fix passes review and a person merges it. Design decisions get escalated instead of guessed. And the aggressive precision tuning exists for a human reason - reviewer attention is the scarcest resource in this whole loop, so the reject-guard spends model tokens to save engineer minutes.
Steal this
Point an AI reviewer at yesterday's merged diffs, not just at open PRs. Day-old code sat in a sweet spot for us: fresh enough that the author still has full context, old enough that integration effects are visible. Tune for precision over recall, make the bot adversarially re-review its own findings before it files anything, and let it be silent. The week ours said nothing, and was right to say nothing, was the week the team started trusting it.
In two weeks I'll zoom out from the individual agents to the whole operating model: how 100% of our code has been written by AI since April, and why the honest version of that sentence is less dramatic than it sounds. Next week, the product track: teaching agents to read an industry's public exhaust.
This post is part of How a Tiny Team Ships Like a Big One, a series on how six builders run a production AI company. Building at Aithon - if this is how you want to work, talk to us.