Anatomy of our SRE fleet: from CloudWatch error to merged fix in 12 minutes

On a Friday a few weeks ago, at 08:17 UTC, the accounts list - the front door of our product - started throwing 500s. One data row with a bad value was poisoning the entire page for any customer whose query touched it.

At 08:29, twelve minutes later, a pull request was waiting for review. It had the root cause, the recurrence count, and a test that failed on the bad row and passed with the fix. Nobody on the team had opened the logs yet, and no customer ever noticed anything.

That PR was written by our SRE fleet, a set of autonomous agents running Claude on Amazon Bedrock inside our own VPC. They detect, triage, fix, and review production bugs in our monorepo, and a human merges every single one of their PRs. This post walks through how the system works and what it took to get here.

What Fridays used to look like

Before the fleet, that same Friday would have gone differently. Someone spots the alert around 9:30, between meetings. They pull up CloudWatch and start grepping. By the time they've reproduced the bug locally it's lunch. The fix itself takes twenty minutes; the test, the PR, and chasing a reviewer eat the afternoon. We're a handful of engineers running a product surface that would keep a much bigger org busy, so each of those afternoons came straight out of the roadmap.

Somewhere in the spring we started asking how much of that loop actually needed a person. The uncomfortable answer was: the merge decision, and not much else.

What we built

We didn't build one big "AI SRE" with god permissions, partly on principle and mostly because we wouldn't have trusted it. What we ended up with is a fleet of small agents, each with one job, its own trigger, a writable scope, a model, and a budget. Here's Friday's bug moving through it:

1. The detector runs hourly over production CloudWatch logs. That morning it saw four occurrences of the same error signature inside an hour (three is the threshold), checked the signature against its ledger to make sure this wasn't already known, and filed a GitHub issue with a machine-readable block: severity, top stack frame, recurrence count.
2. The fixer picked the issue up inside our VPC, on ECS. It read the logs and the production database (read-only), reproduced the failure, wrote the failing test first, then the fix, and opened a draft PR at 08:29. Severity decides which model a fix runs on: this one was customer-facing, so it got Claude Opus 4.8. Routine P2/P3 work goes to Sonnet instead, at roughly a fifth of the cost.
3. The review bot - the same one that reviews our human PRs - looked at the diff and pushed back: the fix was silent. The guard swallowed the bad row without logging anything, so a recurrence would have been invisible.
4. The review-addresser, a second agent, picked up the requested change, added the diagnostic logging, pushed to the branch, and re-triggered review.
5. A human merged it that afternoon.

Log errors aren't the only way in anymore. Every CloudWatch alarm, production and staging, hits a small dispatcher that classifies it by a name policy (there is no allowlist; an unknown alarm gets filed, never dropped) and opens an incident issue in the same machine-readable format. Page-level severities land directly in the fixer's lane; the rest queue for triage. One pipeline, two front doors.

The rest of the fleet exists because things go wrong. A watchdog runs hourly and relaunches crashed or stalled runs; agents die like any other process, so every run records a resumable session ID. An autosync job keeps open fix PRs current with main. And when main moves under an open PR and the update comes back dirty, a conflict resolver trial-merges main and reasons about what it finds; it can reconcile, supersede, or escalate, but it never merges anything itself. On its first live run it noticed that main had already superseded the PR it was trying to save, closed it, and explained why, commit by commit. We had planned to check its work for a few weeks; it made that feel unnecessary on day one.

The unglamorous parts

Most of what makes this work never shows up in a demo.

The fleet's memory is a DynamoDB ledger, not any agent's context window. It holds finding signatures for dedup, per-run state, and daily caps. That's what lets agents be disposable: a run can crash at any point and the watchdog resumes it from the ledger.

Every run has a dollar ceiling, five dollars by default, more for bigger scoped work, and a fleet-wide governor caps the whole day's spend on top; when the day's budget is gone, new dispatches refuse and say so on the issue. When an agent runs out of budget it doesn't push whatever it has; it labels the issue needs-human and writes up where it got stuck. We made refusal, non-convergence, and budget exhaustion all end the same way, with a note for a person, after realizing early on that the worst possible failure mode was an agent quietly papering over a problem it couldn't solve.

Scope works the same way. Each agent declares the paths it may touch, a hook blocks writes anywhere else, and the whole branch gets re-checked before push. We launched with an extra global deny-list of protected files on top of that, and quietly retired it a few months in - the review gate had made it redundant, and it kept blocking work we actually wanted the agents to do.

And one thing nobody warns you about: GitHub's native scheduled workflows fire unreliably. We lost real time to "why didn't the detector run last night" before moving every cron to EventBridge Scheduler, which triggers a small Lambda, which dispatches the workflow. A meaningful share of "autonomous agents" turns out to be plumbing like that.

Real numbers

The fleet's first full month (June 24 - July 22):

1. 67 bugs surfaced, and only about 1 in 11 was noise.
2. 30 agent-written fixes merged, every one through the review gate by a human. Nothing shipped unreviewed.
3. 86% of the agent's fix PRs made it through review.
4. ~12 minutes from detection to fix-PR for production errors; detection to merged the same day, every time.

The trend line matters more than the totals. In the first ten days, 1 in 6 findings was noise and about 4 in 5 PRs survived review; it took real tuning of the dedup and reject thresholds to get to today's numbers. The last full week before this post was the payoff: 18 findings filed, 18 resolved, and for the first time since launch, a fully drained board with zero customer-reported incidents.

Where the humans sit

Every one of those merged PRs ended with a person clicking merge; branch protection enforces it. Anything touching security-sensitive paths gets extra treatment: the strongest model, a security label, and review by the security owners. When an issue turns out to be a design decision rather than a mechanical fix, the agents stop and say so instead of guessing. And if any part of the fleet misbehaves, each lane has a kill switch - one repository variable, no deploy needed.

What changed for us isn't that humans left the loop. It's the seat we occupy in it. For this class of work we've moved from writing the fixes to approving them, and nobody here misses the old seat.

Steal this

If you build something like this, start with the detector and the ledger, not the fixer. Deduplicated detection that files well-formed, machine-readable issues is maybe a fifth of the work, and it's where half the value showed up for us. Run it for a month with humans still writing the fixes; you'll learn your noise rate and build trust in the pipeline before you hand anyone the pen. When you do hand it over, give every agent the three things ours get: a narrow declared scope, a dollar budget, and a way to say "I need a human."

The agent that surprised us most isn't in this post - it files bugs for code that has never failed in production, and it gets its own post in this track two Wednesdays from now. Next week the other half of this blog begins: the product side, starting with why your CRM doesn't know who's actually in your deals.

---

This post is part of How a Tiny Team Ships Like a Big One, a series on how six builders run a production AI company. Building at Aithon - if this is how you want to work, talk to us.