How a Tiny Team Ships Like a Big One · The Product Track
Which model for which job
One production job took 49 minutes because everything ran on the biggest model. On routing: where frontier models earn their price, where small ones win, and cascades for everything between.
A batch job in our data pipeline was taking 49 to 56 minutes a run, and had been for a while. When someone finally profiled it, the diagnosis took minutes: the job was calling a large, expensive model to do work a small one handles fine - light extraction from short documents, thousands of times. We swapped the model. The job now finishes in a fraction of the time, at a fraction of the cost, with no measurable quality change.
The embarrassing part isn't the mistake; it's how ordinary it is. Model choice tends to happen once, at feature-birth, under deadline - someone picks the model that made the demo work, and that choice fossilizes into infrastructure. Nobody re-litigates it, because nothing is broken. It's just slow and expensive forever.
The three routing rules we actually use
After enough of these episodes, our routing settled into three rules.
1. Frontier models where quality is measurably different - and prose is the clearest case. Our narration agents - the ones that write account summaries and deal briefs a human will actually read - run on Claude, full stop. We've tested the alternatives, and the prose quality difference is measurable and customer-visible. The same logic runs in reverse: for extraction and classification, big models produce the same JSON as small ones, slower and pricier. The SRE fleet's severity routing is this rule wearing ops clothes: Opus for P0/P1 and security-sensitive fixes, Sonnet for the routine ones at roughly 60% of the price.
2. Small-first cascades for validation-heavy work. Where output is machine-checkable, we run a cascade: the small model answers, a confidence check evaluates, and only low-confidence items escalate a tier. Per-tier circuit breakers stop a misbehaving tier from hammering retries. On workloads with heavy validation, the cascade cuts cost 40-60% - most items never need the expensive opinion.
3. Routing lives in config, and changing it requires evidence. Our worst cost surprises - the $2k-a-month spike that ran for weeks - happened when model choices were scattered across code, invisible. They live in one routing file now: three tiers, each with an ordered fallback chain, prices and capabilities attached, and callers ask for a tier rather than naming a model. Changing what a tier points at is a one-line diff, and touching that file triggers the eval suites in CI, so a routing change arrives with evidence or it doesn't land (next week's post is about what counts as evidence).
Real numbers
- 49-56 minutes → minutes: the batch job fixed by matching model size to task, in an afternoon.
- ~11x between the top and bottom routing tiers on output tokens, and ~1.7x between the top two - the spread that makes routing worth engineering at all.
- 40-60% cost reduction from small-first cascades on validation-heavy workloads.
- One routing config, three tiers, ordered fallbacks: a model change is a reviewable one-line diff, not an archaeology project.
Where the humans sit
Humans own the routing table. An agent can burn its budget however its run requires, but which model a task class gets is a decision that arrives as a PR, with eval results, and gets reviewed like any other change to production behavior. When quality and cost genuinely trade off - narration is the recurring example - a person makes the call with the numbers in front of them, once, and the config remembers it.
Steal this
Profile your LLM spend by task, not by feature - the query is annoying and the result is usually a list of jobs quietly running on a model two sizes too big. Then flip your default: start every new task on the smallest model with a verifier behind it, and escalate on evidence, instead of starting big and never looking back. The 49-minute job had been shippable in minutes the whole time. Nobody had asked.
Next week the build track rides the deploy rails. Then back here for the evidence part: how we grade our own AI's output, and how we learned to distrust the grader.
This post is part of the product track of How a Tiny Team Ships Like a Big One, a series on how six builders run a production AI company. Building at Aithon - if this is how you want to work, talk to us.