Explained · Economics
Model routing
Not every question needs the biggest, most expensive model. A router reads each request and sends easy ones to a small, cheap model and hard ones to the heavyweight. Users rarely notice the difference — the bill notices, which is why most serious AI products quietly do this.
Where it breaksThe router has to decide how hard a question is before anyone has answered it, which is the hard part of the problem. Misroute a genuinely difficult request to the small model and the user gets a confidently wrong answer at a discount. Worse, routing quality drifts silently: nothing alerts you when last month's rules started sending the wrong third of your traffic to the cheap tier.
~17× cheaper input gap between GPT-4o at $2.50/M and GPT-4o mini at $0.15/M input tokensOpenAI API pricing page · 2024-07-18
Why a price gap exists at all
Models come in sizes, and size sets the bill. A bigger model has more weights to stream through a chip for every token it writes, so it occupies more expensive hardware for longer, and providers price accordingly — the gap between a flagship and its small sibling is routinely more than tenfold on input tokens. Meanwhile the questions arriving at a real product are nothing like uniform. Reformatting a date, classifying a support ticket, extracting a name from an email: a small model does these as well as a large one, and a very large share of production traffic is exactly this kind of work. Routing exists because paying flagship prices for classification is the single most common way an AI product loses its margin without anyone noticing.
How a router decides
The router sits in front of the models and answers one question: how hard is this? The crude version uses rules — request length, whether code or a tool call is involved, which product surface it came from — and is often good enough. The middle version uses a small classifier trained on your own past traffic, labelled by which model actually produced an acceptable answer. The most reliable version does not predict at all: it sends the request to the cheap model, checks the answer against something concrete — did the code run, is the JSON valid, did the model itself flag low confidence — and escalates to the expensive model only on failure. That last design costs a little more on the hard cases and is far more robust, because it replaces a guess about difficulty with evidence about the answer.
Where it breaks
The router is a prediction made before the work is done, and that is a genuinely hard prediction. The costly direction is downward: a hard question sent to the small model comes back fluent, plausible and wrong, and neither the user nor your logs are told that a cheaper model handled it. Second, the mix drifts. You tune the rules against one month of traffic, the product ships a feature that changes what people ask, and the same router now sends the wrong slice to the cheap tier while the dashboards still show a healthy saving. Third, routing adds a step to every request — a classifier call in front of the model is latency and cost on the ninety per cent of requests that never needed the judgement. Any router worth running is measured on quality per pound, never on savings alone.
The number that decides it
Work out two things: the share of your traffic a small model handles acceptably, measured on your own evals rather than a public leaderboard, and the price gap between your tiers. Multiply them and you have the ceiling on what routing can save. If eighty per cent of requests are simple and the gap is tenfold, the ceiling is large and the engineering pays for itself quickly. If your traffic is uniformly hard, routing buys you complexity and a new failure mode. Either way the check is the same one that governs every cost trick in this business: measure quality first, then price, and never let the second move without watching the first.