· AI Engineering · 14 min read
Model Routing and Cascades: Cheap Model First, Escalate on Confidence
Model routing cascades cut cost only when a cheap confidence signal separates hard requests from easy ones. Here is how to test that before you ship.

In my work with teams running LLM features at volume, the cascade is the first cost idea almost everyone reaches for. Send each request to a small, cheap model. If the answer looks good, return it. If not, escalate to a bigger model. It sounds like free money, and on a slide it is.
I’ve also seen it go wrong in a very consistent way. The team builds the plumbing in a week, the bill goes up, and nobody can say why. The plumbing was never the hard part. The hard part is the sentence in the middle of that description: “if the answer looks good.” Who decides that, how, and at what cost?
That question is the whole article. A cascade pays off only if the router can tell hard requests from easy ones cheaply. Without a trustworthy confidence signal, escalation costs more than it saves, because you pay for the cheap attempt and the expensive one, and you still ship some wrong answers.
- A cascade is a bet on one thing: that the check for "good enough" costs far less than the gap between your cheap and expensive model.
- Model the break-even before you build. Escalation rate, verifier cost and the cost of a missed escalation decide the outcome, not the model price list.
- Confidence signals vary widely in trustworthiness. Token probabilities, self-reported scores, validators and agreement between samples each fail in different ways, so measure yours on your own traffic.
- The right tiers (small self-hosted, open weights, frontier) depend on your difficulty mix and quality bar. For leadership, the metric is cost per accepted answer, not cost per call.
Why the Arithmetic Is Less Friendly Than It Looks
Let me start with the math, because it is short and it disciplines everything after it.
Say a cheap model costs 1 unit per call and a strong model costs 20. You always call the cheap one first. A fraction e of requests escalate. Add a verifier that costs v per request to decide whether to escalate. The expected cost per request is:
cost = 1 + v + e * 20Compare that with always calling the strong model at 20. If the verifier is free and only 30 percent escalate, you pay 1 + 6 = 7 units, a 65 percent saving. Now make the verifier a second model call at 2 units and let 60 percent escalate, which happens quickly when the confidence signal is cautious. You pay 1 + 2 + 12 = 15 units. The saving has shrunk to 25 percent, before you account for latency or for the requests where the cheap model was wrong and the router did not notice.
Those numbers are illustrative, not benchmarks. The shape is what matters. Three levers decide whether a cascade wins:
- The escalation rate
e. It is set by your workload’s difficulty mix and by how conservative the router is. - The verifier cost
v. If judging an answer costs nearly as much as producing one, you have rebuilt a more complicated version of calling the big model. - The miss cost. Some cheap-model answers are wrong and pass the check. In a support chatbot a miss is an annoyed customer. In a contract-review flow it is a liability. The quality bar sets how much you can afford to miss.
The research literature backs the general idea. The FrugalGPT paper (Chen, Zaharia and Zou) described LLM cascades and reported large cost reductions on its benchmarks, and the RouteLLM work from LMSYS trained routers on preference data to choose between a strong and a weak model. Both show the approach can work. Both also depend on a scorer or router that was trained and evaluated on data resembling the target workload. That dependency is the part people skip when they quote the headline savings.
What a Cascade Actually Looks Like
There are two related patterns, and mixing them up causes confusion.
Routing decides up front. A classifier looks at the request and picks a model. You pay for one model call plus a small classification step. The risk is that the classifier sees only the question, not how the model will fare on it.
Cascading decides after. You run the cheap model, look at what came back, and escalate if it fails a check. You pay for the cheap call always, and for the expensive call some of the time. The advantage is that you judge the actual output. The cost is paying twice on escalations, plus extra latency.
Many production systems combine them: a cheap pre-router sends obviously trivial or obviously hard requests straight to a tier, and a post-check handles the middle.

Notice the last two nodes. The cascade is only as good as the data that tunes it, so every decision is logged with the tier used, the confidence score and, later, whether the answer turned out to be right.
The Confidence Signal Is the Product
Everything hinges on the check. Here are the common options, and where each one lets you down.
Token log-probabilities. Cheap and available for free from many APIs and from any self-hosted server. They measure how sure the model was about each token, which is not the same as whether the answer is correct. Models can be fluently, confidently wrong. Calibration also drifts when you change models or prompts, and some hosted APIs do not expose log-probabilities at all.
Self-reported confidence. Asking the model “how sure are you, 1 to 10” costs almost nothing. It is also the signal I trust least. Models tend to cluster their answers at the high end, and the number often tracks how the question was phrased more than whether the answer is right. Treat it as a feature to test, not a gate.
Deterministic validators. Does the JSON parse against the schema? Does the SQL run? Do the extracted fields match a regex or a lookup table? Does the cited document ID exist? These are the best signals available because they are nearly free and they are objective. If your task has a checkable structure, build the cascade around the validators first. For extraction, classification into a fixed label set and tool-call generation, they often carry most of the load.
Agreement between samples. Run the cheap model two or three times and escalate if the answers disagree. It catches a real class of uncertainty, but it multiplies the cheap-tier cost and works poorly for long free-form text where “agreement” is hard to define.
A judge model. A second model scores the first answer. This is the most flexible check and the most expensive one. It is also exposed to the same blind spots as the model it is judging if you use a similar size. If the judge needs to be strong to be reliable, v climbs and the arithmetic above turns against you.
A trained router. A small classifier trained on your own labelled history, predicting whether the cheap model will succeed on a given request. This is the RouteLLM style of approach. It can be very cheap at inference time, but it needs labels, retraining as traffic shifts, and an owner.
No option here is the best in all situations. A schema-bound extraction pipeline should lean on validators. An open-ended advice assistant has no cheap ground truth, and its team may find that no check is trustworthy enough to justify a cascade at all. That is a legitimate finding, not a failure.
Worked Example: A Support Triage Pipeline
Here is a composite of the kind of system I have helped reason through, with made-up numbers chosen for clarity.
A company handles 100,000 inbound support messages a day. Each one needs a category, a priority and a draft reply. Relative cost per call: small model 1, mid-tier open-weights model 5, frontier model 20.
Start with a baseline of everything on the frontier model: 2,000,000 units a day.
Now design the cascade. The small model produces the category and priority as structured output. Validators check that the schema is valid, the category is in the allowed list, and the priority is consistent with a few keyword rules. Messages that fail a validator, or that mention billing disputes or legal threats (a pre-router rule), skip ahead.
Suppose a two-week evaluation on labelled history shows:
- 70 percent of messages are plain (password resets, order status, simple how-to). The small model gets these right and passes validators.
- 20 percent are ambiguous. The small model fails a validator or produces borderline output. They go to the mid-tier.
- 10 percent are hard or sensitive. They go straight to the frontier model.
Cost per message: 1 for everyone, plus 5 for the 20 percent, plus 20 for the 10 percent, plus a negligible validator cost. That is 1 + 1 + 2 = 4 units, against 20 for the baseline. An 80 percent saving, because the difficulty mix is lopsided and the checks are deterministic.
Now change the assumptions. Suppose the mix is 30 percent plain, 40 percent ambiguous and 30 percent hard, which is what a technical B2B product with complex tickets might see. The cost becomes 1 + 2 + 6 = 9 units, still a 55 percent saving, though less dramatic and with more latency on the hard tail. Suppose also that the quality bar is strict and the validators are weak, so you add a judge model on every request, and it has to be a strong one to be reliable, say 10 units. The cost is now 19 units against a baseline of 20. That is a 5 percent saving, thin enough that one badly tuned threshold wipes it out, and it does not cover the extra latency or the operating burden.
One simplification: these figures charge the small-model call to every message. In the first scenario the pre-routed hard requests skip it, which brings the cost from 4 to about 3.9 units. It does not change the picture.
Same architecture, three very different outcomes. The difference is the difficulty mix and the checkability of the output, which is why the same cascade is a clear win at one company and a net loss at another.
The Hidden Costs People Forget
Latency. An escalated request takes the cheap attempt, the check and the expensive attempt, in sequence. For interactive use, the slowest path defines the experience for your hardest users, who are often the ones who already care most. Consider running the cheap and expensive calls in parallel for latency-critical routes and cancelling the loser, at the price of some wasted spend.
Silent misses. Escalations show up in your logs. Misses do not. A confident wrong answer from the cheap tier looks like a success. The only defense is a continuous sample: route a small random slice of traffic to the strong tier regardless, compare the answers, and track the disagreement rate. This costs a few percent of your savings and is the cheapest insurance you can buy.
Threshold drift. Thresholds tuned on last month’s traffic degrade when the product ships a new feature, a model gets updated or users change how they write. Treat the confidence threshold as a configuration that needs a regression suite, the same way you would treat a prompt.
Operational surface. Each tier is another endpoint to monitor, another rate limit, another failure mode. A self-hosted small model brings capacity planning and GPU utilization into the picture, which only pays off above a certain steady request volume. A frontier API brings dependency on the vendor’s pricing and availability but removes the capacity work and often offers stronger safety tooling and support terms. Open-weights models in the middle give you control over versions and data residency, with the cost of running and evaluating them yourself. Which mix is right is a function of your volume, your compliance posture and your team’s appetite for running infrastructure.
Evaluation load. A cascade multiplies the evaluation surface. You need to know accuracy per tier, accuracy per route and accuracy of the whole system, on a test set that reflects real traffic. If your team cannot maintain that evaluation pipeline, a single well-chosen model is often the more honest design.
A Decision Framework Before You Build
Run these checks in order. Each one can end the project cheaply.
- Measure the difficulty mix. Take 500 to 1,000 real requests and label them, or have the strong model and a human sample label them. If fewer than a third are plausibly easy for a small model, the ceiling on savings is low.
- Find your cheapest trustworthy check. Can you verify outputs with a validator, a lookup or a test? If yes, you are in good shape. If the only available check is a judge model of similar cost to the strong tier, stop and reconsider.
- Compute the break-even. Plug your measured escalation rate, verifier cost and tier prices into the formula above. Include the shadow-sampling overhead. If the saving is under roughly 20 to 30 percent, the engineering and operating burden may eat it.
- Price the miss. Estimate what one wrong answer costs in your domain. Set the threshold so that the expected miss cost, not the compute cost, is your binding constraint.
- Prototype offline first. Replay logged traffic through the proposed cascade and compare to the strong model’s outputs. Do not touch production routing until the offline numbers hold.
- Ship with a kill switch and a shadow slice. A flag that sends everything to one tier, and a steady sample sent to the strong tier for comparison.
What This Means for Leaders
If you are funding or approving a routing project, ask for three numbers before the architecture review: the share of your traffic that is demonstrably easy, the cost of the check that proves it, and the measured quality gap between the cascade and the single strong model on your own data. A proposal that quotes only per-token price differences between models has not answered the question.
Also change the metric. Cost per call rewards a cascade that returns cheap, wrong answers quickly. Cost per accepted answer, meaning an answer that passed downstream review or was not corrected by a human, captures the real trade. Teams that track it find that some of their cheapest-looking routes are expensive once rework is counted, and some of their pricier routes are bargains.
Finally, be honest about when not to do this. At low request volume, the engineering time to build and maintain a cascade can exceed the compute it saves. Where errors are costly and hard to detect, the cheaper option is often to send everything to the strongest model you trust and spend your effort elsewhere. Where volume is high and outputs are checkable, a cascade across self-hosted, open-weights and frontier tiers can be a sizeable and durable saving. Both are reasonable answers to different situations.
Closing Thought
The model price list is the visible half of the problem. The invisible half is the router’s ability to know, cheaply and reliably, when the cheap model is not enough. Build the check first, measure it on your own traffic, and let the break-even arithmetic decide whether a cascade earns its place. If the signal is trustworthy, the savings follow. If it is not, no amount of clever routing will make up for it.



