Heterogeneous Model Routing: Cost as a Design Axis

We've had this conversation with more than one engineering lead: the LLM bill arrived, it was three times the forecast, and the instinctive fix was to negotiate a better per-token rate with the vendor. That's treating cost as a procurement line item. The actual lever was sitting in the application layer the whole time — most of the token spend was going to a frontier model answering questions a model a tenth its price would have answered identically. Nobody had built the thing that would have caught that: a router.

Complexity is a property of the request, not the product

The mistake we see most often is treating "which model" as a single decision made once, at the architecture review, and then baked into a client library import. In practice every request a system handles has its own complexity profile — a support-ticket classifier deciding "billing" versus "technical" is not the same task as a support-ticket response drafting an apology for a missed SLA with specific account context, even though both arrive through the same endpoint. Routing by task type first, then by complexity signal within the task, is where the savings actually live. A rough split that's held up across the deployments we've run: classification, extraction, and structured-output tasks route to small open-weight or budget-tier models by default; anything requiring multi-step reasoning, long-context synthesis, or free-text generation with brand-voice constraints escalates. That escalation should be a rule, not a preference — because the team that leaves it as a preference finds every request eventually routed to the frontier model out of an abundance of caution.

A concrete routing policy

The policy we implement most often has three signals, evaluated in order, before a request reaches a model: input length, task type (from a fixed enum, not inferred), and a complexity score — usually a small, cheap model or a logistic model over token features, not another LLM call — trained on the client's own historical escalation data. A request under 500 tokens tagged "extraction" goes to the cheapest capable model in the fleet. A request over 4,000 tokens or tagged "synthesis" goes straight to the top tier without touching the classifier, because the classifier's own inference cost stops being worth it once the request is already expensive. Everything in between gets scored, and anything under a threshold we tune per client — usually somewhere around 0.7 confidence that the cheap tier will produce an acceptable answer — routes down; anything above escalates. The threshold isn't set once. It moves as the cheap tier's quality on that client's actual traffic gets measured against the escalation tier's output, sampled and graded, on a rolling basis.

Latency is the axis nobody budgets for

Cost gets the attention because it shows up on an invoice. Latency is the axis that actually breaks user trust, and heterogeneous routing affects it in a direction people don't expect: routing to a smaller, geographically closer, less-loaded model is very often faster than routing to whichever frontier endpoint has the queue depth of every other customer's traffic stacked in front of yours. We've measured p95 latency drop by more than 40% on a support-deflection workload purely from routing simple intents to a smaller self-hosted model instead of a shared frontier API, independent of the cost savings. A routing policy that only optimizes for dollars per token leaves a latency win on the table that a user actually feels.

What breaks without a router, and where this connects to the rest of the stack

Systems without an explicit router don't fail loudly. They fail as a slow bleed: engineers default every new feature to whatever model handled the last feature, because that's the fastest way to ship, and the fleet ossifies around a single vendor and a single price point regardless of what each request actually needs. The fix isn't "add more model options" — it's making the routing decision an explicit, owned, measured piece of the architecture, with its own dashboard, its own regression tests (does the cheap tier still meet the accuracy bar this week, on this client's drifting traffic), and its own on-call rotation for when a vendor has an outage and the router needs to fail over.

None of this works without an agent-native layer that already treats models as interchangeable execution backends behind a stable interface — which is exactly what a planner-executor architecture over MCP gives you for free, since the planner never needs to know which model executed a given step, only that the step's contract was satisfied. Routing policy becomes a runtime configuration change, not a redeployment. That's the actual argument for building it this way from day one: not that it saves money immediately, though it usually does, but that it turns a pricing negotiation into an engineering lever you control.