Model Routing & Escalation

Definition

Model routing selects a model for a request using task characteristics and operating constraints. A cascade tries an initial approach and escalates when its result fails defined acceptance checks.

Routing is another decision component with its own failure modes. A model that works well for frequent short requests may fail on a minority language or unusual document format. Report those slices rather than assuming an attractive aggregate score describes every user’s experience.

What it covers

Eligibility

Restrict routes by data handling, tool support and permitted deployment.

Selection

Use evidence about task difficulty and model performance.

Escalation

Define observable failure or uncertainty conditions.

Accounting

Include router, fallback and repeated-context costs.

Apply eligibility rules before optimization. A request containing restricted information should not be sent to a cheaper but unapproved service. Model selection also cannot expand the task’s permissions: a more capable fallback receives only the authority the workflow actually permits.

In practice

Practical design patterns by Min.

• Evaluate routing on representative task slices, including rare consequential cases.
• Use independent acceptance checks where the outcome is observable.
• Retain route decisions and versions so regressions can be investigated.

For a simple two-stage cascade, expected variable cost includes the first attempt plus the escalation probability multiplied by the incremental fallback cost, with routing overhead added. Estimate these terms from the actual workload; a high escalation rate can erase the expected savings.

Research notes

Source-grounded details and editorial applications. Follow each link for the original document.

Working example

Why the controls matter

In the support example, classification accuracy is distinct from whether the agent may issue a refund. A route can improve text handling without being authorized for consequential execution. Keep policy checks outside the routing model and evaluate the completed workflow.

Common pitfalls

• Choosing the cheapest model before checking data and authority constraints.
• Escalating only when the model admits uncertainty.
• Ignoring the cost of an unsuccessful first attempt in a cascade.

Revisit routing when models, prompts or traffic change. A threshold selected on old data may behave differently after deployment. Preserve an approved fallback and compare both quality and costs before shifting a large share of production traffic.

Questions for operational review

Use these questions to decide what evidence your team needs. They are editorial prompts, not a compliance checklist.

• What important cases are routed incorrectly?
• Does escalation recover quality at an acceptable total cost?
• Are all routes permitted for the input data?