AI Engineer World's Fair 2026
Preferences Over Benchmarks: Model Routing — Archana Kamath & Tyler Gillam, DigitalOcean
Loading slides…
Read the talk
Preferences Over Benchmarks: Model Routing
Archana Kamath and Tyler Gillam show how per-request routing can trade a small amount of measured quality for lower cost and latency, while adding failover and an evaluation loop that a single-model setup lacks.
From a talk by Archana Kamath and Tyler Gillam
At a glance
Ideas worth remembering
Choose models per request rather than per application: task, tools, cost, latency, risk, and user preference all affect what “right” means.
Separate task matching from model selection. After identifying the task, apply its eligible model pool, optimization policy, and failover order.
Do not confuse a convincing side-by-side demo with proof. Use representative evaluations to compare quality, tokens, latency, and cost, and preserve uncertainty in LLM-judged scores.
In the demonstrated coding session, routing reduced cost from $0.25 to $0.08 after the initial build and from $0.44 to $0.14 after tests and documentation, with subjectively similar output.
Routing is a foundation layer. Evaluations, caching, and personalization determine whether it improves a real workload over time.
Why one model is the wrong unit of choice
The familiar model-selection habit starts with a leaderboard: find the highest-ranked model and send every request to it. Archana Kamath argues that this optimizes the wrong unit. An application does not have one uniform workload; it receives individual requests with different difficulty, latency requirements, and consequences for failure.
Three pressures break the one-model habit. Cost: paying frontier-model rates for routine work wastes inference budget. Fit: classification, code generation, and security review do not demand the same capabilities. Risk: when one provider or model degrades, a product built around that single dependency has no failover path. Kamath compares the emerging discipline of model orchestration with FinOps because teams must actively govern where inference spending goes rather than treating every token as equivalent.
Her examples establish a rough capability ladder. Classification and labeling may fit a small open model. Code generation and bug fixing may work well on a mid-sized open-weight model. Accuracy-sensitive work such as code review or security analysis may justify a frontier model. Inline code completion adds another constraint: even an accurate answer can be unsuitable if routing and generation are too slow for an interactive editor.
A public benchmark cannot encode the complete decision because the model is only one component of the deployed system. The relevant variables include the request’s task, the system prompt and tools around the model, the acceptable cost, the required latency, and the end user’s preferences. Routing turns those application-level requirements into a per-request model-selection problem.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Turn preferences into routing policy
Tyler Gillam begins with the main objection to automatic routing: a black box can make a poor choice without giving builders a practical way to correct it. DigitalOcean’s proposed answer sends each request through the open-source Plano-based routing architecture. A purpose-built routing model interprets the request against configured tasks, while explicit preferences and rules constrain what it may choose.
Builders can describe tasks in natural language, set priorities such as cost, latency, or quality, name preferred models, and add hard decision-tree rules. Presets provide a starting point, but the important feature is editability: the routing policy can be changed and then tested against the application’s own evaluation set. Gillam describes the operating loop as route, evaluate, adjust, and feed the result back into the policy.
The software-engineering configuration makes the two-stage decision concrete. First, the router matches the request to a task such as bug fixing, code generation, or test writing. Then the task’s policy selects from its eligible model pool. A manual ranking can prefer one model and use another only as failover. A fastest policy can instead choose whichever eligible model has been fastest during roughly the previous 30 minutes.
This separation matters. Intent matching answers, “What kind of work is this?” Selection policy answers, “Given this task and its allowed models, what should we optimize now?” Keeping those decisions distinct lets a team change model availability, priority, or failover behavior without rewriting the calling application’s task logic.
From request to selected model
Read the diagram as text
- Incoming request. Includes the prompt and surrounding workflow context.
- Declared preferences and rules. Cost, latency, quality, preferred models, task descriptions, or hard rules.
- Matched task. For example: code generation, bug fixing, or test writing.
- Eligible model pool. One or more models configured for the matched task.
- Selection policy. Manual ranking, failover order, or fastest recent model.
- Selected model. Receives this request; another request may take a different route.
- Incoming request → Matched task: classify intent.
- Declared preferences and rules → Matched task: define task descriptions.
- Matched task → Eligible model pool: select eligible set.
- Declared preferences and rules → Selection policy: set optimization and rules.
- Eligible model pool → Selection policy: rank candidates.
- Selection policy → Selected model: choose or fail over.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A playground shows behavior; an evaluation tests it
The first side-by-side demonstration sends three related prompts through a fixed premium model and the software-engineering router. “Write a basic Fibonacci function” matches a code-snippets task and goes to a comparatively inexpensive model. “Optimize my function” matches performance optimization and takes a different route. “Write some unit tests” matches test writing and code verification, selecting yet another configured model.
The playground makes the mechanism visible: related turns in one coding conversation need not use the same model. It also displays cost and speed per request. But Gillam calls the output comparison a “vibe check,” which is the right limitation to preserve. A few plausible-looking answers can reveal routing behavior; they cannot establish dependable quality.
The accompanying evaluation compares the router with the fixed premium model over a dataset. The router scores 90% correctness and the single model scores 95%. Gillam characterizes the difference as close to the uncertainty of the LLM-based judge, while the router uses substantially fewer tokens and returns faster. That interpretation should remain conditional: the talk does not provide the dataset size, judge calibration, repeated-run variance, or task-level breakdown needed to establish statistical equivalence.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The session-level cost gap compounds across agent steps
The larger demonstration moves from isolated prompts to two coding-agent sessions. Both receive the same request: “Build me a spinning wheel app.” The left terminal uses a fixed premium-model configuration; the right sends requests to the software-engineering router. A lower observability panel reports token use, selected models, matched tasks, and accumulating cost as the agents work.
On the routed side, code-generation requests go to the model configured for that task, and the session uses two models over the course of the initial build. On the fixed side, requests largely continue to the premium model, although Gillam notes that the coding tool itself appears to make an occasional different-model choice. That caveat means the comparison is not a perfectly isolated experiment between a literally single-model agent and a router; it is a live comparison between the two demonstrated configurations.
Both builds finish at a similar time, and Gillam judges their visible output as comparable, while explicitly acknowledging the subjectivity of that assessment. After the first feature request, the routed session has spent $0.08 and the other session $0.25—about a threefold difference at that point. This is useful evidence of the session’s observed economics, not proof that every coding workload will preserve the same ratio or quality.
The next prompt asks both agents to write unit tests. The router recognizes test writing and code verification, then uses the model configured for that task. A final prompt asks for README documentation. After those additional steps, the routed session totals $0.14 and the comparison session totals $0.44. The absolute savings are small in one short demo, but agent workflows multiply requests; repeated per-step savings can therefore accumulate across sessions and users.
One coding session, two selection strategies
Read the diagram as text
- Fixed premium-model configuration. fixed — Most requests retain premium-model cost and latency; demonstrated final session cost: $0.44.
- Software-engineering router. routed — Matches each step to a task and its configured model; demonstrated final session cost: $0.14.
- Build spinning wheel app. Both configurations finish at a similar time with subjectively comparable output.
- Write unit tests. The router matches test writing and code verification.
- Write README documentation. The final task extends the measured session cost comparison.
- Fixed premium-model configuration → Build spinning wheel app: mostly one premium model.
- Software-engineering router → Build spinning wheel app: code-generation route.
- Software-engineering router → Write unit tests: test-writing route.
- Build spinning wheel app → Write unit tests: next agent step.
- Write unit tests → Write README documentation: next agent step.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Routing is the foundation, not the finish line
Kamath closes with the implementation claims that make routing practical in an application’s request path: a decision in under 200 milliseconds per request, a purpose-built mixture-of-experts routing model, no application-code changes beyond adopting the router configuration, and no additional router charge. The routing components are presented as open source through Plano, allowing teams to inspect or operate the mechanism rather than depending entirely on an opaque proprietary selector.
Routing alone only chooses where work goes. Kamath places three layers above it. Evaluations test whether those choices work for the application. Caching avoids paying repeatedly for an answer that can be reused. Personalization adapts routing toward what works for a particular team. Together they turn static configuration into a proposed improvement loop rather than a one-time model-picking exercise.
The strongest version of the talk’s conclusion is also the most bounded: benchmarks remain useful signals, but they describe only part of a deployed system. A router still needs representative evaluations, well-written task descriptions, sensible model pools, and explicit cost and latency goals. The live demo shows promising savings and comparable-looking output under its conditions; it does not remove the need to measure failures, judge uncertainty, or verify quality on the workload that will actually ship.
That leaves a practical replacement for leaderboard climbing: define what “right” means for each request, route within those constraints, evaluate the results, and revise the policy. There may be no single best model, but there can be a measured selection process that reflects the application’s real preferences.
The routing improvement loop
Read the diagram as text
- Declare preferences. Describe tasks, cost and latency priorities, model pools, and hard rules.
- Route requests. Choose a model separately for each matched task.
- Evaluate outcomes. Compare correctness, token use, latency, and application-specific quality.
- Adjust policy. Revise descriptions, rules, priorities, or eligible models.
- Learn workload preferences. Use repeated routing and evaluation to adapt to the team over time.
- Declare preferences → Route requests: configure.
- Route requests → Evaluate outcomes: measure results.
- Evaluate outcomes → Adjust policy: identify changes.
- Adjust policy → Learn workload preferences: accumulate what works.
- Learn workload preferences → Declare preferences: update preferences.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
Further reading
A first-party technical article expanding on Plano, intent matching, model ranking, selection policies, failover, routing models, and the proxy request path discussed in the presentation.
Related talks
- The State of Model Routing — NVIDIA, Cognition, OpenRouter
Extends the discussion to application quality, inference cost, local hardware, context compaction, KV-cache-aware routing, and collaborative agents.
- 20 days of compute vs 7 hours: rethinking what state-of-the-art means — Bertrand Charpentier, Pruna AI
Develops the same critique of single-metric leaderboards through a Pareto-front view of task quality, latency, cost, and efficiency.
- LLM Evals That Work IRL
Explains how application-specific evaluations and trace inspection can replace subjective output checks when comparing models and routing behavior.
Full transcript
Loading transcript…