IBM Research has demonstrated that treating model selection as a systems optimization problem can cut agent execution costs by 21% while reducing response latency by 9%. Evaluating 417 tasks on the AppWorld benchmark, researchers showed that traditional pricing models fail to predict real-world costs because they ignore context caching and hardware state. The findings challenge standard difficulty-based routing, offering developers a lightweight framework to balance cost, speed, and accuracy.
Why Token Pricing Sheets Fail in Production
Standard enterprise routers dispatch simple prompts to cheaper models and reserve complex tasks for larger systems. However, IBM Research found that evaluating models based purely on per-token sticker prices leads to incorrect cost assumptions.
In a benchmark of 417 tasks on the AppWorld Test Challenge using a CodeAct agent, Claude Sonnet 4.6 cost $79 total ($0.19 per task), while GPT-4.1 cost $155 total ($0.37 per task). On paper, GPT-4.1 had lower input and output token rates and Sonnet required roughly three times as many reasoning steps to finish identical tasks. Sonnet achieved lower total cost because agentic workloads reuse large context blocks, allowing Sonnet's lower cache-read pricing to drastically reduce effective input costs.
Optimization Over Classification
To address these hidden realities, IBM Research abandoned standard classification heuristics in favor of a multi-objective optimization algorithm. The new approach balances cost, response latency, and task accuracy simultaneously without creating system bottlenecks.
Key results from IBM's router testing on the AppWorld benchmark include:
- Configuration 1 Savings: Delivered an 84% accuracy rate at $93 total cost and 83 seconds per task, achieving a 21% cost reduction and 9% latency reduction compared to running Opus alone with only a 4% accuracy drop.
- Superiority to Difficulty Routers: Standard difficulty-based routers hit a similar accuracy threshold but incurred significantly higher operational costs across the benchmark.
- Low Computational Overhead: The optimization algorithm requires only 6 ms of execution time and 2 kB of memory per task, ensuring the router itself does not add noticeable latency.
Infrastructure and Governance Trade-offs
Despite these gains, the research emphasizes that router performance remains tied to underlying infrastructure variables. Server load, hardware endpoint types, and whether a context cache is warm can easily override theoretical model speeds. Additionally, routing decision frequency introduces trade-offs: routing once per task minimizes overhead, while step-by-step routing adds latency despite offering greater operational flexibility.