CABS Router Uses Surrogate Scores to Reduce LLM Exploration
A new arXiv paper proposes correlation-aware contextual bandits for routing each prompt to an LLM under an accuracy-cost objective. The system uses predicted surrogate rewards from related models to learn from more than the selected model's observed result. Its decoupled variant, CABS-D, keeps the ordinary bandit and surrogate-aware learner separate, then adapts between them when proxies become unreliable. Across three offline routing benchmarks, the authors report stronger utility than standard implementable online bandits. In one low-cost-sensitivity setting, linear CABS-D scored 0.5328 versus 0.4876 for LinUCB. The result is promising evidence for faster simulated learning, not a deployment guarantee: it depends on historical benchmark labels, fixed model pools, learned correlations, and offline reward predictors that may drift in production.
A new arXiv paper proposes correlation-aware contextual bandits for routing each prompt to an LLM under an accuracy-cost objective. The central idea is to learn from the selected model's real reward while also using predicted surrogate rewards for models that appear behaviorally related on that query.
The researchers introduce two designs. CABS-C mixes observed and surrogate rewards into one learner. CABS-D keeps a standard bandit learner and a surrogate-aware learner separate, then uses an adaptive master to combine their policies. This separation is intended to retain the surrogate's early learning advantage without letting a persistently wrong proxy dominate every update.
How the routing loop works
| Stage | System action | Main operational risk |
|---|---|---|
| Encode | Represent the incoming query | New traffic may differ from training data |
| Relate | Estimate which model arms behave similarly | Correlations can change after model updates |
| Predict | Generate proxy rewards for related arms | Proxy bias can spread false confidence |
| Route | Select one model under a cost-aware utility | The utility may omit latency or business risk |
| Learn | Update from observed and surrogate feedback | Delayed or noisy labels can distort adaptation |
Across three offline routing benchmarks, the authors report stronger utility than standard implementable online bandits. In one low-cost-sensitivity setting, linear CABS-D scored 0.5328 versus 0.4876 for LinUCB. The full table shows smaller gains in other datasets and cost regimes, while hypothetical full-feedback methods generally remain stronger. That makes the result a sample-efficiency improvement under restricted feedback, not evidence that routing has reached an oracle.
What the benchmark does and does not test
The evaluation replays existing datasets with responses and labels already available across a fixed pool of models. RouterBench uses an LLM judge for binary correctness, SPROUT uses another judge for numeric scores, and the Open LLM Leaderboard data uses exact-match accuracy. Those labels make controlled simulation possible but do not reproduce live user preferences, safety failures, tool use, latency spikes, provider outages, or model-version changes.
Two of the benchmark pipelines use random data splits. A production router instead faces temporal drift: providers update models, prices change, query mixtures shift, and yesterday's correlation between two models can disappear. The paper acknowledges that the offline reward predictor generalises only under distribution overlap or structural invariance.
LDS analysis: shadow-test every routing update
A deployable version should run in shadow mode before controlling traffic. Log the chosen model, alternatives considered, predicted utilities, real cost and latency, task outcome, safety result, and later user correction. Compare the policy against a fixed baseline within stable traffic slices, then recalculate after every provider or model-version change.
Surrogate rewards should also have a circuit breaker. If prediction error rises for a domain, language, model, or cost band, the router should reduce proxy weight and fall back to the observed-reward policy. That is the practical value of the decoupled design, but it still requires monitoring thresholds and trustworthy delayed feedback.
The paper offers a useful architecture for learning under expensive partial feedback. Its next proof should be a reproducible live or temporally ordered evaluation with changing models, current prices, latency, safety constraints, and user-grounded outcomes.
Key Points
- 1CABS-D separates observed-reward and surrogate-aware learners, allowing the router to reduce reliance on proxies when they become unreliable.
- 2Author-reported benchmark gains show better simulated utility under partial feedback, but hypothetical full-feedback routers generally remain stronger.
- 3Production teams should shadow-test routing changes, monitor proxy error by traffic slice, and fall back when model correlations drift.
Scoring Rationale
The method addresses a practical LLM serving problem with theory and broad offline experiments, while benchmark replay and distribution-shift risks limit deployment confidence.
Sources
Primary source and supporting public references used for this report.
Practice interview problems based on real data
1,625 SQL & Python problems across 15 industry datasets — the exact type of data you work with.
Try 250 free problems