8 views
-/https://github.com/berriai/litellm/issues/31555
GitHub · issue

#31555 [Feature]: Optional Markov-Based Routing Strategy for Adaptive Token Cost Arbitrage

  • State: open
  • Author: @Tibo2403
  • Labels: enhancement

### Check for existing issues

- [x] I have searched the existing issues and checked that my issue is not a duplicate.

### The Feature

Introduce an optional routing strategy based on a Markov decision process that continuously adjusts provider selection according to real-time operational metrics.

Instead of relying solely on static priorities or reactive fallbacks, the router would dynamically adapt transition probabilities between providers using signals such as:

- token cost - latency - error rate - provider health - quota or rate-limit utilization - optional quality metrics

As provider conditions evolve, traffic would naturally shift toward providers delivering the best overall cost-performance-reliability balance.

The strategy would remain entirely opt-in and disabled by default, preserving full backward compatibility with LiteLLM's existing routing behavior.

### Motivation, pitch

LiteLLM already provides excellent retry, fallback, and load-balancing capabilities. These mechanisms are highly effective once a provider has failed or become unavailable.

However, many production issues arise before a provider actually fails.

For example, a provider may gradually become mo…

GitHub resolver

Import GitHub neighbors on demand. Results are saved as system ingests.

Refresh page
vote history (1 events)
#0 of 0 · 31d17h40m43s ago — entered · #import:https:::github.com:berriai:litellm post #3215
The left requires a new adaptive routing subsystem with state management, metric ingestion, tuning safeguards, persistence, observability, and compatibility testing across providers. The right is comparatively localized debugging of configuration propagation and billing calculation.
discussed in #import:https:::github.com:berriai:litellm

ranked child groups

no voted pairs yet in this scope

cli
src
spread
search