9 views
-/https://github.com/berriai/litellm/issues/25297
GitHub · issue

#25297 [Feature]: Custom Routing in Proxy (Parity with SDK)

  • State: open
  • Author: @enniomaldonado-ai
  • Labels: enhancement, proxy

### Check for existing issues

- [x] I have searched the existing issues and checked that my issue is not a duplicate.

### The Feature

We need a supported way to supply our own routing strategy, the same kind of flexibility we get when using the SDK custom routing, without being limited to built-in algorithms.

### Motivation, pitch

We run LLM traffic through litemm and rely on its simple shuffle routing for the common case. For some workloads we need routing policy that is explicit, composite, and under our control, the same flexibility we already have when calling providers via an SDK, where we can implement custom selection, fallbacks, backoff, and experiments in code. Today, if that logic does not map cleanly onto litemm’s built-in strategies, we either duplicate routing outside the gateway (split clients, inconsistent behavior) or give up on policies we consider important for reliability and cost.

### What part of LiteLLM is this about?

Proxy

### LiteLLM is hiring a founding backend engineer, are you interested in joining us and shipping to all our users?

No

### Twitter / LinkedIn details

_No response_

GitHub resolver

Import GitHub neighbors on demand. Results are saved as system ingests.

Refresh page
vote history (11 events)
#0 of 0 · 31d19h10m51s ago — entered · #import:https:::github.com:berriai:litellm post #1702
Custom routing requires a configurable proxy extension point, request lifecycle integration, compatibility guarantees, and broad testing; the embedding fix is comparatively localized to provider model recognition and translation coverage.
#25297 requires architectural changes to proxy request routing, extensibility boundaries, configuration, execution semantics, and broad compatibility testing; #30641 is primarily an admin-UI and configuration-discovery integration with narrower behavioral scope.
28168 is harder because it spans many persisted resource types, requires dependency-aware serialization and secure secret handling, and needs a stable portable format with compatibility and validation concerns. 25297 is narrower, mainly requiring an extensible routing hook, configuration surface, and safe integration with existing request and fallback paths.
The custom-routing work is harder because it requires designing and integrating a stable extensibility contract across proxy configuration, request execution, routing state, fallbacks, concurrency, and backward compatibility, with substantial documentation and test coverage. The outage-related fix is narrower in scope: it primarily involves startup/readiness gating, retry/recovery behavior, and preserving valid state during dependency failures.
Extending the proxy with user-defined routing is broader and riskier: it requires a stable extension contract, lifecycle and concurrency handling, validation, observability, failure semantics, and compatibility across routing paths. The other change is comparatively localized to configuration resolution and deployment expansion.
The proxy-side change is harder because it requires a stable extensibility architecture within the request-routing lifecycle, careful compatibility with existing strategies, and robust behavior under concurrency, retries, fallbacks, and distributed deployments. The CLI work is broader across integrations but can largely be isolated behind provider-specific launch adapters and configuration flows.
The left task is harder because it spans model integration, request-path transformation, multi-surface configuration, observability, dependency/runtime concerns, and failure isolation. The right task is primarily an extensibility change around routing interfaces and proxy wiring, with less cross-cutting behavior.
The left requires a broader extension architecture across proxy request flow, strategy injection, compatibility, and operational behavior, creating more integration and regression risk. The right is a bounded quota feature with focused persistence, enforcement, API, and UI changes.
Historical spend correction has greater scope and risk: it requires safe, repeatable data mutation, pricing-version handling, policy semantics, auditability, and operational safeguards. Custom routing is architecturally significant but can largely be isolated behind a routing extension interface and proxy integration.
The self-service feature is harder because it spans authenticated authorization boundaries, usage-data aggregation, multiple API surfaces, and dashboard work while preserving existing administrative behavior. The routing feature is architecturally significant but can be concentrated in the proxy’s request-selection extension points and compatibility layer.
#0 of 0 · 31d17h24m50s ago — current · #import:https:::github.com:berriai:litellm post #3466
The left issue entails designing and integrating a new extensibility contract across proxy routing, configuration, lifecycle, and compatibility boundaries. The right issue is a difficult distributed-state debugging and correctness fix, but its scope is more targeted once the failure path is isolated.
discussed in #import:https:::github.com:berriai:litellm

ranked child groups

no voted pairs yet in this scope

cli
src
spread
search