8 views
-/https://github.com/berriai/litellm/issues/31823
GitHub · issue

#31823 [Feature]: Provider quota pools and package-based routing

  • State: open
  • Author: @limitjupyter-ops
  • Labels: enhancement, proxy

### Check for existing issues

- [x] I have searched the existing issues and checked that my issue is not a duplicate.

### The Feature

I would like LiteLLM to support provider quota pools, also called provider packages, as a first-class routing concept.

Today LiteLLM Router supports multiple deployments, fallback, load balancing, budgets, and rate limits. However, enterprise deployments often purchase quota from multiple vendors or accounts, and each quota package may have its own usage window and remaining capacity.

Desired provider package fields could include:

- `logical_model` - `provider` - `package_name` - `provider_model_name` - `provider_api_key_ref` - `provider_priority` - `routing_weight` - `max_5h_calls` - `max_daily_calls` - `max_weekly_calls` - `max_monthly_calls` - `used_5h_calls` - `used_daily_calls` - `used_weekly_calls` - `used_monthly_calls` - `cache_ttl_seconds` - `is_active` - `metadata.capabilities` - `metadata.max_context_tokens` - `metadata.stability_score`

Desired behavior:

1. Admins define quota pools/packages for each logical model. 2. Router selects a provider package based on remaining quota, priority, routing weight, and capabilities. 3. LiteLLM …

GitHub resolver

Import GitHub neighbors on demand. Results are saved as system ingests.

Refresh page
vote history (2 events)
#0 of 0 · 31d18h40m1s ago — entered · #import:https:::github.com:berriai:litellm post #2199
The left item is substantially harder because it requires cross-cutting routing architecture, new state and configuration models, quota accounting, selection algorithms, concurrency handling, compatibility work, and broad testing. The right is comparatively localized to request handling and regression coverage.
#0 of 0 · 31d17h58m4s ago — current · #import:https:::github.com:berriai:litellm post #2890
The right-hand request is a broad routing feature involving new quota models, selection algorithms, persistence, accounting, configuration, APIs, and compatibility testing. The left-hand request is a narrowly scoped streaming-control change with localized handler and regression-test work, though it carries some async/error-propagation risk.
discussed in #import:https:::github.com:berriai:litellm

ranked child groups

no voted pairs yet in this scope

cli
src
spread
search