8 views
-/https://github.com/berriai/litellm/issues/31830
GitHub · issue

#31830 [Feature]: Optional priority queue for proxy requests

  • State: open
  • Author: @limitjupyter-ops
  • Labels: enhancement, proxy

### Check for existing issues

- [x] I have searched the existing issues and checked that my issue is not a duplicate.

### The Feature

I would like LiteLLM Proxy to support an optional asynchronous priority queue mode.

The goal is to allow requests to be queued and scheduled based on business priority when upstream quota or capacity is scarce.

Suggested behavior:

1. Requests can be enqueued instead of immediately forwarded. 2. Queue items include model, key, team, user, priority, metadata, and enqueue time. 3. Worker processes claim queued requests and forward them to LiteLLM. 4. Queue ordering can consider request priority, team priority, and enqueue time. 5. Batch or low-priority workloads can be delayed when resources are constrained. 6. Admin UI/API can show queue depth, status, and failed items.

This should be optional and disabled by default so current synchronous proxy behavior remains unchanged.

### Motivation, pitch

LiteLLM is commonly used as a real-time gateway, but enterprise deployments sometimes need to handle scarce model quota or limited upstream capacity.

In those cases, not all traffic has equal priority:

- interactive applications should be served befor…

GitHub resolver

Import GitHub neighbors on demand. Results are saved as system ingests.

Refresh page
vote history (4 events)
#0 of 0 · 31d18h22m55s ago — entered · #import:https:::github.com:berriai:litellm post #2486
31830 requires a cross-cutting proxy architecture change involving asynchronous execution, scheduling, persistence, observability, and compatibility risks, whereas 29274 is primarily a contained SDK model-registry consistency fix.
The priority-queue feature is substantially harder because it introduces cross-cutting scheduling, worker coordination, persistence, concurrency, failure handling, and operational interfaces while preserving synchronous behavior. The security issue is narrower in implementation scope, though it carries high correctness and regression risk around authorization edge cases.
The right issue is substantially harder: it requires an optional, reliable scheduling subsystem spanning queue persistence, worker coordination, priority semantics, retries, observability, and proxy compatibility. The left issue is a narrowly scoped identifier-mapping and logging consistency fix.
#0 of 0 · 31d17h54m21s ago — current · #import:https:::github.com:berriai:litellm post #2974
A full scheduling subsystem requires durable state, worker coordination, fairness and priority semantics, failure recovery, observability, and backward-compatible request handling. The other change is comparatively bounded to an opt-in response/streaming contract and upstream-status translation, despite protocol design and provider compatibility risks.
discussed in #import:https:::github.com:berriai:litellm

ranked child groups

no voted pairs yet in this scope

cli
src
spread
search