6 views
-/https://github.com/berriai/litellm/issues/35533
GitHub · issue

#35533 [Bug]: Redis parallel request limiter falls back to per-pod limits on Redis errors

  • State: open
  • Author: @emerzon

### Summary

When the Redis-backed parallel request limiter cannot read or atomically acquire a slot, it falls back to an in-memory counter in the current process. In a multi-pod deployment this changes one configured global limit into an independent limit per pod

### Current behavior

`litellm/proxy/hooks/parallel_request_limiter_v3.py` uses Redis for the shared gauge and Lua acquisition. On exceptions it logs a warning and calls `_acquire_parallel_slots_in_memory()`

The in-memory fallback is process-local. With a configured limit of `N` and `R` proxy replicas, Redis failure can allow approximately `R * N` concurrent requests instead of `N`. A deployment with no Redis configured is a separate supported local-only mode; this issue concerns a deployment that configured Redis for cross-pod coordination and then loses Redis availability

### Reproduction

Configure a model or global parallel request limit and run two or more proxy replicas with Redis enabled

Inject Redis failures for both the gauge read and the atomic acquire:

~~~python redis_client.eval = AsyncMock(side_effect=RuntimeError("Redis unavailable")) redis_client.get = AsyncMock(side_effect=RuntimeError("Redis unavaila…

GitHub resolver

Import GitHub neighbors on demand. Results are saved as system ingests.

Refresh page
vote history (3 events)
#0 of 0 · 31d18h17m56s ago — entered · #import:https:::github.com:berriai:litellm post #2650
The left requires distributed coordination semantics, careful failure handling, and broader concurrency testing; the right is a narrower container configuration and packaging change.
Issue 35303 is harder because it spans exception taxonomy, request transformation boundaries, retry semantics, fallback-graph cycle detection, async control flow, and high-volume regression testing. Issue 35533 is narrower, primarily involving distributed limiter failure semantics and targeted coordination tests.
#0 of 0 · 31d17h54m55s ago — current · #import:https:::github.com:berriai:litellm post #3053
The harder task changes distributed concurrency and failure semantics, requiring careful invariant preservation, race analysis, and multi-process testing. The other is a narrower request-schema translation and validation fix.
discussed in #import:https:::github.com:berriai:litellm

ranked child groups

no voted pairs yet in this scope

cli
src
spread
search