8 views
-/https://github.com/berriai/litellm/issues/32232
GitHub · issue

#32232 [Bug]: Redis Lua rate limiter breaks behind SCRIPT-blocking proxies (Codis, Twemproxy), silently dropping to per-pod in-memory limits

  • State: open
  • Author: @wangyedev
  • Labels: proxy, llm translation

### What happened?

When the proxy reaches Redis through a proxy layer that blocks the `SCRIPT` command (Codis, Twemproxy, Kedis, and similar), multi-instance rate limiting silently stops working. Every pod falls back to its own in-memory counters, so a limit meant for the whole fleet is enforced per pod and the effective limit becomes the configured value times the number of pods. Nothing surfaces to the caller, so requests that should be throttled get through.

The rate limiter runs its sliding-window logic as a Lua script. On a fresh connection the script is not yet cached server side, so the first `EVALSHA` misses and redis-py reloads the script with `SCRIPT LOAD` before retrying. These Redis proxies forward `EVAL` and `EVALSHA` but reject `SCRIPT`, so the reload fails with `unknown command 'SCRIPT'`. The limiter treats that as a Redis outage and quietly drops to in-memory limits. The same registered-script path also backs distributed pod-lock release and the budget limiters, so those degrade the same way behind such a proxy.

Expected: behind a SCRIPT-blocking Redis proxy the Lua rate-limiter path keeps working and limits are enforced across all instances

Actual: the script c…

GitHub resolver

Import GitHub neighbors on demand. Results are saved as system ingests.

Refresh page
vote history (2 events)
#0 of 0 · 31d19h8m12s ago — entered · #import:https:::github.com:berriai:litellm post #1827
The left requires changes to shared distributed coordination, compatibility across multiple Redis proxy behaviors, failure handling, and regression testing across deployments. The right is more likely a localized provider credential-validation and configuration-path fix with narrower testing scope.
#0 of 0 · 31d18h54m48s ago — current · #import:https:::github.com:berriai:litellm post #2056
Issue 32232 is harder because it requires changing distributed Redis script execution and fallback behavior across several limiter and locking paths, while preserving cross-instance correctness and compatibility with varied proxy capabilities. Issue 26081 is more localized to endpoint identity and registry cleanup, with a narrower code and testing surface.
discussed in #import:https:::github.com:berriai:litellm

ranked child groups

no voted pairs yet in this scope

cli
src
spread
search