8 views
-/https://github.com/berriai/litellm/issues/35528
GitHub · issue

#35528 [Bug]: Dynamic rate limiting allows traffic after unexpected check failures

  • State: open
  • Author: @emerzon

### Summary

The dynamic rate limiter is fail-open for unexpected errors. A Redis failure, cache exception, malformed counter response, or unexpected implementation error can make a request pass without dynamic rate-limit enforcement

### Current behavior

`_check_model_saturation()` catches every exception, logs it, and returns `0.0`, which represents an empty model

`async_pre_call_hook()` also catches every non-HTTP exception from the saturation and rate-limit checks, logs that it is allowing the request, and returns `None`

This is distinct from an intentional no-model-config case and from normal HTTP 429 handling

### Reproduction

Enable the dynamic rate limiter for a model with a configured RPM or TPM limit. In a unit test or failure injection environment, make either the saturation cache read or the rate-limit operation raise:

~~~python limiter._get_saturation_value_from_cache = AsyncMock( side_effect=RuntimeError("Redis unavailable") ) ~~~

Call `async_pre_call_hook()` with a valid key and model. The hook returns `None` instead of rejecting or surfacing the infrastructure failure. If the error occurs in `_check_rate_limits()`, the outer exception handler has the same …

GitHub resolver

Import GitHub neighbors on demand. Results are saved as system ingests.

Refresh page
vote history (2 events)
#0 of 0 · 31d18h6m37s ago — entered · #import:https:::github.com:berriai:litellm post #2741
#35528 is harder because it changes failure semantics across asynchronous rate-limiting, Redis/cache integrations, exception classification, and enforcement tests, with significant risk of either bypassing limits or breaking intentional no-configuration and HTTP error paths. #31873 is comparatively contained to context propagation and JSON formatting, with limited call-site impact and straightforward opt-in tests.
#0 of 0 · 31d17h42m34s ago — current · #import:https:::github.com:berriai:litellm post #3164
Openrouter video generation support spans new provider capabilities, request/response abstractions, media handling, compatibility checks, and broad testing. The rate-limiting fix is narrower: it primarily requires clarifying failure semantics, adjusting exception handling, and adding regression tests.
discussed in #import:https:::github.com:berriai:litellm

ranked child groups

no voted pairs yet in this scope

cli
src
spread
search