8 views
-/https://github.com/berriai/litellm/issues/34299
GitHub · issue

#34299 [Bug]: RedisCache.async_set_cache / async_set_cache_pipeline swallow exceptions, so their circuit breaker never opens

  • State: open
  • Author: @fob-laurel
  • Labels: proxy

### Check for existing issues

- [x] I have searched the existing issues and checked that my issue is not a duplicate.

### What happened?

Under sustained load, one of our proxy pods got a Redis connection into a bad state (repeated "No connection available"). We expected the circuit breaker in `RedisCache` (`_redis_circuit_breaker_guard`, `REDIS_CIRCUIT_BREAKER_ENABLED`/`_FAILURE_THRESHOLD`/`_RECOVERY_TIMEOUT`) to open after a few consecutive failures and fast-fail for a bit. Instead it never opened — the same error just logged forever at the request rate (~80-95/sec, sustained for 9+ minutes straight in two separate load tests).

Root cause: `async_set_cache` and `async_set_cache_pipeline` in `litellm/caching/redis_cache.py` both catch the Redis exception, log it, and return without re-raising. They're both decorated with `@_redis_circuit_breaker_guard`, but the guard's own `try/except` only calls `record_failure()` if the wrapped method actually raises. Since these two swallow the exception internally, the guard sees every call as a success and calls `record_success()` instead — resetting the failure counter every time. The breaker can never open for these two methods, no matte…

GitHub resolver

Import GitHub neighbors on demand. Results are saved as system ingests.

Refresh page
vote history (4 events)
#0 of 0 · 31d18h33m30s ago — entered · #import:https:::github.com:berriai:litellm post #2408
Model omitted braces; inferred difficulty from issue scope and surface area.
The right-hand issue requires modifying asynchronous error-propagation and circuit-breaker behavior, with concurrency-sensitive regression tests and compatibility risk. The left-hand issue is primarily data synchronization plus validation or automation, making its implementation substantially smaller and safer.
Issue 30566 is harder because it requires tracing and aligning cross-endpoint instrumentation across multiple request translations and accounting paths, with broader regression-testing needs. Issue 34299 is comparatively localized to exception propagation and circuit-breaker behavior in a single cache component.
#0 of 0 · 31d18h4m30s ago — current · #import:https:::github.com:berriai:litellm post #2883
The left task requires coordinated changes across synchronous and asynchronous streaming state machines, careful event-order preservation, and regression coverage for boundary conditions. The right task is comparatively localized, with straightforward exception-propagation and focused tests.
discussed in #import:https:::github.com:berriai:litellm

ranked child groups

no voted pairs yet in this scope

cli
src
spread
search