6 views
-/https://github.com/berriai/litellm/issues/34681
GitHub · issue

#34681 DualCache refresh can evict a fresh Redis batch-throttle entry at capacity

  • State: open
  • Author: @HarperZ9

### Summary

At current `litellm_internal_staging` (`f7078e2e086056d535bad79e15908419de0c47b7`), `LimitedSizeOrderedDict.__setitem__()` evicts the oldest entry whenever the map is at capacity, including when updating an existing key.

In the DualCache Redis batch-throttle path, refreshing an expired existing timestamp can therefore remove a different, still-fresh throttle entry. A subsequent lookup for that unrelated key then performs an avoidable Redis batch read instead of remaining throttled.

### Deterministic API-free reproduction

I exercised the pinned `DualCache.async_batch_get_cache()` path with mocked in-memory and Redis caches only. With time fixed at `100`, throttle expiry `10`, and these timestamps:

```python cache.last_redis_batch_access_time["unrelated"] = 99.9 # still fresh cache.last_redis_batch_access_time["expired"] = 0.0 # needs refresh

await cache.async_batch_get_cache(["expired"]) await cache.async_batch_get_cache(["unrelated"]) ```

Observed:

```text capacity=2 Redis calls: [["expired"], ["unrelated"]] throttle keys: ["expired", "unrelated"] ```

Control:

```text capacity=3 Redis calls: [["expired"]] throttle keys: ["unrelated", "expired"] ```

The c…

GitHub resolver

Import GitHub neighbors on demand. Results are saved as system ingests.

Refresh page
vote history (3 events)
#0 of 0 · 31d19h2m36s ago — entered · #import:https:::github.com:berriai:litellm post #1832
The right-hand issue is harder because it requires coordinating failure-safe state ownership, retry behavior, concurrency, and recovery across multiple production paths, with broader regression testing. The left-hand issue is comparatively localized to cache update and eviction semantics with a narrower behavioral fix.
The right issue is harder because it involves subtle eviction semantics in shared caching, Redis interaction, and regression/concurrency testing. The left issue is comparatively localized to request-field filtering or provider-specific translation.
#0 of 0 · 31d17h39m46s ago — current · #import:https:::github.com:berriai:litellm post #3223
#34734 requires a new cross-provider SDK abstraction, capability negotiation, credential/provider resolution integration, multiple adapter implementations, normalization semantics, and broad testing. #34681 is a localized cache-eviction correction with focused regression coverage, so it has substantially lower scope and risk.
discussed in #import:https:::github.com:berriai:litellm

ranked child groups

no voted pairs yet in this scope

cli
src
spread
search