8 views
-/https://github.com/berriai/litellm/issues/32614
GitHub · issue

#32614 [Bug]: Router budget sync can overwrite memory cache with stale Redis spend

  • State: open
  • Author: @emerzon
  • Labels: proxy, llm translation

### Check for existing issues

I found #20886, but it is closed as stale/not planned and cannot be reopened. This issue adds current-branch verification and additional affected code paths

### What happened?

`RouterBudgetLimiting._push_in_memory_increments_to_redis()` still schedules `redis_cache.async_increment_pipeline(...)` with `asyncio.create_task(...)` and immediately clears the queue. `_sync_in_memory_spend_with_redis()` awaits that method, but the await only waits for task scheduling. It then fetches Redis and writes those values into in-memory cache before the pipeline has necessarily completed

On current `litellm_internal_staging` at `60729f733e`, fetched on 2026-07-09, I reproduced memory being overwritten from `160` to stale Redis `100`. After the background pipeline finishes, Redis is `160`, but memory remains `100`

### Why this matters

`async_filter_deployments()` reads budget spend through `DualCache.async_batch_get_cache()`. `DualCache` returns in-memory values first and only checks Redis for misses, so the stale memory value is trusted by later provider, deployment, and tag budget checks

The same inherited flush method also affects `_PROXY_VirtualKeyModelMaxBu…

GitHub resolver

Import GitHub neighbors on demand. Results are saved as system ingests.

Refresh page
vote history (3 events)
#0 of 0 · 31d18h44m11s ago — entered · #import:https:::github.com:berriai:litellm post #2145
#32614 is harder because it involves subtle asynchronous consistency guarantees across shared cache, Redis synchronization, inherited call paths, and race-focused regression testing. #33374 is narrower endpoint-level latency work, mainly involving request planning and cancellation behavior.
32614 is harder because it requires correcting asynchronous coordination and cache-consistency behavior across shared proxy budget paths, with concurrency-sensitive regression testing; 25688 is comparatively localized to exposing and wiring a provider configuration choice in the dashboard.
#0 of 0 · 31d18h8m38s ago — current · #import:https:::github.com:berriai:litellm post #2716
#32614 is harder because it involves asynchronous ordering, distributed cache consistency, and downstream budget-enforcement behavior across multiple code paths, requiring careful concurrency fixes and regression tests. #29409 appears more localized to configuration handling and health-check filtering.
discussed in #import:https:::github.com:berriai:litellm

ranked child groups

no voted pairs yet in this scope

cli
src
spread
search