6 views
-/https://github.com/berriai/litellm/issues/35567
GitHub · issue

#35567 [Bug]: Global proxy spend cache loses concurrent increments

  • State: open
  • Author: @emerzon
  • Labels: proxy

### Check for existing issues

- [x] I searched open and closed issues before filing - [x] [#34732](https://github.com/BerriAI/litellm/issues/34732) covers the separate `max_budget_per_session` admission race

### What happened?

The cache used to enforce `litellm.max_budget` is updated with a non-atomic read, add, and set sequence

After each successful request, `update_cache` reads `GLOBAL_PROXY_SPEND_CACHE_KEY`, computes `global_proxy_spend + response_cost`, and schedules an asynchronous `async_set_cache_pipeline`. Two concurrent callbacks can both read the same starting value and both write their own result, permanently losing one increment from the cached global spend

The auth path reads this cached scalar through the event-driven coordinator and passes it to `_global_proxy_budget_check`. Under sustained concurrency, a lost-update value can be kept alive by later writes that refresh the cache TTL, so this is more than the ordinary delay before a completed request becomes visible

There is also no pre-call reservation for the global proxy budget. Even if the post-call update were atomic, concurrent requests can all be admitted against the same below-budget value. The existing …

GitHub resolver

Import GitHub neighbors on demand. Results are saved as system ingests.

Refresh page
vote history (3 events)
#0 of 0 · 31d18h56m40s ago — entered · #import:https:::github.com:berriai:litellm post #2019
The left issue requires concurrency-safe state management, atomicity or reservation semantics, race-condition testing, and careful compatibility with asynchronous cache and budget enforcement paths. The right issue is primarily a versioned SDK/OTel integration migration with configuration and test updates, making it substantially narrower in scope and risk.
The right issue is harder because it requires concurrency-safe distributed state updates, admission-control semantics, backend compatibility, and race-focused testing. The left issue is narrower: correcting asynchronous stream error classification and ensuring the appropriate logging and metrics callbacks execute.
#0 of 0 · 31d18h47m35s ago — current · #import:https:::github.com:berriai:litellm post #2172
The harder task requires distributed concurrency control, race-resistant accounting, backend compatibility, and stress testing. The easier task is a localized cache-lifecycle fix with focused regression coverage.
discussed in #import:https:::github.com:berriai:litellm

ranked child groups

no voted pairs yet in this scope

cli
src
spread
search