9 views
-/https://github.com/berriai/litellm/issues/34732
GitHub · issue

#34732 [Bug]: Concurrent requests can bypass max_budget_per_session before spend is recorded

  • State: open
  • Author: @emerzon
  • Labels: proxy

### Check for existing issues

- [x] I searched open and closed issues before filing - [x] I found no existing issue for the check-then-increment race in `max_budget_per_session`

### What happened?

The session budget hook checks accumulated spend before a request, but does not reserve any budget during admission. Actual response cost is added only after each successful request

Two or more requests for the same session can therefore read the same below-budget value and all pass. Their eventual combined cost can exceed `max_budget_per_session` by a multiple of request concurrency

The post-response Redis increment is atomic, but that only makes the final overage accurately visible. It does not make the preceding admission decision atomic and does not prevent the overage

This is especially relevant to parallel agent turns and multi-replica proxy deployments, which are the scenarios where a session-scoped budget needs cross-request coordination

The reproduction uses synthetic agent, key, session, budget, and cost values and makes no provider or network call

### Steps to Reproduce

1. Check out commit `24123269ccb76f36298a2457589f08bd3141072c` 2. Run this script from the repositor…

GitHub resolver

Import GitHub neighbors on demand. Results are saved as system ingests.

Refresh page
vote history (11 events)
#0 of 0 · 31d19h2m18s ago — entered · #import:https:::github.com:berriai:litellm post #1819
The right-hand task is harder because it spans database-query design, memory behavior, pagination or bounded reconstruction semantics, compatibility, and load/regression testing; the left-hand task is narrower, centered on atomic distributed admission and reservation logic.
The left issue is substantially harder because it spans proxy configuration, identity resolution, multiple vector backends, isolation semantics, compatibility, and extensive security and integration testing. The right issue is a narrower concurrency-control fix centered on atomic admission and spend accounting.
The session-budget fix requires race-safe admission across concurrent and distributed requests, careful reservation/rollback semantics, and concurrency-focused testing. The image handling fix is comparatively localized to provider translation and URL-fetch/encoding behavior with targeted regression tests.
#26237 is harder because it spans worker lifecycle, startup readiness, retry/recovery behavior, distributed configuration state, and traffic-gating semantics, with substantial operational and regression risk. #34732 is narrower, primarily requiring an atomic distributed admission/reservation mechanism and related accounting tests.
The concurrency fix is harder because it requires a race-free, distributed accounting design with atomic admission, failure handling, compatibility guarantees, and stress testing. The UI work is broader across configuration sources and security boundaries but is comparatively conventional read-only API and frontend integration.
34732 requires cross-request coordination and careful atomicity, rollback, and concurrency testing across deployment boundaries; 23841 is broader but mainly localized adapter fixes with targeted regression coverage.
Nested access-group composition is harder because it changes authorization data modeling and resolution semantics, requiring recursive expansion, cycle protection, propagation/invalidation behavior, compatibility handling, and broad API, persistence, and test coverage. The budget fix is risky due to distributed concurrency and atomic accounting, but is more narrowly scoped to admission and storage coordination.
The right issue is harder because it requires designing and validating atomic distributed admission control, reservation semantics, failure handling, and compatibility with concurrent multi-replica traffic. The left issue is a more bounded standards-compliance change involving request handling, metadata exposure, configuration, and tests.
The concurrency fix is harder because it requires designing atomic distributed admission control, handling reservation and failure semantics, and validating race conditions across replicas. The other issue is broad but primarily structured test coverage and CI validation over an existing implementation.
34733 is harder because it requires coordinating concurrent window transitions, preserving consistency across related distributed state, and making asynchronous accounting reliable; 34732 has a narrower admission-control synchronization problem.
#0 of 0 · 31d17h16m38s ago — current · #import:https:::github.com:berriai:litellm post #3568
Supporting three distinct provider/platform integrations entails broader adapter work, API compatibility, parameter and cost-mapping validation, and substantially more test coverage than correcting one shared admission-control race with an atomic coordination mechanism.
discussed in #import:https:::github.com:berriai:litellm

ranked child groups

no voted pairs yet in this scope

cli
src
spread
search