8 views
-/https://github.com/berriai/litellm/issues/34733
GitHub · issue

#34733 [Bug]: Concurrent router budget-window resets can overwrite spend

  • State: open
  • Author: @emerzon

### Check for existing issues

- [x] I searched open and closed issues before filing - [x] I found no existing issue for last-write-wins spend resets in `RouterBudgetLimiting`

### What happened?

When a provider, deployment, or tag budget window is considered expired, `_handle_new_budget_window` resets the spend key with a plain `SET` to the current response's cost

Concurrent responses crossing the same window boundary can all enter the reset path. Each writes only its own cost, so the final counter contains whichever write wins instead of the sum of all response costs

The start-time and spend keys are also written separately, so another worker can observe a mixed window state. In multi-replica deployments this directly undermines the shared budget counter at the point where traffic crosses a budget boundary

A separate reliability concern in the same accounting path is that queued Redis increments are dispatched with `asyncio.create_task` and the queue is cleared immediately, so the sync method does not wait for the remote write to complete

The reproduction uses synthetic cache keys, timestamps, and costs and makes no provider or network call

### Steps to Reproduce

1. Check …

GitHub resolver

Import GitHub neighbors on demand. Results are saved as system ingests.

Refresh page
vote history (25 events)
#0 of 0 · 31d18h3m55s ago — entered · #import:https:::github.com:berriai:litellm post #3126
34733 requires distributed concurrency-safe accounting, atomic state transitions, asynchronous persistence guarantees, and multi-worker regression coverage; 34522 is a narrower provider-specific translation and identifier-preservation fix.
Model omitted braces; inferred difficulty from issue scope and surface area.
Issue 21347 is harder because it requires designing and integrating broad schema-conformance validation across many providers, response shapes, streaming paths, and ongoing compatibility workflows. Issue 34733 is narrower, focused on making an accounting update path concurrency-safe and reliably persisted, though it still carries distributed-systems testing risk.
27883 is harder because it spans new proxy-facing configuration semantics and integrations for multiple external tool ecosystems, while 34733 is a focused concurrency and persistence fix within an existing budgeting path.
The right issue is harder because it spans persistent data modeling, distributed request enforcement, calendar/time-zone semantics, API contracts, UI work, migrations, and broad testing. The left issue is a focused concurrency and write-order correction within an existing accounting path, though it carries meaningful distributed-systems risk.
The left issue is harder because it spans framework integration, parameter propagation, model-specific request bridging, Azure endpoint construction, and regression coverage across multiple call paths. The right issue is a focused concurrency and Redis-atomicity correction within the router’s budget accounting, though it still carries distributed-systems risk.
The left task requires concurrency-safe distributed state handling, atomicity, race-condition validation, and asynchronous reliability work across shared accounting paths. The right task is a more bounded provider integration with model metadata, routing, and cost translation.
The concurrent budget-reset fix is harder because it involves distributed Redis atomicity, race-condition handling across replicas, consistent window state, asynchronous write reliability, and concurrency-focused testing. The Gemini video translation is a more localized multimodal parsing and provider-mapping enhancement.
#34733 is harder because it requires correcting concurrent distributed budget accounting, atomic window transitions, and asynchronous persistence reliability across replicas. #32201 is narrower, mainly requiring a pass-through response normalization and guardrail enforcement integration.
#26237 is harder because it spans worker lifecycle, readiness gating, persistent-state recovery, retry behavior, and HAProxy/Postgres failure modes. #34733 is comparatively localized to distributed counter atomicity and asynchronous write coordination.
34733 demands distributed-state concurrency correctness, atomicity, and reliable asynchronous persistence across replicas, creating substantially higher implementation and regression risk than the mostly localized translation compatibility work in 23841.
The left issue requires distributed atomicity across concurrent workers, consistent multi-key state transitions, and reliable async persistence semantics, creating substantial race-condition and regression risk. The right issue is security-critical but can be addressed more locally through credential-source gating and targeted validation, with narrower implementation scope.
#34733 requires distributed concurrency-safe accounting, atomic window transitions, and reliable async write coordination, with race-focused tests. #34326 is primarily scoped timeout configuration and application around existing PostgreSQL DDL operations.
33371 is harder because it requires cross-cutting API design, stable semantics, and integration across routing, provider metadata, and compatibility boundaries; 34733 is primarily a focused concurrency and persistence correction with targeted tests.
#29452 requires a broader authentication and credential-management architecture spanning interfaces, security boundaries, persistence, and provider integrations. #34733 is a narrower distributed-state correctness change, with concurrency risk but a more contained implementation and test surface.
#34733 is harder: it requires coordinating atomic distributed state transitions, concurrency-safe accounting, and asynchronous persistence across replicas, with substantial race-condition testing. #32562 is more localized to propagating terminal failures consistently through an existing streaming iterator and adding protocol-focused tests.
The recursive-schema failure requires redesigning bounded expansion semantics across multiple provider translation paths, with careful handling of cycles, fan-out, memory/CPU limits, compatibility, and regression testing. The budget issue is narrower: it mainly needs atomic/concurrency-safe counter-window updates and reliable async write coordination in one accounting path.
34733 is harder because it requires preserving distributed counter correctness across concurrent workers, coordinating related state updates, and safely changing asynchronous persistence semantics with strong regression coverage. 26700 is narrower in scope, primarily involving MCP lifecycle handling and compatibility with an upstream SDK behavior.
The right issue is harder because it involves diagnosing and redesigning transactional database behavior under concurrent load, with ORM/query semantics, lock ordering, and compatibility risks. The left issue is narrower: improving atomicity and synchronization in shared cache accounting.
34733 is harder because it requires concurrency-safe distributed accounting, atomic state transitions, asynchronous write guarantees, and race-focused testing across replicas; 28032 is primarily a contained authorization-model extension with recursive resolution and validation.
34733 is harder because it requires designing and validating atomic distributed accounting behavior across concurrent workers, coordinating related state updates, and correcting asynchronous persistence semantics without regressions. 33921 is broader than a single model mapping but is mainly an integration and parameter-support task across established provider abstractions.
The left task requires correcting distributed concurrency and durability semantics across shared state, with race-condition testing and failure handling. The right task is primarily a bounded proxy/authentication protocol integration with response and routing tests, so it has less implementation risk.
#34733 requires coordinated changes to concurrent distributed accounting, atomic state transitions, asynchronous write guarantees, and race-focused testing; #23348 is a set of mostly localized MCP validation and lookup fixes.
34733 is harder because it requires coordinating concurrent window transitions, preserving consistency across related distributed state, and making asynchronous accounting reliable; 34732 has a narrower admission-control synchronization problem.
#0 of 0 · 31d17h37m22s ago — current · #import:https:::github.com:berriai:litellm post #3552
The right issue is harder because it requires redesigning data retrieval and session reconstruction under large-scale workloads, while preserving ordering and behavioral semantics, plus validating database and memory performance. The left issue is narrower: making shared counter/window updates atomic and ensuring asynchronous writes are reliably awaited.
discussed in #import:https:::github.com:berriai:litellm

ranked child groups

no voted pairs yet in this scope

cli
src
spread
search