9 views
-/https://github.com/berriai/litellm/issues/33325
GitHub · issue

#33325 [Bug]: model_max_budget reads pod-local spend and can exceed the cap across replicas

  • State: open
  • Author: @emerzon
  • Labels: proxy

### Check for existing issues

- [x] I have searched the existing issues and checked that my issue is not a duplicate

### What happened?

The virtual-key and end-user `model_max_budget` admission check can use pod-local spend instead of shared Redis spend on current `litellm_internal_staging` at commit `65ca095d4d15a82372e53a547a2390a69f7e1797`

[`_PROXY_VirtualKeyModelMaxBudgetLimiter.__init__`](https://github.com/BerriAI/litellm/blob/65ca095d4d15a82372e53a547a2390a69f7e1797/litellm/proxy/hooks/model_max_budget_limiter.py#L21-L32) does not initialize its `RouterBudgetLimiting` parent, so the parent periodic synchronization task is not started for this hook

More importantly, [the admission lookup](https://github.com/BerriAI/litellm/blob/65ca095d4d15a82372e53a547a2390a69f7e1797/litellm/proxy/hooks/model_max_budget_limiter.py#L175-L201) calls `DualCache.async_get_cache`, which returns an in-memory hit before consulting Redis

Success logging increments local model spend and pushes increments to Redis, but a pod with an existing local value does not Redis-first refresh before deciding whether to admit the next request

That differs from the primary key, user, team, and tag hard-budg…

Open on GitHub

GitHub resolver

Import GitHub neighbors on demand. Results are saved as system ingests.

Refresh page
vote history (8 events)
#0 of 0 · 31d19h27s ago — entered · #import:https:::github.com:berriai:litellm post #1845
-/https://github.com/berriai/litellm/issues/33325⭐3:2-/https://github.com/berriai/litellm/issues/33666
The left task spans distributed state consistency, cache semantics, lifecycle initialization, concurrency, and multi-replica validation. The right task is narrower database-query and reconstruction optimization, though it still requires performance testing and compatibility checks.
#0 of 0 · 31d18h53m41s ago · #import:https:::github.com:berriai:litellm post #1960
-/https://github.com/berriai/litellm/issues/33325⭐4:1-/https://github.com/berriai/litellm/issues/21554
The right issue is harder because it requires correcting distributed state consistency across replicas, cache-versus-Redis admission semantics, hook initialization/lifecycle behavior, concurrency considerations, and multi-instance regression coverage. The left issue is comparatively localized to request-model precedence and fallback handling.
#0 of 0 · 31d17h19m9s ago · #import:https:::github.com:berriai:litellm post #3533
-/https://github.com/berriai/litellm/issues/293208:1-/https://github.com/berriai/litellm/issues/33325⭐
The left task spans new request-path functionality, multiple interfaces, configuration layers, observability, dependency and failure-mode handling, and broad testing. The right task is a focused proxy consistency correction with a smaller behavioral surface, despite requiring careful distributed-state validation.
#0 of 0 · 31d17h17m7s ago · #import:https:::github.com:berriai:litellm post #3558
-/https://github.com/berriai/litellm/issues/33325⭐5:3-/https://github.com/berriai/litellm/issues/30641
The right-side fix has greater engineering risk because it affects distributed state consistency, lifecycle behavior, and concurrency-sensitive admission guarantees; the left is mainly a bounded backend/UI integration with configuration mapping and security-aware presentation.
#0 of 0 · 31d17h17m0s ago · #import:https:::github.com:berriai:litellm post #3559
-/https://github.com/berriai/litellm/issues/312965:3-/https://github.com/berriai/litellm/issues/33325⭐
The left item is harder because it introduces a new protocol-facing authentication and discovery surface, requiring security-sensitive request handling, configuration integration, compatibility validation, and broader end-to-end testing. The right item is comparatively localized to cache synchronization and limiter initialization, with narrower behavioral scope despite distributed-state correctness risks.
#0 of 0 · 31d17h16m46s ago · #import:https:::github.com:berriai:litellm post #3562
-/https://github.com/berriai/litellm/issues/321405:3-/https://github.com/berriai/litellm/issues/33325⭐
Cross-provider request translation and retry semantics require coordinated changes across provider adapters, conversation state, and fallback paths, with substantial compatibility and regression risk. The other task is narrower: correcting shared-state synchronization and initialization in budget enforcement.
#0 of 0 · 31d17h16m25s ago · #import:https:::github.com:berriai:litellm post #3567
-/https://github.com/berriai/litellm/issues/33325⭐3:1-/https://github.com/berriai/litellm/issues/26897
Issue 33325 is harder because it involves distributed quota correctness across replicas, cache precedence, synchronization, concurrency, and regression testing across proxy budget paths. Issue 26897 is narrower, focused on tracing parameter propagation and correcting Azure response URL handling in a specific SDK integration path.
#0 of 0 · 31d17h15m53s ago — current · #import:https:::github.com:berriai:litellm post #3573
-/https://github.com/berriai/litellm/issues/322013:2-/https://github.com/berriai/litellm/issues/33325⭐
Issue 32201 is harder because it requires integrating post-response policy enforcement with unnormalized pass-through traffic, preserving blocking semantics across response formats, and adding broad regression coverage. Issue 33325 appears more localized to limiter initialization and distributed-cache admission consistency.
discussed in #import:https:::github.com:berriai:litellm

ranked child groups

no voted pairs yet in this scope

cli
src
spread
search