11 views
-/https://github.com/berriai/litellm/issues/30460
GitHub · issue

#30460 [Bug]: Redis spend counters inflate over time → false BudgetExceededError (429) on multi-pod + ElastiCache timeouts (v1.85.3)

  • State: open
  • Author: @johnbfj2041-dot
  • Labels: bug, proxy

### Check for existing issues

- [x] I have searched the existing issues and checked that my issue is not a duplicate.

### What happened?

In a 2-pod EKS deployment with AWS ElastiCache and `use_redis_transaction_buffer: true`, the Redis spend counters (`spend:key:*` and `spend:team:*`) gradually inflate far above the authoritative DB values during normal operation. Once a counter exceeds `max_budget`, legitimate requests are rejected with `429 BudgetExceededError` even though actual spend (from `LiteLLM_VerificationToken.spend` + `LiteLLM_SpendLogs`) is well within budget.

**Confirmed observations:** - Redis `spend:key:<hash>` drifts well above DB over time. Example: Redis = `50`, DB = `14`. - No `LiteLLM_SpendLogs` entries account for the difference — the phantom amount was never a real request cost. - Both `spend:key:*` and `spend:team:*` counters are affected. - ElastiCache timeouts appear repeatedly in logs (see Relevant log output). - Flushing the Redis counters restores correct behavior temporarily, but inflation recurs within hours.

**Expected behavior:** Redis spend counters remain consistent with DB values. Budget enforcement should not block requests whose true spend …

GitHub resolver

Import GitHub neighbors on demand. Results are saved as system ingests.

Refresh page
vote history (28 events)
#0 of 0 · 31d18h55m59s ago — entered · #import:https:::github.com:berriai:litellm post #1933
Model omitted braces; inferred difficulty from issue scope and surface area.
The distributed accounting issue is harder because it requires diagnosing concurrency, retries, timeout behavior, and cross-pod consistency across Redis and database paths, with broader regression testing. The pricing issue is comparatively localized to metadata and cost-calculation coverage.
Issue #30460 is harder because it involves distributed Redis state, timeout and retry behavior, cross-pod consistency, reconciliation, and budget-enforcement correctness. Issue #25628 is comparatively localized to preserving model-alias metadata through the streaming callback path.
A cross-language gateway rewrite spanning core execution paths, compatibility, performance, deployment, and staged migration carries far greater engineering scope and regression risk than a focused distributed-state accounting fix.
The right issue is harder because it requires diagnosing distributed accounting, timeout and retry behavior, concurrency, persistence consistency, and multi-pod regression coverage. The left issue is comparatively narrower, centered on parameter propagation and endpoint selection across an integration path.
The left issue requires diagnosing and correcting distributed state consistency under retries, timeouts, concurrency, and multi-pod failure modes, with safeguards against financial mis-enforcement and regression testing. The right issue is a bounded provider integration involving request translation, model metadata, pricing, and usage reporting, with comparatively clearer implementation boundaries.
Issue 30460 is harder because it requires diagnosing and correcting distributed state, concurrency, timeout, reconciliation, and budget-enforcement behavior across multiple storage layers, with substantial regression risk. Issue 31833 is a more bounded authentication feature spanning API and dashboard flows, validation, persistence, and security tests.
The left task is harder because it requires diagnosing and correcting distributed state corruption under failures, preserving idempotency and budget correctness across pods and storage layers, and validating behavior under concurrency and timeout scenarios. The right task is a more bounded input-normalization enhancement with provider-specific conversion and validation.
34241 requires broad cross-cutting architectural changes, repository-wide dependency and packaging decisions, and careful compatibility and licensing validation. 30460 is a more localized production bug involving distributed state, failure handling, and regression testing.
30460 requires distributed-state debugging, failure-mode analysis, and cross-path correctness testing; 30362 is comparatively bounded transport/configuration work with compatibility testing.
The right issue is harder because it involves diagnosing distributed state consistency, timeout behavior, idempotency, and multi-pod budget accounting, followed by reliable regression coverage. The left issue is broad but primarily structured QA, matrix validation, and CI coordination.
The left task requires diagnosing and correcting distributed state consistency under failure conditions, with concurrency, retries, persistence, and backward-compatibility risks. The right task is primarily a bounded configuration and policy-integration feature with a clearer implementation surface.
Issue 21347 has substantially broader scope: it requires designing and maintaining spec-driven validation across many providers, endpoints, schemas, streaming formats, and mappings, with extensive test infrastructure and compatibility decisions. Issue 30460 is a narrower distributed-state debugging and correction effort, despite meaningful concurrency and failure-mode risk.
31821 is harder because it requires coordinated changes across persistence, request enforcement, calendar-window semantics, APIs, and UI, while 30460 is primarily a focused distributed-state debugging and correctness fix.
Implementing this requires a new CLI workflow spanning multiple external agent integrations, authentication/configuration translation, model discovery, interactive selection, subprocess behavior, and broad compatibility testing. The other issue is a narrower proxy accounting defect, though diagnosing distributed Redis timeout and atomicity behavior still carries meaningful risk.
Distributed counter correctness requires tracing cross-pod concurrency, Redis transaction buffering, timeout/retry semantics, reconciliation, and regression testing across deployment configurations; the other issue is narrower provider request-shape and endpoint-routing work.
The right issue is harder because it spans backend API design, authorization and tenant isolation, usage aggregation, and a new dashboard experience, while the left issue is a narrower but technically difficult distributed-counter debugging and correction effort.
Distributed accounting with timeout and retry behavior requires tracing cross-process state, preserving idempotency, and validating reconciliation under failure; the cache-invalidation change is narrower, mainly involving event publication, subscription handling, and worker synchronization.
30460 requires diagnosing and safely correcting distributed Redis/DB accounting under concurrency, transaction buffering, failures, and multi-pod deployment conditions, with substantial regression and data-integrity risk. 25191 is more localized to endpoint-specific tool-call interception and shared agent-loop behavior, with a narrower implementation and test surface.
31835 is harder because it spans new administrative APIs, pricing-version semantics, configurable policy behavior, repeatable data mutation, auditing, and compatibility across historical records. 30460 is a demanding distributed-consistency and failure-mode investigation, but its scope is more concentrated around correcting one spend-counter path.
30460 requires deeper distributed-state debugging, timeout/retry semantics, atomic accounting, reconciliation, and multi-pod regression testing; 26237 is comparatively narrower lifecycle readiness and retry handling.
The left requires diagnosing and hardening distributed accounting across Redis, database reconciliation, retries, and timeout behavior, with significant concurrency and regression risk. The right is primarily a routing/configuration abstraction that needs defined resolution and compatibility tests, but has a narrower implementation surface.
30460 is harder because it requires diagnosing and correcting distributed state, timeout, retry, and atomicity behavior across multiple persistence layers, with careful regression testing. 31467 is comparatively narrower: it mainly needs secure credential-selection rules and provider/API-base compatibility tests.
#28235 requires coordinated changes across data modeling, configuration flows, authorization, counter reservation, expiration/reset processing, and comprehensive compatibility testing. #30460 is a high-risk distributed-systems bug involving Redis failure handling and counter idempotency, but its implementation scope is more concentrated.
The left issue entails designing and integrating a new extensibility contract across proxy routing, configuration, lifecycle, and compatibility boundaries. The right issue is a difficult distributed-state debugging and correctness fix, but its scope is more targeted once the failure path is isolated.
Issue 29452 is harder because it requires a cross-cutting authentication and secret-storage abstraction with provider compatibility, credential lifecycle handling, and security-sensitive integration. Issue 30460 is a narrower but risky distributed-state debugging and correction effort.
The left issue is harder because it involves diagnosing and correcting distributed state consistency under Redis failures, transactional buffering, concurrency, and budget-enforcement correctness. The right issue is broader in coverage but primarily consists of provider integrations, model mappings, parameter translation, and validation, making its risk and architectural depth lower.
#0 of 0 · 31d17h18m24s ago — current · #import:https:::github.com:berriai:litellm post #3551
The left issue is harder because it requires diagnosing and correcting distributed state, failure handling, concurrency, and backward-compatibility risks across persistence and budget-enforcement paths, with difficult reproducibility and regression testing. The right issue is primarily an API-contract and metadata-plumbing change around existing router state, with comparatively contained implementation and validation.
discussed in #import:https:::github.com:berriai:litellm

ranked child groups

no voted pairs yet in this scope

cli
src
spread
search