18 views
-/https://github.com/berriai/litellm/issues/27736
GitHub · issue

#27736 [Bug]: Deployment-level TPM enforcement is per-pod, not cross-pod — effective limit becomes `tpm_limit × N_replica`

  • State: open
  • Author: @hula-la
  • Labels: bug, proxy

### Check for existing issues

- [x] I have searched the existing issues and checked that my issue is not a duplicate.

### What happened?

In a multi-replica LiteLLM proxy deployment with `usage-based-routing-v2`, the deployment-level TPM limit (`litellm_params.tpm` in `model_list`) is enforced against each replica's local in-memory counter rather than the cross-pod sum. The effective per-deployment TPM ceiling becomes `tpm_limit × N_replica`, and traffic up to that ceiling passes through with zero 429 responses.

### Root cause (code-level)

**RPM keys are batch-synced across replicas** (since #9357 — "support batch writing increments to redis"), but **TPM keys are not**:

`lowest_tpm_rpm_v2.async_log_success_event` (line ~305): ```python await self.router_cache.async_increment_cache( key=tpm_key, value=total_tokens, ttl=..., parent_otel_span=..., ) ```

This calls `DualCache.async_increment_cache`, which writes to both in-memory and Redis directly, **but does not register the key with the queue-based sync mechanism** in `base_routing_strategy.py`. The periodic `_sync_in_memory_spend_with_redis` task only processes keys that go through `_increment_value_in_current_window` (wh…

GitHub resolver

Import GitHub neighbors on demand. Results are saved as system ingests.

Refresh page
vote history (3 events)
#0 of 0 · 31d19h3m27s ago — entered · #import:https:::github.com:berriai:litellm post #1822
The left task requires coordinated state handling across distributed proxy instances, with concurrency, consistency, and regression-testing risks. The right task is comparatively localized to dashboard routing or packaging configuration.
Cross-instance quota enforcement requires changes to shared-state synchronization, concurrency behavior, and distributed integration tests, creating substantially greater engineering risk than a provider metadata and pricing refresh.
#0 of 0 · 31d17h54m20s ago — current · #import:https:::github.com:berriai:litellm post #2972
The proxy fix requires distributed state consistency, synchronization semantics, race-condition handling, and multi-replica testing, while the other is primarily a dependency-resolution and lockfile update.
discussed in #import:https:::github.com:berriai:litellm

ranked child groups

no voted pairs yet in this scope

cli
src
spread
search