8 views
-/https://github.com/berriai/litellm/issues/33032
GitHub · issue

#33032 [Bug]: Usage-based routing (lowest_tpm_rpm) excludes a deployment one request before its rpm limit

  • State: open
  • Author: @abhay-codes07
  • Labels: llm translation

### What happened?

The usage-based routing strategies `lowest_tpm_rpm` (v1) and `lowest_tpm_rpm_v2` exclude a deployment from selection **one request before** its configured `rpm` limit is reached, under-utilizing capacity and triggering unnecessary fallbacks.

In `litellm/router_strategy/lowest_tpm_rpm_v2.py` (line 360) and `litellm/router_strategy/lowest_tpm_rpm.py` (line 227) the RPM filter is:

```python rpm_dict[item] + 1 >= _deployment_rpm # excludes when projected usage == limit ```

A request that brings usage to **exactly** the `rpm` limit is within budget. The TPM check in the same function uses strict `>` (`item_tpm + input_tokens > _deployment_tpm`), the sibling strategies `lowest_cost` and `lowest_latency` use `item_rpm + 1 > rpm`, and RPM enforcement in `pre_call_check` raises only when the incremented count **exceeds** the limit. So the correct boundary is strict `>`.

### Minimal reproduction (offline, no network/keys)

```python from litellm.caching.caching import DualCache from litellm.router_strategy.lowest_tpm_rpm_v2 import LowestTPMLoggingHandler_v2

h = LowestTPMLoggingHandler_v2(router_cache=DualCache(), routing_args={}) dep = {"model_name": "gpt", "litell…

GitHub resolver

Import GitHub neighbors on demand. Results are saved as system ingests.

Refresh page
vote history (1 events)
#0 of 0 · 31d17h37m48s ago — entered · #import:https:::github.com:berriai:litellm post #3247
The dashboard issue likely requires tracing state and data flow across frontend, session handling, and request-log APIs, with broader integration and regression risk. The routing issue is comparatively localized to boundary logic in a small number of strategy implementations with focused tests.
discussed in #import:https:::github.com:berriai:litellm

ranked child groups

no voted pairs yet in this scope

cli
src
spread
search