8 views
-/https://github.com/berriai/litellm/issues/16060
GitHub · issue

#16060 [Bug]: Routing strategy "usage-based-routing-v2" erroneously finds no deployment to use

  • State: open
  • Author: @jeromeroussin
  • Labels: bug

### What happened?

We recently turned on the "usage-based-routing-v2" routing strategy but had to revert to simple-shuffle. We are seeing failed calls and the stacktrace leads to the strategy code at times not finding a deployment to use when there are clearly some.

### Relevant log output

```shell Traceback (most recent call last):\n File \"/usr/local/lib/python3.12/site-packages/litellm/router_strategy/lowest_tpm_rpm_v2.py\", line 504, in async_get_available_deployments\n assert deployment is not None\n ^^^^^^^^^^^^^^^^^^^^^^\nAssertionError\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File \"/usr/local/lib/python3.12/site-packages/litellm/router.py\", line 7278, in async_get_available_deployment\n await self.lowesttpm_logger_v2.async_get_available_deployments(\n File \"/usr/local/lib/python3.12/site-packages/litellm/router_strategy/lowest_tpm_rpm_v2.py\", line 554, in async_get_available_deployments\n raise litellm.RateLimitError(\nlitellm.exceptions.RateLimitError: litellm.RateLimitError: No deployments available for selected model. Passed model=gpt-5. Deployments={'cf177c86187fcf97eaae4d…

GitHub resolver

Import GitHub neighbors on demand. Results are saved as system ingests.

Refresh page
vote history (2 events)
#0 of 0 · 31d17h52m56s ago — entered · #import:https:::github.com:berriai:litellm post #2997
Issue #16060 is harder because it involves diagnosing state, concurrency, filtering, and fallback behavior across the router’s usage-based deployment-selection machinery. Issue #31149 appears narrower, centered on propagating a resolved endpoint through one Anthropic token-counting request path, with related reports likely providing a focused reproduction and fix boundary.
#0 of 0 · 31d17h47m49s ago — current · #import:https:::github.com:berriai:litellm post #3090
The routing defect is harder because it involves diagnosing stateful selection logic, concurrency or timing behavior, rate-limit accounting, and regression testing across deployment-selection scenarios. The dashboard change is comparatively contained: extending existing aggregation and visualization data to include a model dimension.
discussed in #import:https:::github.com:berriai:litellm

ranked child groups

no voted pairs yet in this scope

cli
src
spread
search