8 views
-/https://github.com/berriai/litellm/issues/34399
GitHub · issue

#34399 [Bug]: Router ignores Retry-After (instant retry) for multi-deployment model groups sharing one rate-limited upstream

  • State: open
  • Author: @segevfiner
  • Labels: proxy, llm translation

## Check for existing issues - [x] I have searched the existing issues and checked that my issue is not a duplicate.

## What happened?

`Router._time_to_sleep_before_retry` only honors the upstream `Retry-After` header when a model group has a **single** deployment. As soon as a model group has **more than one** deployment (and there is ≥1 "healthy" deployment), it returns `0` and retries **instantly**, discarding `Retry-After`:

```python # litellm/router.py :: _time_to_sleep_before_retry ## base case - single deployment if all_deployments is not None and len(all_deployments) == 1: pass # -> reads Retry-After from the exception headers elif ( healthy_deployments is not None and isinstance(healthy_deployments, list) and len(healthy_deployments) > 0 ): return 0 # -> instant retry, Retry-After ignored ```

The assumption is that with multiple deployments the router can immediately fail over to a *different, healthy* deployment. But when the deployments in the group all point to the **same upstream / share the same rate-limit or token quota**, they are all throttled at once. The instant retries then just hammer the throttled…

GitHub resolver

Import GitHub neighbors on demand. Results are saved as system ingests.

Refresh page
vote history (1 events)
#0 of 0 · 31d18h27m47s ago — entered · #import:https:::github.com:berriai:litellm post #2589
Issue #34399 is harder because it affects router-wide retry and failover behavior across deployment groups, requiring careful handling of shared upstream limits, health state, timing, and regression coverage. Issue #31959 is more localized to request-scoped token allocation and synchronization in the Presidio guardrail flow.
discussed in #import:https:::github.com:berriai:litellm

ranked child groups

no voted pairs yet in this scope

cli
src
spread
search