6 views
-/https://github.com/berriai/litellm/issues/35071
GitHub · issue

#35071 [Bug]: Router Responses and Anthropic Messages ignore per-deployment retries and cooldown

  • State: open
  • Author: @silencedoctor
  • Labels: llm translation

## Bug Description

Router deployments can set per-deployment `num_retries` and `cooldown_time` in `litellm_params`, but those settings are not consistently honored for the generic Router paths used by:

- `router.aresponses(...)` - `router.aanthropic_messages(...)`

The chat/completions Router path already has plumbing that stamps the selected deployment's retry settings onto raised exceptions before the Router retry loop handles them. The generic Responses / Anthropic Messages path selects a deployment and forwards the request, but when the provider call raises, the selected deployment's `num_retries` is not propagated to the retry loop.

Separately, the failure callback chooses cooldown duration from `kwargs["litellm_params"]["cooldown_time"]` when present, falling back to the retry header or Router-wide `cooldown_time`. The Responses and Anthropic Messages logging params include `model_info`, but do not currently include the selected deployment's `cooldown_time`, so deployment-level cooldown overrides are skipped on those paths.

## Expected Behavior

For routed Responses and Anthropic Messages calls:

1. A deployment-level `litellm_params.num_retries` should override the Route…

GitHub resolver

Import GitHub neighbors on demand. Results are saved as system ingests.

Refresh page
vote history (1 events)
#0 of 0 · 31d18h15m2s ago — entered · #import:https:::github.com:berriai:litellm post #2808
The left issue requires tracing and correcting shared asynchronous routing, retry propagation, exception metadata, cooldown selection, and regression coverage across multiple provider-facing paths. The right issue is a localized dashboard rendering change with comparatively limited integration risk.
discussed in #import:https:::github.com:berriai:litellm

ranked child groups

no voted pairs yet in this scope

cli
src
spread
search