6 views
-/https://github.com/berriai/litellm/issues/35578
GitHub Β· issue

#35578 [Bug]: Mid-stream fallback resolves `fallbacks` by the selected deployment instead of the requested model_group, discarding the entire chain

  • State: open
  • Author: @aguscaba
  • Labels: llm translation

### What happened?

When a **streaming** request to a model group fails *after* the first chunk, the router looks up the fallback chain using the **resolved deployment's** `model_name` instead of the **model group the client actually requested**. Since the deployment name is not a key in the `fallbacks` map, the lookup returns nothing and the router discards the entire remaining chain instead of walking it.

The client requested `pool-free`, but the error reports `original model_group=minimax-m3-nvidia`, which is the deployment the router had selected. The `fallbacks` map is clearly loaded β€” it is printed verbatim in the same error message β€” yet `Available Model Group Fallbacks=None`.

This means a pool with 11 fallback links provides **zero** failover as soon as the upstream fails mid-stream. `Cooldown Deployments=[]` confirms the other 10 deployments were available and simply never tried.

### Relevant config

```yaml model_list: - model_name: pool-free # the group clients request litellm_params: model: nvidia_nim/z-ai/glm-5.2 api_key: os.environ/NVIDIA_API_KEY timeout: 90 stream_timeout: 25 - model_name: minimax-m3-nvidia # one of th…

GitHub resolver

Import GitHub neighbors on demand. Results are saved as system ingests.

Refresh page
vote history (1 events)
#0 of 0 Β· 31d17h42m25s ago β€” entered Β· #import:https:::github.com:berriai:litellm post #3182
The right-hand feature has broader architectural scope across configuration schema, proxy integration, persistence, validation, and backward compatibility, while the left-hand bug appears localized to fallback resolution and regression coverage.
discussed in #import:https:::github.com:berriai:litellm

ranked child groups

no voted pairs yet in this scope

cli
src
spread
search