9 views
-/https://github.com/berriai/litellm/issues/31722
GitHub · issue

#31722 Streaming: response `model` is overridden to the requested model after a fallback (served deployment masked); non-streaming is correct

  • State: open
  • Author: @GiulioSurya
  • Labels: llm translation

### What happened

When a request is routed through the proxy with model-group **fallbacks** and the primary deployment fails (e.g. a `BadRequestError` / `UnsupportedParamsError`), the request correctly falls back to a secondary deployment.

However, the `model` field reported back to the client differs between non-streaming and streaming:

- **Non-streaming** → `response.model` is the **served (fallback) deployment** — correct. - **Streaming** → every chunk's `model` is the **originally requested model group**, not the deployment that actually generated the tokens — the fallback is **silently masked**.

A streaming client therefore cannot tell from the response body that a fallback happened: `model` reports the requested model while a different deployment served the request.

### Reproduction

Config — primary group falls back to a secondary group:

```yaml model_list: - model_name: primary-model litellm_params: model: openai/<provider-model-that-rejects-some-param> api_base: https://.../v1 - model_name: secondary-model litellm_params: model: bedrock/<some-bedrock-model> router_settings: fallbacks: [{"primary-model": ["secondary-model"]}] litellm_se…

GitHub resolver

Import GitHub neighbors on demand. Results are saved as system ingests.

Refresh page
vote history (2 events)
#0 of 0 · 31d18h29m13s ago — entered · #import:https:::github.com:berriai:litellm post #2399
Streaming response metadata must remain consistent across fallback routing, async chunk generation, and provider-specific wrappers, creating broader integration risk. The budget issue is more localized to validation, spend-source selection, and boundary-condition tests.
#0 of 0 · 31d18h23m12s ago — current · #import:https:::github.com:berriai:litellm post #2481
Streaming fallback identity involves cross-cutting routing, response assembly, and streaming regression coverage, whereas the other fix is a more localized translation/schema-filtering change.
discussed in #import:https:::github.com:berriai:litellm

ranked child groups

no voted pairs yet in this scope

cli
src
spread
search