6 views
-/https://github.com/berriai/litellm/issues/28216
GitHub · issue

#28216 [Bug]: Router.aresponses streaming bypasses mid-stream fallback (MidStreamFallbackError not handled)

  • State: open
  • Author: @cwang-otto
  • Labels: llm translation, SDK

### What happened?

`MidStreamFallbackError` raised mid-stream during `Router.aresponses(stream=True)` bypasses the Router's fallback chain. Configured cross-provider fallbacks (e.g. `anthropic → vertex_ai`) never fire when the primary provider's stream fails mid-flight.

Observed in production: an Anthropic socket timed out before the first chunk on `Router.aresponses(stream=True)`. The underlying `CustomStreamWrapper` raised `MidStreamFallbackError` from [`_handle_stream_fallback_error`](https://github.com/BerriAI/litellm/blob/main/litellm/litellm_core_utils/streaming_handler.py) — intended behavior, that's exactly what's supposed to drive fallback. The Router's configured `anthropic → vertex_ai` fallback was not invoked; the error surfaced unhandled to the caller.

### Root cause

The chat completions path wraps its `CustomStreamWrapper` via [`Router._acompletion_streaming_iterator`](https://github.com/BerriAI/litellm/blob/main/litellm/router.py), which catches `MidStreamFallbackError` and re-enters `async_function_with_fallbacks_common_utils`.

The Responses API path doesn't have an equivalent: - `Router.aresponses` is bound via `factory_function(litellm.aresponses, call_type="…

GitHub resolver

Import GitHub neighbors on demand. Results are saved as system ingests.

Refresh page
vote history (2 events)
#0 of 0 · 31d19h13m35s ago — entered · #import:https:::github.com:berriai:litellm post #1956
The right-side work is broader: it requires coordinated changes across virtual-key configuration, request validation, routing behavior, precedence rules, compatibility, and extensive proxy-level testing. The left-side work is a more localized async streaming error-path fix with targeted regression coverage.
#0 of 0 · 31d18h32m59s ago — current · #import:https:::github.com:berriai:litellm post #2615
The right-hand task spans asynchronous streaming control flow, fallback orchestration, and cross-path behavioral testing, with greater risk of regressions. The left-hand task is a localized input-normalization change with a comparatively narrow provider-specific scope.
discussed in #import:https:::github.com:berriai:litellm

ranked child groups

no voted pairs yet in this scope

cli
src
spread
search