10 views
-/https://github.com/berriai/litellm/issues/24004
GitHub · issue

#24004 [Bug]: Mid-stream fallback not supported for anthropic_messages route type

  • State: open
  • Author: @abrekhov
  • Labels: llm translation

### What happened?

When using the LiteLLM proxy's `/v1/messages` endpoint (which goes through the router via `anthropic_messages` route type), mid-stream errors from Anthropic (e.g. `overloaded_error`, `internal_server_error` SSE events) are not caught by the router's fallback system.

### Root Cause

The `_ageneric_api_call_with_fallbacks_helper` method in `router.py` returns the streaming `AsyncIterator` directly without wrapping it in a fallback-aware wrapper. Compare with the `acompletion` path which wraps streaming responses in `FallbackStreamWrapper` via `_acompletion_streaming_iterator` (router.py:2162-2167).

For the generic path at router.py:3855, the response is simply `return response` with no error detection or fallback wrapping.

### Expected Behavior

When Anthropic returns HTTP 200 and starts streaming, then sends an SSE error event (`event: error` with `overloaded_error` or `internal_server_error`), the router should detect this mid-stream error and attempt to fall back to a configured fallback model (e.g. Bedrock).

### Actual Behavior

The SSE error event passes through silently to the client. The request is logged as `status=success` in the spend logs. No fallba…

GitHub resolver

Import GitHub neighbors on demand. Results are saved as system ingests.

Refresh page
vote history (1 events)
#0 of 0 · 31d18h23m49s ago — entered · #import:https:::github.com:berriai:litellm post #2656
The right issue is harder because it spans schema evolution, API validation, persistence, authorization, reporting, routing hooks, and UI changes, creating substantial cross-layer compatibility and migration risk. The left issue is more narrowly scoped to asynchronous stream error detection and fallback wrapping, though it requires careful handling of partial responses and regression testing.
discussed in #import:https:::github.com:berriai:litellm

ranked child groups

no voted pairs yet in this scope

cli
src
spread
search