8 views
-/https://github.com/berriai/litellm/issues/27967
GitHub · issue

#27967 [Bug]: Mid-stream fallback request includes assistant prefill block, breaks for fallback targets that don't support `prefix=True` (Claude Sonnet 4.6 / Opus 4.7)

  • State: open
  • Author: @allendeng-eve
  • Labels: llm translation

# What Happened?

When a streaming request fails mid-stream (e.g. `httpx.ReadTimeout`, provider 5xx, network drop), `Router.stream_with_fallbacks` constructs the fallback request with an extra assistant message carrying `prefix=True` and the partially generated text — asking the fallback model to "continue from where the previous model left off."

This continuation prompt is **unconditionally appended** regardless of whether the fallback target supports assistant prefill. For models that do not support it — notably `claude-sonnet-4-6` / `claude-opus-4-7` on both Anthropic and Vertex AI — the fallback request is rejected with:

``` HTTP 400: "This model does not support assistant message prefill. The conversation must end with a user message." ```

Net effect: the fallback feature is silently unusable for these models. Every mid-stream failure → fallback target rejects the malformed request → the original error is swallowed and the user sees a 400 about prefill instead of the real upstream failure (timeout, overload, etc).

## Expected behavior

One of:

1. **Capability-aware:** if the fallback target deployment doesn't support assistant prefill, fall back with the original `message…

GitHub resolver

Import GitHub neighbors on demand. Results are saved as system ingests.

Refresh page
vote history (1 events)
#0 of 0 · 31d18h56m7s ago — entered · #import:https:::github.com:berriai:litellm post #2143
Adding a new audio provider spans provider architecture, request translation, configuration, dependency/runtime compatibility, tests, and documentation. The fallback defect is narrower, mainly involving conditional request construction and regression coverage, though it carries subtle cross-provider behavior risk.
discussed in #import:https:::github.com:berriai:litellm

ranked child groups

no voted pairs yet in this scope

cli
src
spread
search