10 views
-/https://github.com/berriai/litellm/issues/31067
GitHub · issue

#31067 [Bug]: Mid-stream fallback continuation prompt corrupts tool-calling context — fallback returns prose instead of tool calls

  • State: open
  • Author: @elvis-cai
  • Labels: llm translation, SDK

## Check for existing issues

Related to #27967 (which covers `prefix=True` rejection for claude-sonnet-4-6). This issue covers a **distinct silent failure mode** that occurs even when the fallback model accepts the continuation without error.

## What happened?

When a streaming request that includes `tools` fails mid-stream, `Router.stream_with_fallbacks` unconditionally appends a continuation prompt to the fallback request:

```python # router.py (async path ~line 1842, sync path ~line 1985) initial_kwargs["messages"] = messages + [ {"role": "system", "content": "...respond in continuation of this text: "}, {"role": "assistant", "content": e.generated_content, "prefix": True}, ] ```

This continuation strategy is designed for pure text generation. When the original request included `tools`, the appended assistant partial-text message signals to the fallback model that it is already mid-way through a text response. The fallback model **continues generating prose instead of invoking tools** — even though `tools` is still present in `initial_kwargs`.

The result is a **silent corruption**: no exception is raised, the stream completes successfully, but the response contains …

GitHub resolver

Import GitHub neighbors on demand. Results are saved as system ingests.

Refresh page
vote history (1 events)
#0 of 0 · 31d18h58m47s ago — entered · #import:https:::github.com:berriai:litellm post #1877
Issue 31067 requires coordinated changes across streaming fallback paths, preservation of tool-call semantics, and regression coverage for multiple response states. Issue 20282 appears narrower and likely involves configuration, dependency, or callback initialization diagnosis and documentation.
discussed in #import:https:::github.com:berriai:litellm

ranked child groups

no voted pairs yet in this scope

cli
src
spread
search