6 views
-/https://github.com/berriai/litellm/issues/28580
GitHub · issue

#28580 [Bug]: Anthropic /v1/messages → hosted_vllm silently drops assistant-message prefill (continue_final_message never reaches vLLM)

  • State: open
  • Author: @zenprocess
  • Labels: llm translation, claude code

## What happened

When a request comes in on `/v1/messages` (Anthropic shape) whose last message is `role:"assistant"` — i.e. the [Anthropic-spec prefill semantic](https://docs.anthropic.com/en/api/messages) — and is routed to a `hosted_vllm/*` model, **vLLM never receives `continue_final_message: true`**. The model treats the prefill as a completed prior turn and starts a fresh response instead of continuing.

This silently breaks the Anthropic prefill contract for the entire self-hosted ecosystem behind LiteLLM. Claude Code (and any other Anthropic-SDK client) cannot use prefill against a vLLM backend through LiteLLM, even though it works fine when calling vLLM directly via `/v1/chat/completions` with `continue_final_message: true`.

Verified on **LiteLLM v1.82.6**, vLLM serving `qwen3.6-35b-a3b`.

## Repro

### A — broken (Anthropic shape, current behavior) ```bash curl -sS -X POST "https://<proxy>/v1/messages" \ -H "x-api-key: $KEY" -H "anthropic-version: 2023-06-01" -H "content-type: application/json" \ -d '{"model":"qwen3-coder","max_tokens":50,"system":"You are precise.", "messages":[{"role":"user","content":"Count to three, just the numbers comma-separated."}, …

GitHub resolver

Import GitHub neighbors on demand. Results are saved as system ingests.

Refresh page
vote history (2 events)
#0 of 0 · 31d19h5m56s ago — entered · #import:https:::github.com:berriai:litellm post #1973
The left issue spans cross-provider request translation and preserves nuanced conversational semantics across Anthropic and vLLM pathways, requiring careful compatibility testing. The right issue is a narrower provider-specific parameter mapping and capability change with a smaller implementation surface.
#0 of 0 · 31d19h2m31s ago — current · #import:https:::github.com:berriai:litellm post #2032
The left issue is harder because it involves event-loop safety, CPU-bound execution strategy, concurrency behavior, performance under extreme workloads, and regression testing across streaming paths. The right issue is comparatively localized to request translation and backend-specific propagation with focused coverage.
discussed in #import:https:::github.com:berriai:litellm

ranked child groups

no voted pairs yet in this scope

cli
src
spread
search