6 views
-/https://github.com/berriai/litellm/issues/32491
GitHub Β· issue

#32491 [Bug] Anthropic passthrough /v1/messages streaming: no bytes to client during pre-stream processing + long TTFT on large prompts β†’ client no-progress timeouts

  • State: open
  • Author: @seanGSISG
  • Labels: llm translation, claude code

# [Bug] Anthropic passthrough `/v1/messages` streaming: no bytes sent to client during pre-stream processing + long TTFT on large prompts β†’ client-side no-progress timeouts

### What happened

When a client streams a **large-context** request (~200k input tokens) to the Anthropic passthrough (`POST /v1/messages`, `stream=true`), the proxy sends **zero bytes to the client for >180s** before the first event, tripping the client's no-progress / stream-idle watchdog, which aborts the request. The same request sent **directly to `api.anthropic.com`** (bypassing LiteLLM) starts streaming `message_start`/`ping` within ~1s and completes normally (15+ min generations succeed).

The failure is **strongly size-correlated**: small-context requests through the same proxy succeed; only large-context requests die, and every retry dies at the same ~180s wall (no first token). On the proxy side the only trace is `starlette.requests.ClientDisconnect` in `_read_request_body` β€” i.e. the client gave up; **the proxy itself does not log any timeout or upstream error** for these.

### Why this looks like a proxy pre-stream gap (not upstream, not chunk buffering)

I traced `PassThroughStreamingHandler.chun…

GitHub resolver

Import GitHub neighbors on demand. Results are saved as system ingests.

Refresh page
vote history (1 events)
#0 of 0 Β· 31d18h20m19s ago β€” entered Β· #import:https:::github.com:berriai:litellm post #2524
32491 requires cross-layer investigation of streaming lifecycle, buffering, timeout behavior, and large-request performance, with higher regression risk; 34236 is comparatively localized to container filesystem ownership and Prisma engine setup.
discussed in #import:https:::github.com:berriai:litellm

ranked child groups

no voted pairs yet in this scope

cli
src
spread
search