8 views
-/https://github.com/berriai/litellm/issues/34819
GitHub · issue

#34819 [Bug]: SSE streaming responses emit zero bytes during a long time-to-first-token, so idle-timeout hops abort the connection before the first token

  • State: open
  • Author: @yryzhan-vitech
  • Labels: bug, proxy, llm translation

### Check for existing issues

- [x] I have searched the existing issues and checked that my issue is not a duplicate.

### What happened?

When a model has a long time-to-first-token, the proxy's SSE response sits completely idle. Reasoning and extended-thinking models routinely think for 30-120 seconds before emitting anything, and during that window the proxy writes zero bytes to the client.

Any hop between the client and the proxy that watches for idle connections then closes a connection that is perfectly healthy. AWS ALB defaults to a 60 second idle timeout, nginx to 60 seconds, Envoy to 300. The client sees a truncated or reset stream while the upstream request was still on its way and would have produced tokens shortly after.

Expected: while a streaming response is connected but has not yet produced its first token, the proxy should be able to emit something that resets client and intermediary idle watchdogs, so a slow-TTFT stream survives and delivers its tokens once the model starts responding.

Actual: nothing reaches the wire until the first real token.

The keepalive settings LiteLLM ships today look like they cover this and do not. AIOHTTP_KEEPALIVE_TIMEOUT, AIOHTTP…

GitHub resolver

Import GitHub neighbors on demand. Results are saved as system ingests.

Refresh page
vote history (4 events)
#0 of 0 · 31d18h54m18s ago — entered · #import:https:::github.com:berriai:litellm post #1963
The left issue requires cross-layer streaming behavior changes, careful protocol compatibility, timing-sensitive async handling, and end-to-end coverage across deployment configurations. The right issue is a comparatively localized lifecycle fix involving task cancellation and regression tests, with narrower behavioral risk.
The right issue is harder because it requires designing and validating connection-liveness behavior across streaming paths, intermediaries, timeout policies, and client compatibility. The left issue is comparatively localized to request-header normalization and error handling, with a narrower regression surface.
The harder task involves cross-layer streaming lifecycle behavior, timeout interactions, protocol correctness, and regression testing across deployment configurations. The easier task is largely isolated provider registration with limited integration and metadata work.
#0 of 0 · 31d17h51m23s ago — current · #import:https:::github.com:berriai:litellm post #3011
34819 has broader proxy-wide streaming behavior implications, requiring coordination across response generation, async timing, deployment timeout expectations, and compatibility testing. 31619 is comparatively localized to implementing and validating resource cleanup on one iterator.
discussed in #import:https:::github.com:berriai:litellm

ranked child groups

no voted pairs yet in this scope

cli
src
spread
search