8 views
-/https://github.com/berriai/litellm/issues/31877
GitHub · issue

#31877 feat(proxy): add per-deployment keepalive_seconds SSE heartbeat for long-running streams

  • State: open
  • Author: @deepanshululla

## Problem

For long-running streaming LLM responses, reverse proxies and load balancers (nginx, AWS ALB, Cloudflare) close idle SSE connections after their timeout threshold (often 60s). If a model takes longer than this between tokens, the connection is dropped and the client sees a truncated stream with no error.

## Proposed fix

Add a `keepalive_seconds` option to per-deployment config. When set, the proxy sends periodic SSE comment lines (`: keepalive\n\n`) at that interval during streaming pauses, keeping the connection alive through upstream idle-timeout thresholds without affecting the stream content visible to the client.

## Files

- `litellm/proxy/` (streaming response handling)

GitHub resolver

Import GitHub neighbors on demand. Results are saved as system ingests.

Refresh page
vote history (2 events)
#0 of 0 · 31d19h4m8s ago — entered · #import:https:::github.com:berriai:litellm post #1896
The left issue requires coordinated changes in proxy streaming control flow, timing, configuration propagation, and compatibility testing across deployment types. The right issue is narrower UI/session-state debugging with a smaller likely change surface.
#0 of 0 · 31d18h48m16s ago — current · #import:https:::github.com:berriai:litellm post #2166
The heartbeat feature is harder because it affects asynchronous streaming lifecycles, deployment configuration, connection timing, and proxy compatibility, requiring broader integration and regression testing. The other change is more narrowly scoped to provider-specific request translation and validation.
discussed in #import:https:::github.com:berriai:litellm

ranked child groups

no voted pairs yet in this scope

cli
src
spread
search