16 views
-/https://github.com/berriai/litellm/issues/24929
GitHub · issue

#24929 [Bug]: Streaming responses fail in bursts aligned with httpx client ttl

  • State: open
  • Author: @micahjsmith
  • Labels: bug, proxy, llm translation

### Check for existing issues

- [x] I have searched the existing issues and checked that my issue is not a duplicate.

### What happened?

I have LiteLLM Proxy deployed (self-managed) and use it for streaming completions via /v1/messages endpoint. The client uses anthropic.AsyncAnthropic => proxy => Vertex. However, the client observes periodic large spikes of httpx.ReadTimeout errors. Timeouts spike every 3600s, which matches _DEFAULT_TTL_FOR_HTTPX_CLIENTS. When I monkeypatch this TTL in my proxy deployment to 5400s, the spikers shift to every 90m, conclusively implicating the httpx client TTL mechanism in this bug.

Spikes in client-experienced httpx.ReadTimeout with default settings (TTL is 60m) <img width="697" height="339" alt="Image" src="https://github.com/user-attachments/assets/39a2bc35-a524-410d-b19e-a985c6fc9bc3" />

Spikes in client-experienced httpx.ReadTimeout after TTL is monkeypatched to 90m <img width="699" height="338" alt="Image" src="https://github.com/user-attachments/assets/0c03fe24-278a-4d94-8dd9-0dda329412b7" />

Note that we confirmed that the center of the spikes is aligned with exactly the minute that a new revision of the proxy service was deployed. So …

Open on GitHub

GitHub resolver

Import GitHub neighbors on demand. Results are saved as system ingests.

Refresh page
vote history (6 events)
#0 of 0 · 31d19h16m33s ago — entered · #import:https:::github.com:berriai:litellm post #1695
-/https://github.com/berriai/litellm/issues/24929⭐5:1-/https://github.com/berriai/litellm/issues/34574
The left issue is harder because it requires diagnosing and safely changing shared HTTP-client lifecycle behavior under sustained streaming and deployment rollover conditions, with broader regression and concurrency risk. The right issue is comparatively localized to parameter validation and provider-specific mapping with focused unit coverage.
#0 of 0 · 31d19h6m15s ago · #import:https:::github.com:berriai:litellm post #1855
-/https://github.com/berriai/litellm/issues/24929⭐5:2-/https://github.com/berriai/litellm/issues/26787
The right-hand task is harder because it requires diagnosing and safely changing shared HTTP-client lifecycle behavior under long-lived streaming, concurrency, timeout, and deployment conditions. The left-hand task is a localized request-parameter validation and filtering fix with a narrower regression surface.
#0 of 0 · 31d19h1m6s ago · #import:https:::github.com:berriai:litellm post #1942
-/https://github.com/berriai/litellm/issues/24929⭐5:1-/https://github.com/berriai/litellm/issues/29156
#24929 requires diagnosing and safely changing shared HTTP-client lifecycle behavior under concurrency, streaming, timeout, and deployment conditions, with broader regression testing. #29156 is comparatively localized response-shape normalization in one provider adapter.
#0 of 0 · 31d18h59m39s ago · #import:https:::github.com:berriai:litellm post #1967
-/https://github.com/berriai/litellm/issues/24929⭐10:1-/https://github.com/berriai/litellm/issues/31702
Requires cross-layer diagnosis, client-lifecycle/concurrency changes, and regression testing; the other has no actionable technical scope.
#0 of 0 · 31d18h53m45s ago · #import:https:::github.com:berriai:litellm post #2069
-/https://github.com/berriai/litellm/issues/24929⭐5:3-/https://github.com/berriai/litellm/issues/35526
The right issue is harder because it likely requires diagnosing and redesigning shared HTTP-client lifecycle and concurrency behavior across long-lived streaming workloads, with timing-sensitive reproduction and regression testing. The left issue is narrower: it mainly requires reconciling authorization control flow across existing authentication paths, updating defaults or guards, and adding security-focused tests.
#0 of 0 · 31d18h33m2s ago — current · #import:https:::github.com:berriai:litellm post #2414
-/https://github.com/berriai/litellm/issues/24929⭐3:1-/https://github.com/berriai/litellm/issues/35570
Issue #24929 is harder because it likely requires tracing long-lived streaming connections, HTTP client lifecycle, timeout behavior, and concurrency across proxy layers, whereas #35570 is a narrower accounting-state correction with a more localized implementation path.
discussed in #import:https:::github.com:berriai:litellm

ranked child groups

no voted pairs yet in this scope

cli
src
spread
search