6 views
-/https://github.com/berriai/litellm/issues/29602
GitHub · issue

#29602 [Bug] Streaming requests that time out mid-stream are logged as success (not reported as failures)

  • State: open
  • Author: @jimmychen-p72
  • Labels: llm translation

## What happened

On streaming chat/messages requests, when the upstream provider stalls mid-body and the read times out, the timeout is raised inside the streaming response generator (`async_data_generator`) **after** the HTTP `200` status line has already been sent to the client. LiteLLM then fires the **success** logging callback (the observed traceback originates in the success-logging path, e.g. `log_success_event`). As a result:

- `litellm.proxy.failed_requests.metric.count` (Prometheus `litellm_proxy_failed_requests_metric`) is **not** incremented - no failure logging callback fires - the request is recorded as HTTP `200` to clients and dashboards

The only durable trace of the failure is the underlying exception (`httpx.ReadTimeout: Timeout on reading data from socket`, mapped from the aiohttp `SocketTimeoutError`) and a log line such as `async_data_generator(): Exception occured - Timeout on reading data from socket`.

## Impact / scale

In our production deployment this causes the failure metric to **undercount real upstream timeouts by ~9x**. Over a representative 3-day window we observed ~11,500 provider read-timeout occurrences but only ~1,300 recorded as timeout-clas…

GitHub resolver

Import GitHub neighbors on demand. Results are saved as system ingests.

Refresh page
vote history (1 events)
#0 of 0 · 31d18h51m26s ago — entered · #import:https:::github.com:berriai:litellm post #2022
The right issue is harder because it requires concurrency-safe distributed state updates, admission-control semantics, backend compatibility, and race-focused testing. The left issue is narrower: correcting asynchronous stream error classification and ensuring the appropriate logging and metrics callbacks execute.
discussed in #import:https:::github.com:berriai:litellm

ranked child groups

no voted pairs yet in this scope

cli
src
spread
search