8 views
-/https://github.com/berriai/litellm/issues/30667
GitHub ยท issue

#30667 [Bug]: Streaming pass-through requests that fail mid-stream (timeout / connection error) log no usage/cost

  • State: open
  • Author: @billyang-scale
  • Labels: llm translation

### What happened?

For provider **pass-through** endpoints with streaming (e.g. `POST /anthropic/v1/messages` with `stream: true`), usage/cost is logged **only when the stream completes successfully**. If the upstream stream raises **mid-flight** โ€” request timeout, connection reset, or a provider 5xx mid-stream โ€” the request logs **no usage at all**, even though the provider has already generated and **billed** the tokens produced before the failure.

This causes a systematic, often large, under-count of tracked spend vs the provider invoice โ€” and it's worst exactly where it hurts most: **long agentic streams** (large/cached context + server tools like web_search) are both the biggest token consumers *and* the most likely to hit the request timeout. So the largest requests contribute **zero** to SpendLogs.

**Root cause** โ€” `litellm/proxy/pass_through_endpoints/streaming_handler.py`, `PassThroughStreamingHandler.chunk_processor()`:

```python async for chunk in response.aiter_bytes(): raw_bytes.append(chunk) ... yield chunk # logging is scheduled ONLY here, after the loop completes: asyncio.create_task(PassThroughStreamingHandler._route_streaming_logging_to_handler(...โ€ฆ

GitHub resolver

Import GitHub neighbors on demand. Results are saved as system ingests.

Refresh page
vote history (1 events)
#0 of 0 ยท 31d18h2m16s ago โ€” entered ยท #import:https:::github.com:berriai:litellm post #2922
The left issue spans streaming error recovery, partial accounting, asynchronous logging, and provider-specific edge cases, requiring broader backend changes and reliability testing. The right issue is a localized dashboard data-resolution bug with a narrower UI/data-flow fix.
discussed in #import:https:::github.com:berriai:litellm

ranked child groups

no voted pairs yet in this scope

cli
src
spread
search