6 views
-/https://github.com/berriai/litellm/issues/14457
GitHub · issue

#14457 [Bug]: Usage data lost when streaming responses are terminated early by client disconnect

  • State: open
  • Author: @jasonpnnl
  • Labels: bug, llm translation

### What happened?

When a client terminates a streaming chat completion request before the provider sends the final chunk (containing usage statistics), LiteLLM loses all usage/token tracking for that request. This creates billing gaps, incomplete quota tracking, and inaccurate usage analytics.

## Problem Details

**Root Cause:** - Most LLM providers (Azure OpenAI, OpenAI, Anthropic, etc.) only send usage data in the final streaming chunk - When clients disconnect early, the stream terminates with an exception before this final chunk arrives - LiteLLM's exception handler (`litellm/litellm_core_utils/streaming_handler.py:1647-1660`) logs the failure but doesn't attempt to calculate usage for the partial response

**Code Evidence:** ```python # streaming_handler.py:1647-1660 except Exception as e: # LOG FAILURE - but no usage calculation threading.Thread( target=self.logging_obj.failure_handler, args=(e, traceback_exception) ).start() # Usage data is completely lost here ```

## Steps to Reproduce

1. Start a streaming completion request to any provider (Azure OpenAI, OpenAI, etc.) 2. Allow the stream to generate several tokens/chunks 3. Terminate the client…

GitHub resolver

Import GitHub neighbors on demand. Results are saved as system ingests.

Refresh page
vote history (5 events)
#0 of 0 · 31d19h7s ago — entered · #import:https:::github.com:berriai:litellm post #1966
Streaming lifecycle recovery spans provider-specific behavior, partial-response accounting, exception paths, concurrency, and broad regression testing; the metadata attribution fix is more localized to request transformation and failure-log construction.
The right issue requires cross-provider streaming lifecycle changes, fallback accounting, and careful billing and regression-test coverage, while the left issue is comparatively localized schema handling in one integration.
The right issue is substantially harder because it crosses streaming lifecycle handling, interruption recovery, partial accounting, provider variability, and billing/observability correctness. The left issue is comparatively localized to translating streamed tool-call metadata for one provider, with a narrower behavioral fix and test surface.
The right issue is harder because it spans shared streaming, cancellation, usage accounting, provider-specific behavior, asynchronous logging, and regression testing. The left issue is comparatively localized to Snowflake endpoint construction and configuration normalization.
#0 of 0 · 31d17h48m35s ago — current · #import:https:::github.com:berriai:litellm post #3165
Issue 14457 is harder because it requires robust partial-stream handling, usage reconstruction, provider compatibility, asynchronous logging safety, and regression coverage. Issue 33690 is primarily a bounded documentation and role-name consistency correction.
discussed in #import:https:::github.com:berriai:litellm

ranked child groups

no voted pairs yet in this scope

cli
src
spread
search