10 views
-/https://github.com/berriai/litellm/issues/34875
GitHub · issue

#34875 [Bug]: Streaming: sync success handler races the async handler on shared model_call_details, so custom callbacks intermittently read response_cost=None

  • State: open
  • Author: @fvcci
  • Labels: llm translation

## What happened

For streaming responses, custom callbacks that read `kwargs["response_cost"]` in `async_log_success_event` intermittently (for us: *usually*) observe `None` even though the call succeeded, litellm computed a real cost, and token usage is fully populated. Any cost/spend tracking built on that key silently records $0.

In our proxy deployment (`litellm[proxy]==1.83.14`, Python 3.13), **80.7% of streaming success rows (245,562 of 304,148)** recorded a zero cost with real token counts. The rate was load-independent but strongly model-correlated (~93–97% for gpt-5.x streams, ~0% for Claude streams), consistent with a scheduling race rather than a data problem.

## Root cause

Line references are into `litellm==1.83.14`.

1. **Both success handlers fire concurrently for a finished stream.** `litellm_core_utils/streaming_handler.py:2209-2224`: when a stream completes, `async_success_handler` is scheduled as an asyncio task and `success_handler` is simultaneously submitted to the thread-pool `executor`. This happens even when no sync callbacks are registered.

2. **Both handlers mutate the same dict.** Each handler reads and writes `self.model_call_details`…

GitHub resolver

Import GitHub neighbors on demand. Results are saved as system ingests.

Refresh page
vote history (1 events)
#0 of 0 · 31d17h50m38s ago — entered · #import:https:::github.com:berriai:litellm post #3039
34875 requires careful concurrency redesign across streaming completion paths, shared-state ownership, callback ordering, and regression coverage, making it riskier and broader than the targeted Redis client integration fix in 28379.
discussed in #import:https:::github.com:berriai:litellm

ranked child groups

no voted pairs yet in this scope

cli
src
spread
search