7 views
-/https://github.com/berriai/litellm/issues/33166
GitHub · issue

#33166 [BUG] prompt_tokens_details silently reset to None in streaming usage aggregation when a later chunk omits it

  • State: open
  • Author: @khankaholic
  • Labels: llm translation

## Bug description

`ChunkProcessor._calculate_usage_per_chunk` in `litellm/litellm_core_utils/streaming_chunk_builder_utils.py` aggregates usage across all chunks in a stream. `completion_tokens_details` is only overwritten when the current chunk's value is not `None`:

```python if usage_chunk_dict["completion_tokens_details"] is not None: completion_tokens_details = usage_chunk_dict["completion_tokens_details"] ```

`prompt_tokens_details` has no equivalent guard, it's unconditionally reassigned on every usage-bearing chunk:

```python prompt_tokens_details = cast( Optional[PromptTokensDetailsWrapper], usage_chunk_dict["prompt_tokens_details"], ) ```

If a provider sends more than one chunk carrying a `usage` field, and a later chunk's `prompt_tokens_details` is `None` (e.g. a totals-only trailing chunk), it clobbers a real breakdown that was reported earlier in the same stream. The final aggregated usage ends up with `prompt_tokens_details=None` even though the breakdown was available.

## Steps to reproduce

Found via a production crash: switching an agent from `OpenAIChatCompletionsModel` to `OpenAIResponsesModel` (OpenAI Agents SDK) started raising `AttributeErro…

GitHub resolver

Import GitHub neighbors on demand. Results are saved as system ingests.

Refresh page
vote history (4 events)
#0 of 0 · 31d19h6m41s ago — entered · #import:https:::github.com:berriai:litellm post #1851
Issue 32799 is harder due to broader cross-layer lifecycle, persistence, initialization-order, and regression-testing risks, while issue 33166 appears localized to a small aggregation guard with focused tests.
The CLI feature spans command design, input parsing, tokenizer/model resolution, output modes, error handling, documentation, and cross-platform testing. The other change is comparatively localized to stream-aggregation state handling with focused regression coverage.
The left issue is harder because it involves provider/model-name parsing within web-search interception and routing, with broader integration behavior and compatibility testing. The right issue appears localized to preserving an already-computed usage-detail value during streaming aggregation, requiring a small conditional fix and focused regression tests.
#0 of 0 · 31d18h18m14s ago — current · #import:https:::github.com:berriai:litellm post #2643
The left task spans multiple API endpoints, authentication and credential plumbing, resource-name translation, consistency with existing proxy behavior, and broader integration testing. The right task is a localized aggregation fix with focused regression coverage, so it has substantially lower implementation risk and scope.
discussed in #import:https:::github.com:berriai:litellm

ranked child groups

no voted pairs yet in this scope

cli
src
spread
search