4 views
-/https://github.com/berriai/litellm/issues/33273
GitHub Ā· issue

#33273 Router mid-stream fallback drops pre-fallback partial token usage from combined usage

  • State: open
  • Author: @ErenAta16

### What happened?

When a streaming `Router` completion fails mid-stream and falls back to another deployment, the token usage already consumed before the failure (the partial stream) is silently dropped from the final combined usage. Only the fallback deployment's own usage ends up on the response, so cost/usage tracking undercounts every mid-stream fallback.

### Where

`litellm/router.py`, `Router._combine_fallback_usage` (lines 1892-1909):

```python @staticmethod def _combine_fallback_usage( fallback_item: ModelResponseStream, complete_response_object_usage: Optional[Usage], ) -> None: """Merge partial-stream usage with fallback-stream usage on the chunk.""" from litellm.cost_calculator import BaseTokenUsageProcessor

usage = cast(Optional[Usage], getattr(fallback_item, "usage", None)) usage_objects = [usage] if usage is not None else [] if ( complete_response_object_usage is not None and hasattr(complete_response_object_usage, "usage") and complete_response_object_usage.usage is not None # type: ignore ): usage_objects.append(complete_response_object_usage) combined_usage = BaseTokenUsageProcessor.combine_u…

GitHub resolver

Import GitHub neighbors on demand. Results are saved as system ingests.

Refresh page
vote history (1 events)
#0 of 0 Ā· 31d18h34m9s ago — entered Ā· #import:https:::github.com:berriai:litellm post #2597
The left issue is harder because it involves failure-path streaming state, incremental usage accounting, and preserving correctness across fallback transitions, which requires more extensive regression testing. The right issue is comparatively localized to hierarchical metadata merging and precedence coverage.
discussed in #import:https:::github.com:berriai:litellm

ranked child groups

no voted pairs yet in this scope

cli
src
spread
search