6 views
-/https://github.com/berriai/litellm/issues/30816
GitHub · issue

#30816 [Feature]: Report actual usage cost to SSE clients in streaming mode

  • State: open
  • Author: @awschmeder
  • Labels: llm translation

## Summary

When responses are streamed (`stream: true`), there is currently no standard way for a client to learn the proxy's authoritative cost for the completed request. The proxy already computes this cost server-side; it is just not delivered back to the client. As a result, clients tend to re-implement their own cost estimation from raw token counts, which cannot account for proxy-side modifiers (discounts, margins, MCP tool usage, prompt caching) that only the proxy knows about.

This issue proposes surfacing the proxy's already-computed cost to the client at stream end. Because the value is calculated synchronously before logging happens, this should be a relatively low-risk addition.

---

## Background

A streaming request is delivered as a sequence of SSE events ending with `data: [DONE]`. Server-side, the proxy calculates the true cost accounting for:

- Model-specific token pricing (input, output, cached, reasoning) - Prompt caching discounts (Anthropic, OpenAI, Bedrock) - MCP tool usage costs - Operator-configured discounts (`cost_discount_config`) and margins (`cost_margin_config`) - Provider-specific uplifts (geo routing, speed tiers) - Built-in tool costs (web sear…

GitHub resolver

Import GitHub neighbors on demand. Results are saved as system ingests.

Refresh page
vote history (3 events)
#0 of 0 · 31d19h1m56s ago — entered · #import:https:::github.com:berriai:litellm post #1937
Streaming cost reporting spans response assembly, SSE protocol semantics, provider compatibility, and end-to-end accounting tests, creating broader integration and backward-compatibility risk. The regional model-key defect is more localized to persistence, pricing lookup, and dashboard handling.
30816 spans streaming lifecycle, SSE contract design, client compatibility, and end-to-end cost propagation/testing, creating broader integration risk. 27942 is comparatively localized to model identity propagation in an existing spend-logging path.
#0 of 0 · 31d17h53m1s ago — current · #import:https:::github.com:berriai:litellm post #3091
The right-hand change is harder because it crosses the streaming response pipeline, requires preserving compatibility across event formats and clients, and needs broader integration and regression testing. The left-hand change is comparatively isolated to CLI argument handling, existing pricing utilities, and command-level tests.
discussed in #import:https:::github.com:berriai:litellm

ranked child groups

no voted pairs yet in this scope

cli
src
spread
search