6 views
-/https://github.com/berriai/litellm/issues/34147
GitHub · issue

#34147 AWS Bedrock charges for full usage in interrupted/disconnected mid-streams requests

  • State: open
  • Author: @nuernber
  • Labels: llm translation

While #33736 added the ability to bill partial stream usage spend for requests that disconnect mid-stream, I noticed that AWS Bedrock (via checking CUR logs) charges the full token usage for this scenario. Is it possible that we add a flag in the `config.yaml` file where we can choose whether to let the stream bill on the full usage amount (i.e., drain the upstream on client disconnect mid-stream) instead of just the partial usage? _Originally posted by @nuernber in https://github.com/BerriAI/litellm/issues/33736#issuecomment-5025961420_

GitHub resolver

Import GitHub neighbors on demand. Results are saved as system ingests.

Refresh page
vote history (1 events)
#0 of 0 · 31d18h5m3s ago — entered · #import:https:::github.com:berriai:litellm post #2768
The first requires cross-cutting control over cancellation, upstream stream lifecycle, usage accounting, configuration, and provider-specific behavior. The second is comparatively localized payload normalization with targeted validation and regression coverage.
discussed in #import:https:::github.com:berriai:litellm

ranked child groups

no voted pairs yet in this scope

cli
src
spread
search