8 views
-/https://github.com/berriai/litellm/issues/34497
GitHub · issue

#34497 [Bug] Bedrock Invoke streaming drops cacheRead/cacheWrite token counts → cache-heavy traffic billed as fresh input

  • State: open
  • Author: @yryzhan-vitech
  • Labels: bug, proxy, llm translation

### Check for existing issues

- [x] I have searched the existing issues and checked that my issue is not a duplicate.

### What happened?

For Anthropic Claude models called over the Bedrock Invoke streaming path (bedrock/invoke/…), the final chunk carries token usage in amazon-bedrock-invocationMetrics. LiteLLM maps inputTokenCount and outputTokenCount from that block into usage, but silently drops cacheReadInputTokenCount and cacheWriteInputTokenCount.

As a result, prompt-cache usage never reaches usage.cache_read_input_tokens / usage.cache_creation_input_tokens, and the cost calculator bills cache-heavy traffic (e.g. a client with a large cached system prompt) as if every request were fresh input — a 4–7× over-report of input cost and a total under-count of cached tokens.

Root cause is in two places on the Invoke path:

1. Decoder — AmazonAnthropicClaudeMessagesStreamDecoder._chunk_parser in litellm/llms/bedrock/messages/invoke_transformations/anthropic_claude3_transformation.py. The amazon-bedrock-invocationMetrics mapping reads only inputTokenCount and outputTokenCount; the two cache fields are never read. This is distinct from _promote_message_stop_usage (LIT-2…

GitHub resolver

Import GitHub neighbors on demand. Results are saved as system ingests.

Refresh page
vote history (3 events)
#0 of 0 · 31d18h1m14s ago — entered · #import:https:::github.com:berriai:litellm post #2845
Issue 34497 is harder because it spans provider-specific streaming decoding, normalized usage fields, billing calculations, and regression coverage. Issue 27591 is comparatively localized input normalization with platform and parser compatibility considerations.
34241 is harder because it requires a broad repository-wide licensing and feature-boundary audit, coordinated refactoring across many modules, and careful validation of legal and architectural consequences. 34497 is comparatively localized to a provider-specific usage-decoding path with focused regression tests.
#0 of 0 · 31d17h19m54s ago — current · #import:https:::github.com:berriai:litellm post #3539
Nested access-group support is substantially harder: it affects authorization data models, recursive resolution, mutation propagation, cycle and consistency handling, compatibility, and broad API/test coverage. The other issue is comparatively localized to provider-response translation, usage propagation, billing validation, and targeted regression tests.
discussed in #import:https:::github.com:berriai:litellm

ranked child groups

no voted pairs yet in this scope

cli
src
spread
search