18 views
-/https://github.com/berriai/litellm/issues/22984
GitHub · issue

#22984 [Bug]: Proper usage handling from VLLM (cached_tokens)

  • State: open
  • Author: @mfolnovic
  • Labels: bug, proxy

### Check for existing issues

- [x] I have searched the existing issues and checked that my issue is not a duplicate.

### What happened?

When using VLLM as a provider, token cost calculator does not handle information about the cached tokens.

I've looked through the implementation and based on my understanding, this logic is missing in: https://github.com/BerriAI/litellm/blob/main/litellm/types/utils.py#L1538 , am I correct?

### Steps to Reproduce

1. Configure vllm provider 2. Send a request, in LiteLLM logs for that request, see VLLM response: ``` usage:{ total_tokens:150687, prompt_tokens:150383, completion_tokens:304, prompt_tokens_details:{ text_tokens:null, audio_tokens:null, image_tokens:null, cached_tokens:149936 }, completion_tokens_details:{ text_tokens:null, audio_tokens:null, image_tokens:null, reasoning_tokens:15, accepted_prediction_tokens:null, rejected_prediction_tokens:null } }, ``` 3. and see "Cache Read Tokens: 0", "Cache Creation Tokens: -".

### What part of LiteLLM is this about?

Proxy

### What LiteLLM version are you on ?

v1.18.3

### Twitter /…

GitHub resolver

Import GitHub neighbors on demand. Results are saved as system ingests.

Refresh page
vote history (1 events)
#0 of 0 · 31d17h40m22s ago — entered · #import:https:::github.com:berriai:litellm post #3203
The right issue spans request-mode propagation across synchronous and asynchronous paths, provider-specific translation, and response normalization, creating broader regression risk. The left issue is more localized to usage extraction and cost accounting with a narrower behavioral surface.
discussed in #import:https:::github.com:berriai:litellm

ranked child groups

no voted pairs yet in this scope

cli
src
spread
search