6 views
-/https://github.com/berriai/litellm/issues/33967
GitHub · issue

#33967 [Bug]: custom_openai handler does not forward prompt_tokens_details to client response

  • State: open
  • Author: @ihannsen
  • Labels: llm translation

## Summary

When using a custom OpenAI-compatible provider (via `custom_openai` handler) that returns `prompt_tokens_details.cached_tokens` in the response, LiteLLM does not forward this field to the client. The `usage` object in the response only contains `completion_tokens`, `prompt_tokens`, and `total_tokens` — `prompt_tokens_details` is completely missing.

## The Bug

1. The upstream provider returns `prompt_tokens_details: { cached_tokens: 1088 }` on cache hits 2. LiteLLM's response to the client only contains top-level token fields 3. `prompt_tokens_details` is not forwarded, even though the provider returns it

This breaks: - Cache hit detection in clients - Cost tracking accuracy (cache vs. non-cache pricing) - The fixes in #19681 and #27191 which rely on `prompt_tokens_details.cached_tokens` being present

## Evidence

**Direct API call (without LiteLLM) — Request 2 (cache hit):** ```json { "usage": { "completion_tokens": 50, "prompt_tokens": 1095, "total_tokens": 1145, "prompt_tokens_details": { "cached_tokens": 1088 } } } ```

**Same request through LiteLLM (v1.93.0) — Request 2:** ```json { "usage": { "completion_tokens": 50, "pr…

GitHub resolver

Import GitHub neighbors on demand. Results are saved as system ingests.

Refresh page
vote history (7 events)
#0 of 0 · 31d19h19m48s ago — entered · #import:https:::github.com:berriai:litellm post #1639
The right issue is harder because it spans startup configuration loading, database-backed state synchronization, lifecycle ordering, and regression coverage across persistence modes, while the left issue is comparatively localized to response-field propagation and serialization.
Multi-path proxy routing and identifier preparation span several request flows and provider boundaries, creating broader regression risk; the other change is a localized response-schema propagation fix.
The right issue is harder because it requires tracing and correcting response normalization in the custom provider path, preserving compatibility, and adding regression coverage; the left issue is a localized documentation correction.
#33967 requires tracing response normalization and serialization across the custom-provider path, preserving compatibility, and adding regression coverage; #27325 is a contained metadata/configuration update with limited implementation risk.
The left issue spans multiple cross-cutting workflows, identity integrations, notification delivery, security-sensitive credential handling, configuration gating, and UI/API behavior. The right issue is comparatively localized to response-schema propagation in one handler, with focused code changes and tests.
The right-hand task requires tracing provider-response normalization, preserving nested usage metadata across typed models and serialization layers, and adding regression coverage. The left-hand item appears primarily diagnostic or configuration-related, with little evidence of a required code change.
#0 of 0 · 31d18h37m8s ago — current · #import:https:::github.com:berriai:litellm post #2345
The left issue requires debugging stateful realtime event sequencing, coordinating proxy behavior with backend VAD semantics, and validating timing-sensitive regressions across streaming scenarios. The right issue is comparatively localized to response schema propagation and focused unit or integration coverage.
discussed in #import:https:::github.com:berriai:litellm

ranked child groups

no voted pairs yet in this scope

cli
src
spread
search