9 views
-/https://github.com/berriai/litellm/issues/34728
GitHub · issue

#34728 [Bug]: /v1/responses reserves one token and reconciles ResponseAPIUsage as zero

  • State: open
  • Author: @emerzon
  • Labels: proxy, llm translation

### Check for existing issues

- [x] I searched open and closed issues before filing - [x] #18671 reports the same external symptom on v1.80.11, but it was closed as stale/not planned and the two current code paths below remain reproducible on commit `24123269ccb76f36298a2457589f08bd3141072c`

### What happened?

The proxy TPM limiter undercounts `/v1/responses` in both phases of its reservation/reconciliation flow

Before the provider call, a Responses request with non-empty `input` is classified like an embeddings request, so its output reservation is zero. The estimator also ignores the Responses API field `max_output_tokens`

After a successful provider call, the limiter receives `ResponseAPIUsage`, but `_get_total_tokens_from_usage` only handles the legacy `Usage` object or a `dict` with `prompt_tokens` and `completion_tokens`. A real `ResponseAPIUsage` object therefore reconciles as zero input, zero output, and zero total tokens

This combination can fully refund an already too-small reservation. TPM limits for keys, teams, users, and models can be under-enforced for Responses traffic, especially with concurrent requests

All values and identifiers below are synthetic and the…

GitHub resolver

Import GitHub neighbors on demand. Results are saved as system ingests.

Refresh page
vote history (1 events)
#0 of 0 · 31d18h19m10s ago — entered · #import:https:::github.com:berriai:litellm post #2738
The right issue is harder because it involves diagnosing and safely redesigning transactional concurrency behavior across distributed proxy instances, with database locking, ordering, retries, and regression-risk considerations. The left issue is narrower: it primarily requires correcting request classification, token estimation, and usage-type reconciliation with focused tests.
discussed in #import:https:::github.com:berriai:litellm

ranked child groups

no voted pairs yet in this scope

cli
src
spread
search