6 views
-/https://github.com/berriai/litellm/issues/32865
GitHub · issue

#32865 [Bug]: /v1/responses TPM rate limiter massively over-estimates tokens with base64 image input

  • State: open
  • Author: @chanckben
  • Labels: bug, proxy, llm translation

### Check for existing issues

- [x] I have searched the existing issues and checked that my issue is not a duplicate.

### What happened?

### Description of Issue

When these 2 criteria are met: 1. TPM rate limit on key / team set to a reasonable value, around 50k, and 2. A request is made to `/v1/responses` endpoint with a base64-encoded image in the request, with input tokens significantly less than the TPM limit

a 429 response will be returned to the user, with the error message: `Rate limit exceeded for api_key: <hashed-api-key>. Limit type: tokens. Current limit: 50000, Remaining: 50000. Limit resets at: 2026-07-11 01:40:13 UTC`.

Pod logs show: `parallel_request_limiter_v3.py:436 - TPM reservation estimate: input=633912, max_tokens=0 (explicit=False), total=633912`

When the TPM rate limit is removed or increased to a very high value, the request succeeds and the usage object in the corresponding spend log shows that there were only ~1k input tokens, which could not have burst the 50k TPM limit: ``` "usage_object": { "total_tokens": 1108, "prompt_tokens": 1049, "completion_tokens": 59, "prompt_tokens_details": { "text_tokens": null, "audio_t…

GitHub resolver

Import GitHub neighbors on demand. Results are saved as system ingests.

Refresh page
vote history (2 events)
#0 of 0 · 31d19h8m56s ago — entered · #import:https:::github.com:berriai:litellm post #1734
The right issue spans authorization-sensitive routing, multiple synchronous code paths, and cache behavior, requiring careful regression testing across router and cache layers. The left issue is more localized to token accounting and request parsing, with a narrower validation surface.
#0 of 0 · 31d18h34m53s ago — current · #import:https:::github.com:berriai:litellm post #2292
The left issue is substantially harder because it implies a cross-cutting authentication abstraction, provider integrations, credential lifecycle handling, configuration semantics, security review, and compatibility work. The right issue is comparatively localized to request token accounting and rate-limiter behavior, with focused reproduction, correction, and regression tests.
discussed in #import:https:::github.com:berriai:litellm

ranked child groups

no voted pairs yet in this scope

cli
src
spread
search