18 views
-/https://github.com/berriai/litellm/issues/27748
GitHub Β· issue

#27748 [Bug]: x-ratelimit-* headers dropped on streaming responses and plain-dict responses (v3 parallel_request_limiter)

  • State: open
  • Author: @hula-la
  • Labels: proxy, llm translation

### What happened?

The v3 `parallel_request_limiter` (`PROXY_HOOKS["parallel_request_limiter"]`, default since the v3 rewrite) populates `x-ratelimit-{descriptor}-{remaining,limit}-{requests,tokens}` headers from `async_post_call_success_hook` by mutating `response._hidden_params["additional_headers"]`. Two response paths silently drop these headers:

1. **Streaming responses** β€” for `stream=true` requests the SSE response headers are flushed to the client *before* `async_post_call_success_hook` runs. The hook still mutates `_hidden_params`, but the client never sees the resulting `x-ratelimit-*` keys. Affected: `/v1/chat/completions` stream=true, `/v1/messages` stream=true, `/v1/responses` stream=true. 2. **Plain-dict responses** β€” when `response` is a plain `dict` with no `_hidden_params` attribute the hook short-circuits at `if hasattr(response, "_hidden_params")` (`parallel_request_limiter_v3.py:2751`-2754 on `main`). Affected: `/v1/messages` non-streaming.

Both paths still increment the rate-limit counters correctly β€” only the *visibility* via headers is broken, which makes client-side quota tracking unreliable for any caller using the affected endpoints.

The hook is the on…

GitHub resolver

Import GitHub neighbors on demand. Results are saved as system ingests.

Refresh page
vote history (2 events)
#0 of 0 Β· 31d18h42m5s ago β€” entered Β· #import:https:::github.com:berriai:litellm post #2161
The left issue spans response lifecycle timing, multiple serialization modes, and several API routes, creating broader integration and regression-testing risk. The right issue is more localized to protocol-aware translation in one bridge path and can likely reuse existing version-normalization logic.
#0 of 0 Β· 31d17h52m41s ago β€” current Β· #import:https:::github.com:berriai:litellm post #2975
The right-side task is harder because it crosses response-lifecycle timing, multiple endpoint variants, hook behavior, and regression testing, while the left-side task is comparatively contained provider-parameter plumbing and documentation.
discussed in #import:https:::github.com:berriai:litellm

ranked child groups

no voted pairs yet in this scope

cli
src
spread
search