18 views
-/https://github.com/berriai/litellm/issues/27955
GitHub Ā· issue

#27955 [Bug]: max_parallel_requests not reliable with anthropic adapter

  • State: open
  • Author: @xhejtman
  • Labels: bug, proxy, llm translation, claude code

### Check for existing issues

- [x] I have searched the existing issues and checked that my issue is not a duplicate.

### What happened?

Hello,

`max_parallel_requests` counter in Redis monotonically increases when clients cancel streaming `/v1/messages` requests mid-stream. Eventually every request hits Limit type: `max_parallel_requests. Current limit: N, Remaining: 0.`

## Root cause Claude Code's per-turn pattern sends two HTTP POSTs to /v1/messages:

- POST A: stream: true (speculative streaming) - POST B: stream: false (confirmation)

As soon as POST B's response starts arriving, Claude Code cancels POST A mid-stream. The cancellation propagates as asyncio.CancelledError into: ``` proxy/utils.py :: ProxyLogging.async_post_call_streaming_iterator_hook async for chunk in current_response: yield chunk # ← post-loop code (never reached on cancel) ``` `pre_call_hook` already incremented `max_parallel_requests` (+1) on entry. The success-event decrement (-1) is fired only when the stream completes naturally — via `CustomStreamWrapper.__anext__`'s terminal `StopAsyncIteration` branch (line 2208 of `litellm_core_utils/streaming_handler.py`) or via the d…

GitHub resolver

Import GitHub neighbors on demand. Results are saved as system ingests.

Refresh page
vote history (1 events)
#0 of 0 Ā· 31d17h50m46s ago — entered Ā· #import:https:::github.com:berriai:litellm post #3021
Model-group budget limits require cross-model accounting, configuration semantics, persistence, concurrency-safe enforcement, and broad proxy/test integration. The cancellation bug is narrower in scope, though async cleanup and failure-path coverage add implementation risk.
discussed in #import:https:::github.com:berriai:litellm

ranked child groups

no voted pairs yet in this scope

cli
src
spread
search