8 views
-/https://github.com/berriai/litellm/issues/23749
GitHub · issue

#23749 [Bug]: dynamic_rate_limiter_v3 do not trigger fallback

  • State: open
  • Author: @MaximeBOUDIER
  • Labels: bug, proxy, llm translation

### Check for existing issues

- [x] I have searched the existing issues and checked that my issue is not a duplicate.

### What happened?

When using [dynamic_rate_limiter_v3](https://docs.litellm.ai/docs/proxy/dynamic_rate_limit) for priority-based rate limiting alongside configured fallbacks, rate-limited requests return 429 to the client instead of falling back to the configured fallback model. This is because the rate limiter raises `HTTPException(429)` in `async_pre_call_hook`, which executes before the router's fallback logic.

My goal is to: - Reserve capacity for high-priority requests (key tagged "priority":"high") - Automatically redirect low-priority requests to Fallback when model TPM is saturated.

This example setup is designed for a PTU reservation scenario with Pay-As-You-Go fallback on Azure OpenAI / vertex ai

### Current Setup

**Models:** ```yaml model_list: - model_name: gpt-4.1-ptu-20250414 litellm_params: model: azure/gpt-4.1-ptu api_base: ... tpm: 40000

- model_name: gpt-4.1-20250414 litellm_params: model: azure/gpt-4.1 api_base: ... ```

**LiteLLM Settings:** ```yaml litellm_settings: callbacks: ["dynamic_rate_…

GitHub resolver

Import GitHub neighbors on demand. Results are saved as system ingests.

Refresh page
vote history (3 events)
#0 of 0 · 31d18h57m56s ago — entered · #import:https:::github.com:berriai:litellm post #1914
The left issue is harder because it requires changing cross-cutting request-control flow between middleware hooks and routing while preserving prioritization, retries, and fallback semantics. The right issue is comparatively localized to identifier handling and persistence, with focused database and regression tests.
The right issue is harder because it requires coordinating rate-limit hook behavior with router fallback control flow, preserving priority and failure semantics across providers, and adding broader integration coverage. The left issue is comparatively localized to an Azure-specific request translation guard with focused regression tests.
#0 of 0 · 31d18h8m5s ago — current · #import:https:::github.com:berriai:litellm post #2732
The harder task crosses shared proxy control flow, exception semantics, fallback selection, and async regression coverage, creating greater compatibility and state-management risk. The easier task is comparatively localized to provider-specific request normalization with focused adapter tests.
discussed in #import:https:::github.com:berriai:litellm

ranked child groups

no voted pairs yet in this scope

cli
src
spread
search