8 views
-/https://github.com/berriai/litellm/issues/27470
GitHub · issue

#27470 Cooldown TTL doesn't distinguish 429-rate-limit from 429-quota-exhausted (question / possible bug)

  • State: open
  • Author: @hernaninverso
  • Labels: llm translation

hey, posting this as a question first since i may have misread the code. happy to be told i'm wrong.

reading `litellm/router.py` at the current main, the `Router` class uses `cooldown_time` (a single TTL) and `allowed_fails` (a count). when a deployment exceeds `allowed_fails`, it goes into `CooldownCache` for `cooldown_time`. operators can configure both per deployment via `cooldown_time` kwarg or environment.

the failure mode i hit and that prompted this question: a free-tier provider (groq, in this case) returned 429 with body containing "you exceeded your monthly quota". my router (a homegrown thing modeled on this pattern) treated it the same as a transient 429-rate-limit, retried every 60s for hours.

reading the code i couldn't find a path where the body of a 429 is inspected to disambiguate "wait 60 seconds" from "wait until the period rolls over". `cooldown_time` is uniform.

questions:

1. is there a config knob i missed that does this distinction by body keyword? 2. if not, is "user should configure `cooldown_time = 86400` per provider that has monthly quota" the intended workaround? 3. would you accept a PR that adds an optional body-keyword classifier (off by default…

GitHub resolver

Import GitHub neighbors on demand. Results are saved as system ingests.

Refresh page
vote history (1 events)
#0 of 0 · 31d18h21m54s ago — entered · #import:https:::github.com:berriai:litellm post #2584
27470 requires broader behavioral design, configuration semantics, provider-specific classification, and regression coverage, while 25757 is a more localized identifier-handling fix across an existing request path.
discussed in #import:https:::github.com:berriai:litellm

ranked child groups

no voted pairs yet in this scope

cli
src
spread
search