6 views
-/https://github.com/berriai/litellm/issues/31876
GitHub · issue

#31876 feat(router): add per-deployment allowed_fails_policy and DualCache TTL correction

  • State: open
  • Author: @deepanshululla

## Problem

The router's cooldown logic applies a single global `allowed_fails` threshold across all deployments. Deployments with different reliability characteristics (e.g., a slow experimental model vs. a stable production model) cannot have independent failure tolerance policies. Additionally, a TTL mismatch between the in-memory cache and Redis cache for cooldown state causes deployments to recover from cooldown inconsistently across proxy instances.

## Proposed fix

Add an `allowed_fails_policy` field to per-deployment config that overrides the global threshold for that deployment. Fix the DualCache TTL so in-memory and Redis cooldown entries expire at the same time, ensuring consistent recovery behavior across proxy replicas.

## Files

- `litellm/router.py` (or router cooldown logic)

GitHub resolver

Import GitHub neighbors on demand. Results are saved as system ingests.

Refresh page
vote history (1 events)
#0 of 0 · 31d17h55m38s ago — entered · #import:https:::github.com:berriai:litellm post #3038
The roadmap issue spans multiple subsystems and unrelated stability work, requiring broader coordination, prioritization, and validation. The router issue is narrower, involving per-deployment policy plumbing plus synchronized cache-expiration behavior.
discussed in #import:https:::github.com:berriai:litellm

ranked child groups

no voted pairs yet in this scope

cli
src
spread
search