6 views
-/https://github.com/berriai/litellm/issues/32574
GitHub · issue

#32574 [Bug]: Router cooldown can be skipped across retries/fallbacks because failure logging is deduplicated on the shared Logging object

  • State: open
  • Author: @Cheny-chui
  • Labels: bug, SDK

### Check for existing issues

- [x] I have searched the existing issues and checked that my issue is not a duplicate.

### What happened?

## Summary

When using `Router` with multiple deployments under the same model group, a failed deployment may not be put into cooldown on later retry/fallback attempts.

From reading the code, it looks like Router cooldown depends on the sync failure logging callback:

- `Logging.failure_handler(...)` - `litellm.failure_callback` - `Router.deployment_callback_on_failure(...)` - `_set_cooldown_deployments(...)`

However, `Logging.failure_handler()` deduplicates `sync_failure` events using `has_logged_sync_failure` on `logging_obj.model_call_details`.

In Router retry/fallback flows, the same `litellm_logging_obj` appears to be reused across attempts. This means:

1. First provider/deployment failure runs `failure_handler()` normally. 2. `has_logged_sync_failure` is set to `True`. 3. Later retry/fallback provider failures call `failure_handler()` again, but the handler returns early due to deduplication. 4. `Router.deployment_callback_on_failure()` is not called for those later failed deployments. 5. Those failed deployments are not added to cool…

GitHub resolver

Import GitHub neighbors on demand. Results are saved as system ingests.

Refresh page
vote history (3 events)
#0 of 0 · 31d17h58m23s ago — entered · #import:https:::github.com:berriai:litellm post #2885
32574 is harder because it involves coordinating retry/fallback state, shared logging lifecycle, callback semantics, and regression coverage across multiple execution paths; 29397 is comparatively localized to persisted-state reconstruction and validation.
#32574 is harder because it requires tracing and safely changing shared logging state across router retries, fallbacks, callbacks, and cooldown behavior, with broader regression and concurrency risk. #27532 is comparatively localized to request-header inference and provider-specific validation.
#0 of 0 · 31d17h17m48s ago — current · #import:https:::github.com:berriai:litellm post #3556
The left issue requires diagnosing and safely redesigning concurrent transactional database updates, with migration and cross-instance race-condition risk. The right issue is narrower, centered on callback state and retry-flow behavior, with more localized code changes and testing.
discussed in #import:https:::github.com:berriai:litellm

ranked child groups

no voted pairs yet in this scope

cli
src
spread
search