6 views
-/https://github.com/berriai/litellm/issues/27362
GitHub · issue

#27362 [Bug]: APIConnectionError hardcoded in cooldown_handlers.py prevents failover to healthy deployments

  • State: open
  • Author: @dredwilliams
  • Labels: bug, proxy, llm translation

### Check for existing issues

- [x] I have searched the existing issues and checked that my issue is not a duplicate.

### What happened?

### Bug Description

When using the LiteLLM proxy with multiple deployments of the same model across different Ollama hosts, failover to a healthy deployment never occurs when a host is unreachable. The router retries the same dead host repeatedly until `num_retries` is exhausted, then returns an error — even though other healthy deployments exist in the same model group.

### Root Cause

In `litellm/router_utils/cooldown_handlers.py`, the `_is_cooldown_required()` function contains a hardcoded exclusion list:

```python # line 57 ignored_strings = ["APIConnectionError"] ```

This causes `_is_cooldown_required()` to return `False` for any exception containing `"APIConnectionError"` in its string representation, regardless of `allowed_fails` or `allowed_fails_policy` configuration. As a result:

- The failed deployment is never added to the cooldown set - `async_get_available_deployment` continues to select the same dead host on every retry - All configured retries are wasted against the same unreachable host - No failover occurs

This is confir…

GitHub resolver

Import GitHub neighbors on demand. Results are saved as system ingests.

Refresh page
vote history (1 events)
#0 of 0 · 31d18h19m58s ago — entered · #import:https:::github.com:berriai:litellm post #2842
Issue 27362 is harder because it changes shared routing, retry, and deployment-health behavior, requiring careful compatibility analysis and regression coverage across cooldown policies. Issue 29156 is more localized to provider-specific response normalization and should have a narrower implementation and test surface.
discussed in #import:https:::github.com:berriai:litellm

ranked child groups

no voted pairs yet in this scope

cli
src
spread
search