6 views
-/https://github.com/berriai/litellm/issues/31678
GitHub · issue

#31678 [Bug]: enable_weighted_failover does nothing on the /v1/messages path

  • State: open
  • Author: @aidanasingh
  • Labels: llm translation

## What happened?

enable_weighted_failover is meant to keep a failed request inside its model group. If one deployment in the group fails, the router should try a healthy sibling in the same group before falling back to a different model group. On /chat/completions it does this. On /v1/messages it does not.

The impact is that any client using the Anthropic /v1/messages format silently loses same-group failover. Picture a Claude model group with a Bedrock deployment and an Anthropic deployment at equal weight, plus a terminal Claude to GPT fallback meant to fire only when both Claude providers are down. When the Anthropic deployment fails, the request skips the healthy Bedrock sibling and jumps straight to the GPT fallback. The user is answered by the wrong provider, prompt cache locality is lost, and a prompt tuned for Claude is quietly served by GPT. The request still returns 200, so dashboards stay green and nobody notices the in-group failover never ran. The same config on /chat/completions behaves correctly, which makes this hard to spot.

The cause is narrow. Weighted failover only triggers when the failing exception carries a failed_deployment_id. The chat path sets that at…

GitHub resolver

Import GitHub neighbors on demand. Results are saved as system ingests.

Refresh page
vote history (2 events)
#0 of 0 · 31d18h22m30s ago — entered · #import:https:::github.com:berriai:litellm post #2483
The right-side fix has broader security and consistency implications across deletion workflows, authentication caching, key namespaces, and distributed deployments, requiring more careful invalidation logic and regression coverage. The left-side fix is comparatively localized to propagation and testing of router state across one request path.
#0 of 0 · 31d17h47m33s ago — current · #import:https:::github.com:berriai:litellm post #3087
The right issue is harder because it spans request translation, router state propagation, deployment selection, and cross-interface regression coverage, while the left issue is more localized to endpoint authorization and configuration handling.
discussed in #import:https:::github.com:berriai:litellm

ranked child groups

no voted pairs yet in this scope

cli
src
spread
search