7 views
-/https://github.com/berriai/litellm/issues/32308
GitHub · issue

#32308 [Bug]: enable_weighted_failover loses to a stale affinity pin and skips the healthy same-group deployment

  • State: open
  • Author: @aidanasingh
  • Labels: llm translation

## What happened?

enable_weighted_failover is meant to retry a failed request inside its model group before falling back to a different group. When the model group also has deployment affinity enabled (session_affinity or another DeploymentAffinityCheck-based check), the retry can lose to a stale affinity pin and fall through to cross-group fallback even though a healthy sibling deployment exists.

Picture a model group with two deployments, A and B, weighted failover on, and session affinity on. A session's first request lands on A and pins there. A then fails. Weighted failover correctly works out that B is healthy and not on cooldown, so it should retry on B. But before the retry actually picks a deployment, the affinity callback runs first, sees the session is still pinned to A, and narrows the candidate list down to just A. Only after that does the exclusion filter drop A, leaving nothing, so the retry errors out and weighted failover gives up. The caller then falls through to the cross-group fallback, or fails outright if none is configured, instead of landing on the healthy B that was right there.

This defeats the purpose of weighted failover for any model group that also …

GitHub resolver

Import GitHub neighbors on demand. Results are saved as system ingests.

Refresh page
vote history (5 events)
#0 of 0 · 31d19h5m43s ago — entered · #import:https:::github.com:berriai:litellm post #1782
32308 requires coordinating multiple deployment-selection mechanisms, preserving retry semantics, and validating stateful edge cases across routing paths; 28927 is comparatively localized to MCP outcome classification and observability-field handling. The former has greater cross-cutting regression risk.
The left issue is harder because it introduces a new key-level authorization concept spanning key schemas, request validation, tag precedence, routing integration, backward compatibility, and comprehensive security-focused tests. The right issue is a narrower behavioral defect in the interaction between two existing selection mechanisms, likely requiring targeted control-flow changes and regression coverage.
Issue 32308 requires coordinated changes across routing, affinity, retry, and fallback behavior, with substantial regression testing for interacting policies. Issue 25210 is a narrowly scoped package-metadata correction.
#35563 is harder because it requires changing identifier and persistence semantics across logging, database constraints, batching, backward compatibility, and audit correctness. #32308 is comparatively localized to coordinating two existing routing policies, with focused behavioral tests.
#0 of 0 · 31d18h52m31s ago — current · #import:https:::github.com:berriai:litellm post #2007
31609 is harder because it requires tracing caller identity through MCP discovery, credential resolution, and subsequent tool invocation while preserving authorization semantics and compatibility across multiple authentication modes. It also carries higher regression risk and needs end-to-end coverage. 32308 is comparatively localized to retry candidate selection and affinity ordering, with focused routing tests likely sufficient.
discussed in #import:https:::github.com:berriai:litellm

ranked child groups

no voted pairs yet in this scope

cli
src
spread
search