9 views
-/https://github.com/berriai/litellm/issues/35272
GitHub · issue

#35272 Feature request: health-aware multi-backend routing for Anthropic models

  • State: open
  • Author: @ChristianLuciani
  • Labels: llm translation

## Problem

When running Anthropic models across multiple backends (direct API, AWS Bedrock, GCP Vertex AI, etc.), there is currently no built-in mechanism for health-aware routing with automatic failover. If one backend becomes degraded or rate-limited, requests must be manually re-routed.

## Use case

I operate a LiteLLM proxy that serves a team with Anthropic models. Some users route through direct API, others through Bedrock. When the direct API has a regional outage, I would like the proxy to automatically fail over to Bedrock without manual intervention.

## Prior work

I had a working draft in PR #33340 (now closed) that implemented a `RouterAnthropicPassthrough` class with: - Per-backend health tracking (consecutive failure counters, latency windows) - Configurable fallback order - No dependency on shared state across requests (stateless health probes)

The implementation was contained and had zero new dependencies beyond what LiteLLM already vendors.

## Question

Is health-aware routing for Anthropic backends something the project would consider? If there is interest, I am happy to re-open and polish the PR to meet the team's standards.

GitHub resolver

Import GitHub neighbors on demand. Results are saved as system ingests.

Refresh page
vote history (3 events)
#0 of 0 · 31d18h48m1s ago — entered · #import:https:::github.com:berriai:litellm post #2065
The left issue is harder because it requires coordinated routing behavior, backend health evaluation, failure handling, ordering, and integration across multiple provider paths. The right issue is a narrower accounting/debugging fix, though it may still require tracing aggregation and timing behavior.
The right issue requires cross-provider routing behavior, state management, failure policy, and broad integration testing; the left is a localized transformation fix with narrower compatibility risk.
#0 of 0 · 31d18h30m56s ago — current · #import:https:::github.com:berriai:litellm post #2347
Issue #35272 is harder because it requires designing and integrating backend health tracking, routing decisions, failover behavior, configuration, and broad regression coverage across multiple provider paths. Issue #34934 is comparatively localized to readiness handling and an existing availability setting.
discussed in #import:https:::github.com:berriai:litellm

ranked child groups

no voted pairs yet in this scope

cli
src
spread
search