6 views
-/https://github.com/berriai/litellm/issues/31557
GitHub · issue

#31557 Fallback chain silently fails when fallback model has smaller context window than primary

  • State: open
  • Author: @ttxs69
  • Labels: llm translation

## Observed behavior

When a Router is configured with a generic `fallbacks` list and the primary model fails for a non-context reason (rate limit, timeout, server error), the same `messages` payload is forwarded to each fallback model as-is. If a fallback model has a smaller `max_input_tokens` than the primary and the request exceeds the fallback's context window, the fallback raises `ContextWindowExceededError` and the chain ends. No preflight token counting or message trimming happens between fallbacks.

Repro (router config):

```yaml model_list: - model_name: gpt-4-turbo litellm_params: model: gpt-4-turbo api_key: os.environ/OPENAI_API_KEY - model_name: gpt-4 litellm_params: model: gpt-4 api_key: os.environ/OPENAI_API_KEY router_settings: fallbacks: - gpt-4-turbo: [gpt-4] ```

Send a ~10K-token request. If the primary fails on a rate limit (or any non-context error), the fallback `gpt-4` (`max_input_tokens=8192`) receives the same payload and fails with `ContextWindowExceededError`. The user gets a context-window error from the fallback, with no indication that the fallback was a bad choice.

## Why this happens

In `litellm/router_uti…

GitHub resolver

Import GitHub neighbors on demand. Results are saved as system ingests.

Refresh page
vote history (2 events)
#0 of 0 · 31d18h20m13s ago — entered · #import:https:::github.com:berriai:litellm post #2503
The right issue has greater implementation risk because it changes security-sensitive request-gating semantics, requires configuration propagation across proxy and guardrail execution paths, and needs careful backward-compatibility and failure-mode testing. The left issue is narrower, mainly involving router fallback selection, token-limit checks, and targeted retry-path tests.
#0 of 0 · 31d18h10m17s ago — current · #import:https:::github.com:berriai:litellm post #2669
The left issue requires a broader provider integration with new request semantics, authentication/configuration handling, response translation, streaming behavior, and end-to-end coverage. The right issue is primarily a focused Router fallback enhancement involving context estimation and retry-path handling, with comparatively lower architectural risk.
discussed in #import:https:::github.com:berriai:litellm

ranked child groups

no voted pairs yet in this scope

cli
src
spread
search