9 views
-/https://github.com/berriai/litellm/issues/33371
GitHub · issue

#33371 [RFC] Expose structured provider-error/route-health contract from router cooldown state

  • State: open
  • Author: @niStee

## Summary

LiteLLM already tracks per-deployment cooldown state internally (`failed_calls` InMemoryCache, `allowed_fails_policy`) and OpenRouter already returns per-request `openrouter_metadata` with `requested`, `attempts[]`, and `endpoints[]`. However, neither exposes a structured, machine-readable error classification that downstream frameworks can consume to make fallback routing decisions without re-parsing raw error strings.

## Problem

Frameworks building on top of LiteLLM maintain independent regex pattern lists to classify provider errors — `Insufficient balance`, `permission_error`, provider-specific 429/402 bodies, quota exhaustion messages. As a concrete example, `oh-my-openagent` currently has three diverged pattern lists (tracked in [code-yeongyu/oh-my-openagent#5451](https://github.com/code-yeongyu/oh-my-openagent/issues/5451)):

- `runtime-fallback-error-classifier.ts`: ~44 regex/name checks - `hooks/runtime-fallback/constants.ts`: 35 regex patterns (8 extras not in core) - `model-error-classifier.ts`: 89 string patterns in a parallel classifier

Adding support for a new provider requires touching 3+ files in every downstream framework. This is a maintenance antip…

GitHub resolver

Import GitHub neighbors on demand. Results are saved as system ingests.

Refresh page
vote history (17 events)
#0 of 0 · 31d18h10m0s ago — entered · #import:https:::github.com:berriai:litellm post #2697
The left requires cross-cutting design across routing state, provider error normalization, public contract/schema, compatibility, and broad test coverage, while the right is a localized logging-behavior adjustment with comparatively limited regression risk.
Issue #33371 is harder because it requires designing and integrating a cross-cutting, provider-agnostic contract across router state, error normalization, response metadata, fallback behavior, and compatibility tests. Issue #33062 is comparatively contained: it primarily involves correcting startup synchronization and configuration coupling for persisted agents.
28125 requires deeper changes to deployment resolution and routing semantics, including nested expansion, inheritance, validation, and edge-case handling across existing configuration paths. 33371 is broader in API design and classification consistency, but can be implemented incrementally around existing cooldown and error-handling state.
Cross-provider integration testing, proxy orchestration, version handling, and CI-gate maintenance create broader execution and environment risk for #26535; #33371 is a substantial API and router-contract change but has a more concentrated implementation scope.
The right issue is harder because it requires designing and integrating a durable cross-provider error taxonomy and router-health API, aligning internal state with externally consumed metadata, preserving compatibility, and adding broad tests and documentation. The left issue is comparatively localized debugging and correction of an Azure request-path parameter propagation defect.
The right-hand work is harder because it requires a new cross-cutting public contract, integration with routing and provider error paths, compatibility decisions, and broad validation. The left-hand fix has substantial security risk and wide coverage, but can likely be concentrated in shared credential-selection and endpoint-validation logic.
28235 is harder because it requires coordinated persistence, concurrency-safe accounting, authentication enforcement, lifecycle resets, backward compatibility, and broad regression coverage across core budget paths. 33371 is mainly an interface and classification-contract extension around existing routing state, with narrower implementation boundaries.
33371 is harder because it requires cross-cutting API design, stable semantics, and integration across routing, provider metadata, and compatibility boundaries; 34733 is primarily a focused concurrency and persistence correction with targeted tests.
29452 is harder because it spans security-sensitive credential abstractions, multiple authentication modes, persistence, provider integration, lifecycle management, compatibility, and extensive edge-case testing. 33371 is comparatively contained around normalizing existing routing and error state into a documented contract.
The right-hand RFC is harder because it requires cross-provider contract design, classification semantics, compatibility decisions, and coordinated API/test coverage, while the left-hand fix is primarily lifecycle and readiness handling within existing state-loading behavior.
#33371 is harder because it spans provider-specific error normalization, router state, public contract design, backward compatibility, and downstream integration semantics. #28032 is substantial but more contained within recursive access-group resolution, persistence, validation, and update propagation.
The left issue is harder because it spans public API design, cross-provider error normalization, router-state integration, compatibility, and downstream contract validation. The right issue is a narrower translation-layer safety fix, requiring bounded recursion behavior, caller updates, and regression tests, though it carries significant performance-risk validation.
#33371 requires a new cross-cutting public contract, provider-specific normalization, router integration, compatibility decisions, and broader testing. #33666 is comparatively localized to query shape, bounded retrieval, and session-reconstruction correctness.
33371 is harder because it requires a cross-provider, backward-compatible contract spanning error normalization, router state, fallback behavior, and public API design. 27883 is broader in product scope but can be implemented more independently as proxy configuration and endpoint support.
The left issue is harder because it requires diagnosing and correcting distributed state, failure handling, concurrency, and backward-compatibility risks across persistence and budget-enforcement paths, with difficult reproducibility and regression testing. The right issue is primarily an API-contract and metadata-plumbing change around existing router state, with comparatively contained implementation and validation.
The left requires a cross-cutting API/design change spanning routing state, error normalization, provider behavior, and compatibility guarantees, while the right is a focused request-rewriting fix with narrower validation scope.
#0 of 0 · 31d17h17m23s ago — current · #import:https:::github.com:berriai:litellm post #3572
Issue 33371 is harder because it requires designing and integrating a durable cross-layer error and routing-health contract, with compatibility, classification semantics, and consumer-facing API implications. Issue 33921 is broader in provider count but is comparatively bounded to conventional provider mappings, parameter handling, and integration tests.
discussed in #import:https:::github.com:berriai:litellm

ranked child groups

no voted pairs yet in this scope

cli
src
spread
search