9 views
-/https://github.com/berriai/litellm/issues/26237
GitHub · issue

#26237 [Bug]: Worker with transient Postgres/HAProxy outage causing valid models to fail with "Invalid model name"

  • State: open
  • Author: @mishaja12
  • Labels: bug, proxy, llm translation

### Check for existing issues

- [x] I have searched the existing issues and checked that my issue is not a duplicate.

### What happened?

We hit an incident where LiteLLM started returning Invalid model name for a model that was valid and configured (e.g. google/gemini-2.5-flash, but actually all 140 models that we've configured).

<img width="1864" height="647" alt="Image" src="https://github.com/user-attachments/assets/915d864f-d429-4f46-b258-d62e904e831b" />

This happened in a setup where LiteLLM uses a Postgres-backed config/state layer behind HAProxy. The leading pattern we observed is:

there is a transient Postgres / HAProxy instability a LiteLLM worker starts or refreshes during that instability the worker appears to miss its initial successful model/router load the worker still serves traffic all requests going through that worker fail with Invalid model name for otherwise valid models the issue stops after the worker is restarted or after a later successful refresh

What we expected to happen:

LiteLLM should not serve traffic until router/model state is successfully loaded at least once, or LiteLLM should keep retrying and recover automatically once Postgres connectiv…

GitHub resolver

Import GitHub neighbors on demand. Results are saved as system ingests.

Refresh page
vote history (18 events)
#0 of 0 · 31d18h3m46s ago — entered · #import:https:::github.com:berriai:litellm post #2776
The left issue requires coordinated changes to worker initialization, state recovery, failure handling, and distributed-system testing. The right issue is more likely a localized endpoint registration, routing, or release/build discrepancy with a narrower test surface.
#26237 is harder because it spans worker lifecycle, readiness gating, persistent-state recovery, retry behavior, and HAProxy/Postgres failure modes. #34733 is comparatively localized to distributed counter atomicity and asynchronous write coordination.
The left issue is harder because it requires broad specification ingestion, schema-validation infrastructure, provider-normalization coverage, and extensive compatibility testing. The right issue is comparatively narrower, centered on worker lifecycle, state readiness, and recovery behavior.
30460 requires deeper distributed-state debugging, timeout/retry semantics, atomic accounting, reconciliation, and multi-pod regression testing; 26237 is comparatively narrower lifecycle readiness and retry handling.
The managed-settings feature spans multiple provider-specific integrations, configuration schemas, persistence, enforcement, authorization, and compatibility testing. The reliability bug is narrower in scope, primarily involving startup readiness, state-loading retries, and failure handling.
The left issue spans a multi-provider compatibility test framework, CLI-driven execution, matrix generation, fixtures, proxy configuration, and CI gating, creating substantially broader integration and maintenance scope. The right issue is technically risky because it concerns distributed startup and recovery behavior, but is more concentrated in the worker/model-loading lifecycle.
The right issue is harder because it requires cross-cutting worker lifecycle, readiness gating, retry recovery, state consistency, and race-condition testing across distributed dependencies. The left issue is narrower, mainly requiring response interception, format handling, and guardrail enforcement across pass-through paths.
26237 requires coordinated changes to startup/readiness behavior, state synchronization, retry and recovery logic, and failure-path testing across distributed deployments. 9753 is primarily an architectural dependency-evaluation and migration-planning task, with implementation scope still undefined.
The right issue is harder because it requires reliable distributed-state initialization, failure handling, readiness behavior, and automatic recovery across worker lifecycles. The left issue is broader than a single budget field but can largely extend established persistence, accounting, enforcement, and reset mechanisms.
The right issue is harder because it requires robust startup and refresh-state coordination across workers, database failures, retries, readiness, and recovery behavior. The left issue is broader in security impact but can likely be addressed through centralized credential-selection rules plus provider-focused regression tests.
The custom-routing work is harder because it requires designing and integrating a stable extensibility contract across proxy configuration, request execution, routing state, fallbacks, concurrency, and backward compatibility, with substantial documentation and test coverage. The outage-related fix is narrower in scope: it primarily involves startup/readiness gating, retry/recovery behavior, and preserving valid state during dependency failures.
#29452 has substantially greater scope and risk: it requires designing a durable credential abstraction, integrating multiple authentication modes and storage locations, defining secure lifecycle behavior, and preserving compatibility across provider and client workflows. #26237 is primarily a contained worker-readiness and recovery problem involving state loading, retries, and traffic gating.
The left issue is narrower operational-state handling: enforcing readiness, preserving a valid model snapshot, and adding retry/recovery behavior around transient dependency failures. The right issue is substantially riskier because it involves distributed concurrency, cache key/request isolation, Redis Cluster semantics, and potential security-sensitive data leakage; diagnosing it requires reliable reproduction and broad auditing to avoid regressions.
The right-hand RFC is harder because it requires cross-provider contract design, classification semantics, compatibility decisions, and coordinated API/test coverage, while the left-hand fix is primarily lifecycle and readiness handling within existing state-loading behavior.
26237 is harder because it requires coordinated changes across worker readiness, persistent-state recovery, refresh behavior, and request admission under partial infrastructure failure. 34328 is more localized to bounding and safely testing schema expansion across a small set of translation paths.
#26237 is harder because it spans worker lifecycle, startup readiness, retry/recovery behavior, distributed configuration state, and traffic-gating semantics, with substantial operational and regression risk. #34732 is narrower, primarily requiring an atomic distributed admission/reservation mechanism and related accounting tests.
Issue 26237 is harder because it requires diagnosing and hardening distributed startup, persistence, retry, readiness, and recovery behavior across failure states, with high risk of serving incorrect authorization or routing decisions. Issue 28032 is substantial but more bounded to recursive group resolution, dependency propagation, cycle handling, and related storage/API changes.
#0 of 0 · 31d17h15m41s ago — current · #import:https:::github.com:berriai:litellm post #3574
Model-group composition requires cross-referencing and resolving nested configuration, defining routing and inheritance semantics, preventing cycles, and updating validation and broad test coverage. The other change is more localized to initialization readiness, retry handling, and recovery behavior.
discussed in #import:https:::github.com:berriai:litellm

ranked child groups

no voted pairs yet in this scope

cli
src
spread
search