6 views
-/https://github.com/berriai/litellm/issues/33276
GitHub · issue

#33276 [Bug]: health_check() connection errors during planned engine recreate are mislabeled as disconnect() and alert as db_exceptions

  • State: open
  • Author: @tushar8408
  • Labels: proxy

### What happened?

On a proxy configured with RDS IAM auth (proactive token refresh), the writer Prisma query-engine is periodically recreated via `recreate_prisma_client` (kill-then-construct). This is a *planned* recreate and is coordinated correctly since the engine-death-watcher race fix (#29176).

However, during the brief (~0.5–1s) recreate window, a concurrent `/health/readiness` probe calls `PrismaClient.health_check()` → `query_raw("SELECT 1")`, which fails with:

``` httpx.ConnectError: All connection attempts failed ```

This transient, expected failure is reported to `proxy_logging_obj.failure_handler(...)`, which drives the `db_exceptions` alerting. Because engine recreates happen every IAM-refresh cycle (~every 10–15 min per process), this produces recurring **false-positive DB-exception alerts** even though the reconnect is benign — no pod restart, no request-failure impact.

There are two distinct problems:

**1. Mislabeled error string.** In `litellm/proxy/utils.py`, the `health_check()` exception handler builds:

```python error_msg = f"LiteLLM Prisma Client Exception disconnect(): {str(e)}" ```

but this is inside `health_check()` (the `failure_handler` call eve…

GitHub resolver

Import GitHub neighbors on demand. Results are saved as system ingests.

Refresh page
vote history (1 events)
#0 of 0 · 31d17h58m19s ago — entered · #import:https:::github.com:berriai:litellm post #3220
The right-side feature requires coordinated changes across authorization, budget accounting, persistence, request identity, configuration, and comprehensive compatibility testing. The left-side fix is comparatively localized to error classification and alert suppression around an existing lifecycle. Therefore the right-side work is substantially harder.
discussed in #import:https:::github.com:berriai:litellm

ranked child groups

no voted pairs yet in this scope

cli
src
spread
search