7 views
-/https://github.com/berriai/litellm/issues/27704
GitHub · issue

#27704 Prisma Query Engine startup race condition causes spend data loss during rolling deployments

  • State: open
  • Author: @jimmychen-p72

## Summary

During Kubernetes rolling deployments, LiteLLM starts Uvicorn and immediately schedules background jobs (`update_spend`, `load_credentials`, `add_deployment`, budget cache population) **before** the embedded Prisma Query Engine subprocess is ready to accept connections. This causes a ~5–6 minute window per pod where all DB operations fail silently, resulting in lost spend data and degraded startup state.

## Environment

- **LiteLLM version:** 1.83.14 - **Deployment:** Kubernetes rolling deploy (multiple pods) - **Container base image:** Wolfi-based (`wolfi` Linux distro) - **Database:** PostgreSQL via Prisma

## Steps to Reproduce

1. Deploy LiteLLM as a Kubernetes rolling deployment (multiple replicas) 2. Trigger a rolling pod restart (e.g. version upgrade) 3. Observe logs on newly started pods

## What Happens

On pod startup, Uvicorn comes up and immediately starts background jobs. The embedded Prisma Query Engine (a Rust binary that runs as a local subprocess) is still initializing — especially slow on Wolfi-based containers because Prisma does not recognize the `wolfi` distro and falls back to Debian engine binaries at runtime:

``` prisma:warn Prisma doesn't know…

GitHub resolver

Import GitHub neighbors on demand. Results are saved as system ingests.

Refresh page
vote history (1 events)
#0 of 0 · 31d18h13m55s ago — entered · #import:https:::github.com:berriai:litellm post #2620
The left issue is harder because it spans application startup ordering, subprocess readiness detection, failure handling, background task coordination, and deployment-level regression testing. The right issue is more localized to protocol translation state management and stream conformance, with a narrower test surface.
discussed in #import:https:::github.com:berriai:litellm

ranked child groups

no voted pairs yet in this scope

cli
src
spread
search