6 views
-/https://github.com/berriai/litellm/issues/34805
GitHub · issue

#34805 [Bug]: in-memory spend buffers are dropped on proxy shutdown

  • State: open
  • Author: @yryzhan-vitech
  • Labels: bug, proxy

### Check for existing issues

- [x] I have searched the existing issues and checked that my issue is not a duplicate.

### What happened?

In-memory spend buffers are silently dropped when the proxy shuts down.

`update_spend` (key/team/user/end_user spend) and `update_daily_tag_spend` are apscheduler jobs that accumulate rows in in-memory buffers between intervals. On shutdown:

- the scheduler is never stopped anywhere, so a job can still be mid-flight, and - `proxy_shutdown_event` (litellm/proxy/proxy_server.py) goes straight to `await prisma_client.disconnect()` without flushing either buffer.

Everything not yet committed is lost. On a uvicorn worker recycle (MAX_REQUESTS_BEFORE_RESTART) or any SIGTERM (rolling update, scale-down, liveness kill) this happens on every single restart, because the replacement worker only sees its own future traffic.

Observed symptoms in the recycle window: `ClientNotConnectedError` on update_spend, "Daily tag spend batch upsert failed", and "Error in spend logs queue monitor: Server disconnected without sending a response."

Expected: buffered spend rows are flushed to the DB before Prisma disconnects. Actual: they are discarded.

Note that `…

GitHub resolver

Import GitHub neighbors on demand. Results are saved as system ingests.

Refresh page
vote history (1 events)
#0 of 0 · 31d17h44m38s ago — entered · #import:https:::github.com:berriai:litellm post #3234
The right-hand issue is harder because it spans retention semantics, database cleanup ordering, migration or indexing decisions, batching, and compatibility testing across supported databases. The left-hand issue is narrower: coordinating existing shutdown lifecycle, draining buffers safely, and adding race/failure-path tests.
discussed in #import:https:::github.com:berriai:litellm

ranked child groups

no voted pairs yet in this scope

cli
src
spread
search