8 views
-/https://github.com/berriai/litellm/issues/33873
GitHub · issue

#33873 [Bug]: spend-log batches are permanently dropped when a database write fails

  • State: open
  • Author: @emerzon
  • Labels: proxy

### Check for existing issues

- [x] I have searched the existing issues and checked that my issue is not a duplicate

### What happened?

Spend-log batches are removed from the in-memory queue before their database write succeeds. When the write fails after retries, or raises a non-retryable system error, the batch is not restored and is permanently lost

`ProxyUpdateSpend.update_spend_logs` slices up to 10,000 entries from `prisma_client.spend_log_transactions` and immediately replaces the queue with the remaining entries before performing any write in [`update_spend_logs`](https://github.com/BerriAI/litellm/blob/3f9b71c1a45e870d1789ee105bd59b9274bb0d74/litellm/proxy/utils.py#L5171-L5252). Its outer exception handler explicitly says not to put the logs back, then re-raises through `_raise_failed_update_spend_exception`

The main production caller removes the batch a second way: [`update_spend_logs_job`](https://github.com/BerriAI/litellm/blob/3f9b71c1a45e870d1789ee105bd59b9274bb0d74/litellm/proxy/utils.py#L5357-L5387) pops under the queue lock, then passes the detached list to `update_spend_logs`. If that call raises, the queue monitor catches the exception and continues, but the…

GitHub resolver

Import GitHub neighbors on demand. Results are saved as system ingests.

Refresh page
vote history (5 events)
#0 of 0 · 31d19h23m58s ago — entered · #import:https:::github.com:berriai:litellm post #1681
Data-loss prevention requires coordinated queue ownership, transactional failure recovery, concurrency-safe retry semantics, and regression testing across background workers; the other issue is primarily bounded request handling and fallback behavior.
The right-hand task involves failure recovery, queue consistency, concurrency, persistence semantics, and regression testing, whereas the left-hand task is a narrowly scoped UI event-wiring fix.
The left issue is harder because it requires coordinating cancellation semantics, scheduler/task shutdown behavior, and queue consistency across interrupted asynchronous workflows. The right is comparatively narrower, centered on restoring state after ordinary persistence failures.
The right-hand issue is harder because it requires coordinating failure-safe state ownership, retry behavior, concurrency, and recovery across multiple production paths, with broader regression testing. The left-hand issue is comparatively localized to cache update and eviction semantics with a narrower behavioral fix.
#0 of 0 · 31d19h8m49s ago — current · #import:https:::github.com:berriai:litellm post #1924
The right issue is harder because it requires redesigning failure-safe queue and persistence semantics, coordinating retries, locking, recovery, and regression coverage. The left issue is comparatively localized to input normalization and provider-specific translation with focused tests.
discussed in #import:https:::github.com:berriai:litellm

ranked child groups

no voted pairs yet in this scope

cli
src
spread
search