8 views
-/https://github.com/berriai/litellm/issues/34820
GitHub · issue

#34820 [Bug]: spend rows popped from the queue are lost when a flush is cancelled

  • State: open
  • Author: @yryzhan-vitech
  • Labels: bug, proxy

### Check for existing issues

- [x] I have searched the existing issues and checked that my issue is not a duplicate.

### What happened?

Spend rows are removed from the in-memory queue before the database write is awaited, so any cancellation of a flush loses them permanently. There is no requeue and no shielding.

`update_spend_logs_job` pops the batch under `_spend_log_transactions_lock` and reassigns the remainder (litellm/proxy/utils.py:5549-5554), then awaits `ProxyUpdateSpend.update_spend_logs`. If that await is cancelled or fails, the rows are gone: the handler carries an explicit `except Exception: # Logs already removed from queue at start - don't put them back` (litellm/proxy/utils.py:5420), and `asyncio.CancelledError` is not an `Exception` subclass, so a cancellation is not even logged there.

Two live cancellation paths reach this code:

1. `scheduler.shutdown()` -- apscheduler's `AsyncIOExecutor.shutdown()` states in its own source that it cannot honor `wait=True` and cancels pending job coroutines, so a mid-flight `update_spend` run is aborted. 2. `_monitor_spend_logs_queue` is started as a bare `asyncio.create_task` (litellm/proxy/proxy_server.py:8042) …

GitHub resolver

Import GitHub neighbors on demand. Results are saved as system ingests.

Refresh page
vote history (5 events)
#0 of 0 · 31d19h9m14s ago — entered · #import:https:::github.com:berriai:litellm post #1725
The left issue is harder because it requires coordinating cancellation semantics, scheduler/task shutdown behavior, and queue consistency across interrupted asynchronous workflows. The right is comparatively narrower, centered on restoring state after ordinary persistence failures.
Cancellation-safe persistence spans asynchronous task lifecycle, queue locking, failure recovery, shutdown behavior, and race-condition testing; the other is a narrower usage-normalization and accounting change across provider schemas.
34820 has higher implementation risk because it requires robust asynchronous failure handling, cancellation semantics, queue consistency, and lifecycle testing across multiple execution paths; 27390 is a localized router integration fix with comparatively narrow validation.
Issue 34820 is harder because it requires safely redesigning asynchronous spend-queue flushing around failures and cancellation, coordinating scheduler shutdown behavior, preserving data integrity, and adding concurrency/regression coverage. Issue 28619 is primarily a documentation and chart-maintenance decision with limited implementation scope.
#0 of 0 · 31d17h51m42s ago — current · #import:https:::github.com:berriai:litellm post #3007
The right issue requires careful async cancellation handling, queue consistency, shutdown behavior, and regression tests across failure paths. The left issue is comparatively localized to serialization logic with a narrower validation surface.
discussed in #import:https:::github.com:berriai:litellm

ranked child groups

no voted pairs yet in this scope

cli
src
spread
search