8 views
-/https://github.com/berriai/litellm/issues/28907
GitHub · issue

#28907 [Bug]: S3 batch flush drops queued logs before uploads finish

  • State: open
  • Author: @lullu57

### Check for existing issues

- [X] I have searched the existing issues and checked that my issue is not a duplicate.

### What happened?

`S3Logger.async_send_batch` schedules one upload per queued event with `asyncio.create_task` and returns immediately without awaiting any of them. `CustomBatchLogger.flush_queue` then calls `self.log_queue.clear()` as soon as `async_send_batch` returns, and `async_upload_data_to_s3` catches upload exceptions silently. As a result, if an upload task fails or is cancelled (process shutdown, network error, AWS throttling) the event is gone from the queue and never retried, permanently dropping audit/log entries.

**Observed:** `async_send_batch` does `for payload in self.log_queue: asyncio.create_task(self.async_upload_data_to_s3(payload))` and returns. `flush_queue` sees the function return and runs `self.log_queue.clear()` unconditionally (since `preserve_events_added_during_flush = False`). The fire-and-forget tasks may complete later, fail, or be cancelled — failures are only logged via `verbose_logger.exception(...)` and `handle_callback_failure(...)`. There is no path that returns the failed event to the queue.

**Expected:** A successful fl…

GitHub resolver

Import GitHub neighbors on demand. Results are saved as system ingests.

Refresh page
vote history (2 events)
#0 of 0 · 31d18h22m26s ago — entered · #import:https:::github.com:berriai:litellm post #2475
The left task has substantially higher engineering risk because it spans asynchronous lifecycle coordination, failure recovery, queue durability, shutdown behavior, and regression testing. The right task appears narrower and would primarily require profiling followed by a targeted performance fix.
#0 of 0 · 31d17h35m13s ago — current · #import:https:::github.com:berriai:litellm post #3287
The left issue is substantially harder: it spans multiple integrations, requires a new centrally administered configuration model, and raises API, validation, persistence, security, and compatibility concerns. The right issue is a comparatively localized reliability fix in an existing asynchronous logging workflow, with focused tests around task completion and failure handling.
discussed in #import:https:::github.com:berriai:litellm

ranked child groups

no voted pairs yet in this scope

cli
src
spread
search