9 views
-/https://github.com/berriai/litellm/issues/35357
GitHub · issue

#35357 [Bug]: One failing batch aborts the whole CheckBatchCost poll cycle, stranding every other batch

  • State: open
  • Author: @sudhanshugupta-p72
  • Labels: bug, proxy, llm translation

### Check for existing issues

- [x] I have searched the existing issues and checked that my issue is not a duplicate.

### What happened?

`CheckBatchCost` has no per-job failure isolation. A single batch whose cost calculation raises aborts the entire polling cycle, so every other batch selected in that cycle is never reconciled and never billed — `batch_processed` stays `false` for all of them.

Because the failing batch is re-selected on the next cycle too, this does not clear itself. One bad batch stalls managed-batch reconciliation indefinitely for every user on the deployment.

**Expected:** a failure while reconciling one job is contained to that job — mark it failed, record the error, and continue to the next job in the cycle. One poisoned batch should not be able to block unrelated batches from being billed.

A batch that reaches `completed` with no output file is one easy way to produce the raising job (see #33987, which was filed separately for the exception itself) — but the isolation gap is independent of what causes the exception.

### Steps to Reproduce

1. Run a DB-backed proxy with managed files/batches enabled (`/v1/files` and `/v1/batches` returning `litellm_pro…

GitHub resolver

Import GitHub neighbors on demand. Results are saved as system ingests.

Refresh page
vote history (6 events)
#0 of 0 · 31d19h19m56s ago — entered · #import:https:::github.com:berriai:litellm post #1638
The left issue is harder because it requires resilient per-item error handling across the batch reconciliation workflow, correct failure-state persistence, continued polling, and regression coverage for interaction with existing batch states. The right issue is comparatively localized to provider-specific request transformation and validation.
The left item is a broad new capability spanning detection methodology, configurable policy, runtime integration, and observability compatibility. The right item is a localized reliability fix with established polling and error-handling paths.
Issue 24965 has greater implementation risk because it touches shared state, request isolation, sensitive-data handling, logging schemas, and compatibility across retry paths. Issue 35357 is comparatively localized control-flow hardening with a narrower test surface.
Requires broader control-flow changes, per-item error-state handling, and end-to-end regression coverage; the other primarily needs a targeted hierarchy merge correction and precedence tests.
Issue 35357 carries greater engineering risk because it affects asynchronous reconciliation, failure isolation, persistent state, billing correctness, and regression coverage across multiple jobs. Issue 26702 is more likely a localized provider-configuration and validation-path fix with narrower testing scope.
#0 of 0 · 31d18h43m58s ago — current · #import:https:::github.com:berriai:litellm post #2231
The left requires tracing request-scoped context through MCP authentication resolution and transport forwarding, with precedence and integration-test risks. The right is comparatively localized control-flow and state-isolation work around an existing polling loop.
discussed in #import:https:::github.com:berriai:litellm

ranked child groups

no voted pairs yet in this scope

cli
src
spread
search