8 views
-/https://github.com/berriai/litellm/issues/33955
GitHub Ā· issue

#33955 [Bug]: CheckBatchCost fully buffers batch output (incl. base64 images) in memory, unlocked across workers — OOM on image-generation batches

  • State: open
  • Author: @michalgubik
  • Labels: llm translation

**Check for existing issues:** Searched existing issues — related but distinct: #33316, #30635, #28294, #27944, #27891 (all cost-attribution bugs — wrong IDs, 403s, cost showing 0 — not the memory/concurrency issue below).

**Issue classification:** Out of Memory (OOM) events or pod restarts.

**What happened?**

Running litellm proxy v1.91.0 with `num_workers=2` per pod (4 replicas, 8 worker processes total) and a client submitting Vertex AI image-generation batches (`vertex_ai/gemini-*-image*`) via `/v1/files` + `/v1/batches`. 4 replicas were OOM-killed within a 15-minute window, each showing the same pattern: memory climbs to a steady ~4-4.5GB plateau over 1.5-2h, then a fast spike (confirmed via `container_memory_max_usage_bytes`, up to 5.59GB) crosses the container's memory limit in under a minute — faster than any periodic metrics scrape resolves.

No recent litellm version upgrade preceded this incident — the deployment had been stable on v1.91.0.

Root cause, traced through the source: 1. `CheckBatchCost`'s output-file cost-tracking path (`enterprise/litellm_enterprise/proxy/common_utils/check_batch_cost.py`) fully downloads a completed batch's GCS output file, transforms i…

GitHub resolver

Import GitHub neighbors on demand. Results are saved as system ingests.

Refresh page
vote history (4 events)
#0 of 0 Ā· 31d18h30m58s ago — entered Ā· #import:https:::github.com:berriai:litellm post #2536
The left issue requires a cross-cutting backend data-path redesign involving bounded memory, streaming or chunked processing, worker-concurrency behavior, cleanup, and regression testing under large payloads. The right issue is a localized dashboard/API pagination adjustment with a clear existing fix path and limited validation scope.
#0 of 0 Ā· 31d18h15m55s ago Ā· #import:https:::github.com:berriai:litellm post #2793
#33955 requires cross-cutting changes to batch processing, buffering, concurrency, resource limits, and regression coverage, with significant production-risk around large payloads and worker coordination. #28989 is comparatively localized to response translation and error propagation in two provider-specific paths.
#33955 is harder because it requires safely redesigning large-payload processing and concurrency behavior across batch workers, with substantial memory, performance, and regression risk. #25738 is narrower, primarily involving provider/model cost attribution for a streaming transport.
#0 of 0 Ā· 31d17h50m7s ago — current Ā· #import:https:::github.com:berriai:litellm post #3246
33955 is harder because it requires redesigning batch-output processing for bounded memory, safe concurrency, and regression-resistant cost tracking across workers, whereas 35133 is primarily localized defensive error handling and status propagation.
discussed in #import:https:::github.com:berriai:litellm

ranked child groups

no voted pairs yet in this scope

cli
src
spread
search