8 views
-/https://github.com/berriai/litellm/issues/34031
GitHub · issue

#34031 Proxy ingests unbounded upstream error-response bodies -> event-loop stall / RSS blowup

  • State: open
  • Author: @rmonsurate
  • Labels: llm translation

### Summary

The proxy ingests **upstream error-response bodies without any size bound**. When an OpenAI-compatible backend returns a very large error body (we hit a ~114 MB HTTP 400 from a backend's request-validation dump), the proxy reads the whole body, runs regex masking over it, and propagates it through the fallback chain and logs. This **stalls the asyncio event loop for seconds** (health/liveness checks time out, the accept queue saturates) and **inflates RSS** — a single malformed upstream response degraded the whole proxy.

### Environment

- litellm `1.85.0`, proxy mode, single uvicorn worker, heavy streaming chat-completions traffic, OpenAI-compatible self-hosted backends with a fallback cascade.

### Where

On an upstream error, the masked-error path reads the entire response body before doing anything else:

- `litellm/llms/custom_httpx/http_handler.py` — `_raise_masked_sync_error` / `_raise_masked_async_error` read the full body (`_safe_get_response_text` / streaming aread) and then run `mask_sensitive_info(...)` over the whole thing. - `litellm/llms/custom_httpx/llm_http_handler.py` — `_handle_error` uses `error_text = e.response.text` (full body).

With a 100 MB+ …

GitHub resolver

Import GitHub neighbors on demand. Results are saved as system ingests.

Refresh page
vote history (2 events)
#0 of 0 · 31d18h36m6s ago — entered · #import:https:::github.com:berriai:litellm post #2367
34031 requires coordinated changes across multiple transport and error-handling paths, careful memory and streaming semantics, compatibility validation, and regression/load testing; 27759 is comparatively narrower protocol-format diagnosis and adaptation.
#0 of 0 · 31d18h35m29s ago — current · #import:https:::github.com:berriai:litellm post #2377
Issue 34031 is harder because it requires coordinated changes across synchronous and asynchronous transport paths, bounded resource handling, masking, exception propagation, and regression coverage under failure conditions. Issue 35191 is comparatively localized to telemetry span lifecycle and metadata propagation.
discussed in #import:https:::github.com:berriai:litellm

ranked child groups

no voted pairs yet in this scope

cli
src
spread
search