8 views
-/https://github.com/berriai/litellm/issues/33666
GitHub · issue

#33666 [Bug]: Responses API previous_response_id session reconstruction runs an unbounded SELECT * over LiteLLM_SpendLogs (query-engine OOM)

  • State: open
  • Author: @wangyedev

### What happened?

On a proxy serving `/v1/responses` traffic, the Prisma **query-engine** process grows to **12–16 GB RSS** (≈100% anonymous memory) and fluctuates with load, pushing pods past their memory limit / into OOM.

**Root cause:** `ResponsesSessionHandler.get_all_spend_logs_for_previous_response_id` (`litellm/responses/litellm_completion_transformation/session_handler.py`) reconstructs a session by running an **unbounded** query:

```sql SELECT * FROM "LiteLLM_SpendLogs" WHERE session_id IN (SELECT session_id FROM matching_session) ORDER BY "endTime" ASC; ```

- No `LIMIT`, and `SELECT *` pulls the large blob columns (`messages`, `response`, `proxy_server_request`, `metadata`). - `session_id` comes from `trace_id` (`_get_session_id_for_spend_log`). Any session that maps to many rows makes the query engine **buffer the entire session into memory** before returning it to Python.

This happens for: - a **reused `litellm_trace_id`** across many requests (they collapse into one `session_id`), - a **very long-running conversation** chaining `previous_response_id`, - heavy **tool/MCP fan-out** (multiple spend-log rows per turn).

Because every turn replays the whole prior sess…

GitHub resolver

Import GitHub neighbors on demand. Results are saved as system ingests.

Refresh page
vote history (11 events)
#0 of 0 · 31d19h4m41s ago — entered · #import:https:::github.com:berriai:litellm post #1785
The left requires backend data-access redesign, memory-safe pagination or projection, compatibility validation, and load/regression testing across session reconstruction paths. The right is a localized dashboard rendering change with comparatively limited integration and testing scope.
The right-hand task is harder because it spans database-query design, memory behavior, pagination or bounded reconstruction semantics, compatibility, and load/regression testing; the left-hand task is narrower, centered on atomic distributed admission and reservation logic.
The left task spans distributed state consistency, cache semantics, lifecycle initialization, concurrency, and multi-replica validation. The right task is narrower database-query and reconstruction optimization, though it still requires performance testing and compatibility checks.
The right issue is harder because it requires safely changing persistence-layer behavior and session reconstruction under large-scale workloads, with greater compatibility, performance, and regression risk. The left issue is a narrower request-path integration fix with a more localized validation surface.
The right issue is harder because it requires redesigning data retrieval and session reconstruction for bounded memory while preserving ordering and conversation correctness across large histories, with database-level performance validation. The left issue is more localized to adapting guardrail response inspection for an alternate response representation and adding focused enforcement tests.
#33371 requires a new cross-cutting public contract, provider-specific normalization, router integration, compatibility decisions, and broader testing. #33666 is comparatively localized to query shape, bounded retrieval, and session-reconstruction correctness.
The right issue is harder because it requires redesigning data retrieval and session reconstruction under large-scale workloads, while preserving ordering and behavioral semantics, plus validating database and memory performance. The left issue is narrower: making shared counter/window updates atomic and ensuring asynchronous writes are reliably awaited.
Hierarchical authorization composition requires changes across data modeling, recursive resolution, inheritance updates, cycle handling, compatibility, and broad testing. The other task is a more localized production-query optimization with a narrower behavioral surface, though it still carries performance and regression risk.
Issue 33666 is harder because it requires redesigning data retrieval and session reconstruction to bound database and application memory while preserving response-chain correctness, with performance validation and regression coverage. Issue 31296 is comparatively localized to authentication response behavior, metadata discovery, configuration, and endpoint tests.
Issue 26535 is harder because it spans multiple implementation slices, provider integrations, compatibility behaviors, test infrastructure, and CI gating, creating substantially broader coordination and validation scope. Issue 33666 is a serious production-impacting defect, but its implementation is comparatively focused on query shaping, bounded reconstruction, and targeted regression coverage.
#0 of 0 · 31d17h16m54s ago — current · #import:https:::github.com:berriai:litellm post #3569
Supporting three distinct provider/platform integrations across SDK, proxy routing, parameter translation, multimodal behavior, and cost accounting creates broader compatibility and testing scope than isolating and bounding a single database access path.
discussed in #import:https:::github.com:berriai:litellm

ranked child groups

no voted pairs yet in this scope

cli
src
spread
search