19 views
-/https://github.com/berriai/litellm/issues/24965
GitHub · issue

#24965 [Bug]: previous_models in metadata leaks cross-request data and bloats spend logs

  • State: open
  • Author: @madhu19991
  • Labels: llm translation

## Bug

`previous_models` in the Router's request metadata has two issues:

### 1. Cross-request data leakage

`self.previous_models` is stored on the Router instance (router.py:544), not per-request. When request A fails and request B fails next, request B's `previous_models` contains request A's full metadata — including a different user's `user_api_key_auth`, `end_user_id`, tools, and `extra_headers` (which may contain OAuth bearer tokens).

Example: a request from user `UVk9MB0JsREqR7tH1liddPw=` with 10 tools had `previous_models` containing a completely different request from user `U2UlBAuuRG0q_faub5t4xyA=` with 19 tools and 3.2MB of base64 image data.

### 2. Massive metadata bloat

`log_retry` (router.py:5850) copies almost everything from `kwargs` into each `previous_models` entry — tools, full metadata dict, user_api_key_auth dumps, extra_headers with auth tokens, etc. With 3 retries and large tool schemas, this adds megabytes of redundant data to the spend log for each failed request.

Each failed attempt already has its own spend log entry with full error details, so duplicating all this data in `previous_models` is redundant.

## Suggested Fix

Replace the full kwargs d…

GitHub resolver

Import GitHub neighbors on demand. Results are saved as system ingests.

Refresh page
vote history (4 events)
#0 of 0 · 31d19h13m2s ago — entered · #import:https:::github.com:berriai:litellm post #1653
Issue 24965 has greater implementation risk because it touches shared state, request isolation, sensitive-data handling, logging schemas, and compatibility across retry paths. Issue 35357 is comparatively localized control-flow hardening with a narrower test surface.
The right-hand task is harder because it requires redesigning shared router state, defining safe data-retention boundaries, and validating concurrency and compatibility across retry and logging paths, whereas the left-hand task is a localized transformation fix with comparatively narrow tests.
#24965 is harder because it requires correcting request isolation and sensitive-data handling in shared router state, redesigning what gets persisted in retry/spend-log metadata, and adding concurrency, regression, and compatibility coverage. #12118 is comparatively localized to callback configuration precedence and merge behavior across UI and config sources.
#0 of 0 · 31d18h26m58s ago — current · #import:https:::github.com:berriai:litellm post #2417
Issue #24965 is substantially harder because it involves correcting shared router state, preventing cross-request data leakage under concurrency, reducing sensitive metadata retention, and preserving retry/logging behavior with comprehensive regression testing. Issue #29826 is a localized schema-default change with limited scope.
discussed in #import:https:::github.com:berriai:litellm

ranked child groups

no voted pairs yet in this scope

cli
src
spread
search