7 views
-/https://github.com/berriai/litellm/issues/32112
GitHub · issue

#32112 Proxy: non-standard request param sent once is permanently re-injected into ALL subsequent requests to that deployment (in-memory state poisoning)

  • State: open
  • Author: @michaelchasse
  • Labels: llm translation

### What happened

On the LiteLLM proxy (v1.81.16, `openai/` custom provider entries pointing at llama.cpp `llama-server` backends), **a single request that includes a non-standard body parameter permanently contaminates the proxy's in-memory state for that deployment: the parameter is silently re-injected into every subsequent request to that model — from any client — until the proxy is restarted.**

We hit this in production: one client sent `dry_multiplier: 0.8` (a llama.cpp sampler param) in its request body. That client was later fixed to stop sending it — but the proxy kept injecting `dry_multiplier: 0.8` into all requests for that model for the rest of the day. Because DRY penalizes repeated token sequences, this silently corrupted code generation (repeated identifiers got mangled mid-file, e.g. `LEFT_ASSOCIATIVE` → `LEFT_ASSOCIAIVE`, and commas/brackets were dropped in long files, causing syntax errors). It was extremely hard to trace because the on-disk config was clean and only the *arriving* request body (observed via llama-server's `/slots`) showed the extra param.

### 4-step reproduction

Backend: llama.cpp `llama-server` (any model), config entry:

```yaml model_list…

GitHub resolver

Import GitHub neighbors on demand. Results are saved as system ingests.

Refresh page
vote history (3 events)
#0 of 0 · 31d18h35m49s ago — entered · #import:https:::github.com:berriai:litellm post #2263
The left issue requires tracing and correcting shared mutable request or deployment state, ensuring isolation across requests and clients, handling concurrency and lifecycle behavior, and adding regression coverage. The right issue is a narrower provider-authentication and configuration integration problem.
#32112 is harder because it involves diagnosing and eliminating cross-request mutable-state leakage in the proxy, preserving provider-specific parameter behavior, and adding concurrency-focused regression coverage. #8842 is comparatively localized to async router callback dispatch and hook handling.
#0 of 0 · 31d17h20m57s ago — current · #import:https:::github.com:berriai:litellm post #3515
Nested access-group support requires recursive resolution, cycle prevention, update propagation, persistence and authorization compatibility, plus broad API and regression testing. The other issue is comparatively localized to isolating mutable request or deployment state and adding focused regression coverage.
discussed in #import:https:::github.com:berriai:litellm

ranked child groups

no voted pairs yet in this scope

cli
src
spread
search