7 views
-/https://github.com/berriai/litellm/issues/29955
GitHub · issue

#29955 [Feature]: per-team scoping for the response cache (prevent cross-tenant cache reuse)

  • State: open
  • Author: @dmitrymaranik

## The gap

On a multi-tenant proxy (multiple teams / virtual keys), the response-cache key is derived only from the request parameters (model + messages + settings) — **not** the caller's team or key. So two different teams sending the same request **share cache entries**: team B is served team A's cached response, and the hit is observable via the `x-litellm-cache-key` header (identical across teams). With the Redis semantic cache, even a *similar* request from team B hits team A's entry.

## Reproduce

1. Enable the Redis cache: `litellm_settings: {cache: true, cache_params: {type: redis}}`. 2. Create two virtual keys on two different teams (`/team/new` + `/key/generate`). 3. Send the same `/chat/completions` request with each key.

→ Both responses carry the **same** `x-litellm-cache-key`; team B's response is team A's cached generation.

## Why it matters

- Cross-tenant response reuse (one tenant served another's generated/billed output). - A cache-probing oracle: a tenant can infer whether another tenant asked something via cache hits. - No documented warning, and no built-in mitigation (only a manual per-request `namespace`).

## Proposed fix

A `cache_params` option to fol…

GitHub resolver

Import GitHub neighbors on demand. Results are saved as system ingests.

Refresh page
vote history (4 events)
#0 of 0 · 31d18h19m41s ago — entered · #import:https:::github.com:berriai:litellm post #2512
29955 requires cross-cutting cache-key and isolation changes across cache backends, request paths, compatibility behavior, and extensive regression/security testing; 26420 is comparatively localized to authorization-aware model-list construction and its tests.
The right issue is harder because it requires changing cache-key semantics across multiple cache backends and request paths, incorporating authenticated tenant context, preserving configurable behavior and compatibility, and validating isolation under both exact and semantic matching. The left issue is comparatively localized to retention cleanup and database consistency, with bounded schema/query and migration considerations.
The right issue is substantially harder because it requires designing tenant-aware cache identity and configuration across multiple cache implementations, preserving backward compatibility, handling security-sensitive isolation semantics, and adding broad integration coverage. The left issue is a localized provider-response parsing adjustment with comparatively limited testing and regression risk.
#0 of 0 · 31d17h47m7s ago — current · #import:https:::github.com:berriai:litellm post #3075
29955 is harder because it requires cross-cutting cache-key, tenancy, configuration, backend, compatibility, and security validation changes, while 33687 is comparatively localized to translation-path preservation and targeted tests.
discussed in #import:https:::github.com:berriai:litellm

ranked child groups

no voted pairs yet in this scope

cli
src
spread
search