8 views
-/https://github.com/berriai/litellm/issues/31260
GitHub · issue

#31260 Router sync _embedding bypasses team/access-group scoping (no request_kwargs); sync cache read path strips metadata

  • State: open
  • Author: @kingdoooo
  • Labels: llm translation

### What

Sync `Router._embedding` selects its deployment without forwarding `request_kwargs`, while every comparable path forwards it. The team filter and the model-access-group filter therefore never see `user_api_key_team_id` or `user_api_key_auth` on sync embedding calls, so a sync caller can land on a deployment outside their team or access group.

Concretely, `_embedding` calls `get_available_deployment(model=model, input=input, specific_deployment=...)` with no `request_kwargs` (litellm/router.py around line 5072), whereas the async twin `_aembedding` calls `async_get_available_deployment(..., request_kwargs=kwargs)` (around 5158), and sync `_completion` calls `get_available_deployment(..., request_kwargs=kwargs)` (around 1896). Deployment selection reads the team id from `request_kwargs["metadata"]["user_api_key_team_id"]` in `_common_checks_available_deployment` and the auth object in `_filter_deployments_by_model_access_groups`; with no `request_kwargs`, `team_id` is `None` and `should_include_deployment` treats every deployment as globally accessible. Sync `_image_generation` has the same omission

### Related cache-layer gap

There is a second, related gap. `Cache._get_…

GitHub resolver

Import GitHub neighbors on demand. Results are saved as system ingests.

Refresh page
vote history (6 events)
#0 of 0 · 31d19h9m23s ago — entered · #import:https:::github.com:berriai:litellm post #1712
The UI performance problem is harder because it requires broad profiling and regression isolation across potentially many frontend and backend paths, while the router issue has a more localized fix with targeted validation and security-focused tests.
The right issue spans authorization-sensitive routing, multiple synchronous code paths, and cache behavior, requiring careful regression testing across router and cache layers. The left issue is more localized to token accounting and request parsing, with a narrower validation surface.
31260 is harder because it spans router deployment-selection paths and cache metadata propagation, requiring careful security-sensitive fixes across synchronous behavior and regression coverage. 33671 is comparatively localized to extending one AWS Guardrails request and adding targeted tests.
31260 is harder because it requires careful cross-cutting authorization and metadata propagation fixes across synchronous router, image, and cache paths, with substantial regression risk. 34648 is primarily a bounded provider adapter and proxy integration.
#35524 requires designing a reliable atomic reservation fallback across unknown-cost and multimodal requests, preserving accounting correctness under concurrency and adding broad regression coverage. #31260 is primarily a scoped propagation and cache-metadata consistency fix across a few synchronous paths.
#0 of 0 · 31d18h33m31s ago — current · #import:https:::github.com:berriai:litellm post #2302
The NeMo integration spans multiple public layers, provider behavior, configuration, request lifecycles, and compatibility testing, creating substantially broader implementation and maintenance risk. The router issue is a focused authorization/data-propagation fix with targeted regression coverage.
discussed in #import:https:::github.com:berriai:litellm

ranked child groups

no voted pairs yet in this scope

cli
src
spread
search