7 views
-/https://github.com/berriai/litellm/issues/34766
GitHub · issue

#34766 [Feature]: Auto Router v2: derive a session key from request content when `session_id` is absent

  • State: open
  • Author: @MuFazil
  • Labels: enhancement, proxy, llm translation

### Check for existing issues

- [x] I have searched the existing issues and checked that my issue is not a duplicate.

### The Feature

## Summary

`session_affinity` and `adaptive` both depend on a resolvable `session_id`. When the client doesn't send one, both silently degrade — affinity stops pinning, and the bandit has no key to attribute reward against. Nothing errors. The router just quietly stops doing the two things you enabled it for.

Proposal: add an optional fallback chain for deriving the session key, so a missing `session_id` degrades to a best-effort key instead of to nothing.

## Current behaviour

Per the Auto Routing docs:

> `session_id` is read from request metadata; when no `session_id` is resolvable the router classifies every turn as usual.

That's the documented path for `session_affinity`. For `adaptive: true`, the docs state that feedback attributes back to the model that actually served the previous response — but with no session key there is nothing to correlate a later turn against, so reward attribution has nowhere to land.

Net effect with both flags on and no `session_id`:

| Feature | Configured | Actual behaviour | | --- | --- | --- | | `session_…

GitHub resolver

Import GitHub neighbors on demand. Results are saved as system ingests.

Refresh page
vote history (3 events)
#0 of 0 · 31d19h8m22s ago — entered · #import:https:::github.com:berriai:litellm post #1823
#34766 requires cross-cutting routing design, session-key semantics, state handling, and compatibility testing across affinity and adaptive behavior; #26320 is a comparatively localized provider-translation fix with narrower regression scope.
The left issue is harder because it spans security-sensitive request validation, routing boundaries, and asynchronous context-lifecycle behavior, requiring careful concurrency analysis and regression protection. The right issue is comparatively bounded to extending router key derivation with defined precedence, stability, privacy, and test coverage.
#0 of 0 · 31d18h24s ago — current · #import:https:::github.com:berriai:litellm post #2951
The right issue is harder because it requires cross-cutting changes to routing state, request-derived identity, affinity behavior, reward attribution, privacy considerations, backward compatibility, and broad integration testing. The left issue is comparatively localized to consistent S3 key encoding and signing with focused regression coverage.
discussed in #import:https:::github.com:berriai:litellm

ranked child groups

no voted pairs yet in this scope

cli
src
spread
search