8 views
-/https://github.com/berriai/litellm/issues/31595
GitHub · issue

#31595 [Feature]: ADEPT Router for Template-Based Routing and SLM Distillation from Production Traffic

  • State: open
  • Author: @QuickHawk
  • Labels: enhancement, proxy

### Check for existing issues

- [x] I have searched the existing issues and checked that my issue is not a duplicate.

### The Feature

Introduce an ADEPT router strategy in LiteLLM Proxy to support adaptive delegation of agentic workflows to specialized SLMs trained from real production traffic

The feature adds a new ADEPT deployment type that routes requests by prompt structure, not just model alias. It derives a stable template signature from incoming prompts by masking variable spans and combining that template with the system prompt for per-tool isolation. If a matching trained route exists, it forwards to the template target model. If no match exists, it safely falls back to a configured default model

Each successful interaction is stored as training data in Postgres, linked to its template. At configurable conversation thresholds, LiteLLM can notify an external trainer endpoint. After training finishes, the trainer writes back a target model for that template, and subsequent matching traffic is delegated to the trained SLM with no proxy restart required

The feature also includes operator-facing setup and management in the dashboard.

### Motivation, pitch

Agentic system…

GitHub resolver

Import GitHub neighbors on demand. Results are saved as system ingests.

Refresh page
vote history (6 events)
#0 of 0 · 31d19h37s ago — entered · #import:https:::github.com:berriai:litellm post #1870
31595 is substantially harder because it spans a new routing strategy, prompt/template analysis, persistence, asynchronous trainer integration, dynamic model updates, fallback behavior, and dashboard/operator workflows. 28173 is a localized authorization/model-listing logic correction with focused regression coverage.
#31595 requires a cross-cutting proxy feature spanning routing, persistence, asynchronous training integration, runtime model updates, and dashboard operations, with substantial design and regression risk. #28554 is a narrowly scoped type-definition correction with targeted validation coverage.
The right issue is substantially harder because it spans new proxy routing behavior, prompt-template derivation, persistent training-data workflows, asynchronous trainer coordination, dynamic model updates, fallback semantics, and dashboard operations. The left issue is a comparatively focused SDK/provider integration bug requiring localized diagnosis and compatibility fixes.
The Rust migration is substantially harder because it is a large architectural rewrite requiring broad behavioral parity, performance validation, interoperability, and long-term maintenance across the gateway. The ADEPT work is a complex but bounded feature spanning routing, persistence, training integration, and dashboard operations, with fewer system-wide compatibility risks.
31595 is harder because it requires coordinated runtime, persistence, external-service, model-routing, and dashboard work with substantial integration and behavioral risk. 34241 spans many files and carries legal sensitivity, but is primarily repository auditing, boundary refactoring, and validation rather than a new distributed feature.
#0 of 0 · 31d17h21m35s ago — current · #import:https:::github.com:berriai:litellm post #3525
31595 entails a new adaptive routing lifecycle spanning request analysis, persistent training-data workflows, asynchronous trainer coordination, dynamic model updates, fallback behavior, and dashboard operations. 35109 has substantial cross-provider and security scope, but its core behavior is comparatively configuration and request-routing isolation. The former therefore carries greater architectural and integration risk.
discussed in #import:https:::github.com:berriai:litellm

ranked child groups

no voted pairs yet in this scope

cli
src
spread
search