8 views
-/https://github.com/berriai/litellm/issues/28125
GitHub · issue

#28125 [Feature]: Support model group composition ("group of groups")

  • State: open
  • Author: @First-Non-Interesting-Username
  • Labels: enhancement, proxy, llm translation

### Check for existing issues

- [x] I have searched the existing issues and checked that my issue is not a duplicate.

### The Feature

Allow a `model_name` entry to reference other model groups as its members, enabling a "group of groups" or model group composition pattern.

### Motivation, pitch

Currently, if you want a single top-level group (e.g. `fast`) that load-balances across multiple named sub-groups with different providers and rate limits, you have to duplicate every deployment under both the sub-group name and the top-level name. In a real config this adds up fast, in my case, around 1000 lines of duplication.

Example of what's currently required:

```yaml # Sub-group 1: two providers for GPT-OSS-120b - model_name: fast/GPT-OSS-120b litellm_params: model: groq/openai/gpt-oss-120b rpm: 30 order: 2 - model_name: fast/GPT-OSS-120b litellm_params: model: cerebras/gpt-oss-120b rpm: 5 order: 1

# Sub-group 2 - model_name: fast/glm-4.7 litellm_params: model: cerebras/zai-glm-4.7 rpm: 5

# To have a "fast" umbrella you must duplicate everything again: - model_name: fast litellm_params: model: groq/openai/gpt-oss-120b rpm: 30 …

GitHub resolver

Import GitHub neighbors on demand. Results are saved as system ingests.

Refresh page
vote history (16 events)
#0 of 0 · 31d17h47m42s ago — entered · #import:https:::github.com:berriai:litellm post #3084
Nested group composition has broader architectural scope, affecting configuration resolution, routing semantics, validation, and compatibility testing; the other is comparatively localized to data retrieval and UI presentation.
Model-group composition is harder because it affects core proxy configuration parsing, deployment expansion, routing, load balancing, limits, validation, backward compatibility, and broad regression-test coverage. The Bedrock change is comparatively localized to multipart image-edit translation and provider-specific parameter handling.
29320 is harder because it spans new runtime processing, configuration layers, observability, dependency management, failure handling, and SDK/proxy compatibility. 28125 is primarily a routing and configuration-model change, with recursion, validation, and backward-compatibility risks but a narrower surface area.
The left issue is harder because it spans a new user-facing CLI workflow, credential handling, remote capability discovery, interactive selection, and integrations with several external agent tools across environments. The right issue is a substantial routing and configuration change, but it is more contained within LiteLLM's existing model-group and deployment-selection architecture.
The left requires diagnosing and hardening distributed accounting across Redis, database reconciliation, retries, and timeout behavior, with significant concurrency and regression risk. The right is primarily a routing/configuration abstraction that needs defined resolution and compatibility tests, but has a narrower implementation surface.
The left task has substantially greater architectural risk: it affects model resolution, routing semantics, recursive dependency handling, validation, compatibility, and broad regression coverage. The right task is comparatively contained to administrative configuration delivery and provider-specific integration surfaces.
The left is harder because it requires cross-cutting changes to persistence, pricing/version selection, billing semantics, repeatable execution, auditability, and operational safety. The right is mainly a configuration-resolution and routing-graph feature, with recursion, validation, and compatibility risks but a narrower runtime scope.
Model-group composition is harder because it changes core deployment resolution and routing semantics, requiring recursive expansion, compatibility handling, and careful interaction with load balancing and rate limits. The budget-window change is substantial but can largely extend existing persistence, reservation, and reset mechanisms to another scope.
31824 is harder due to cross-cutting authorization, privacy isolation, usage aggregation, API design, and dashboard work across backend and frontend surfaces; 28125 is substantial routing/configuration work but is more concentrated in core model-group resolution and validation.
Extending the proxy with user-defined routing is broader and riskier: it requires a stable extension contract, lifecycle and concurrency handling, validation, observability, failure semantics, and compatibility across routing paths. The other change is comparatively localized to configuration resolution and deployment expansion.
Monthly quota enforcement spans persistent schema changes, atomic/distributed usage accounting, calendar-time and timezone edge cases, request-path integration, API/UI exposure, and backward compatibility. The composition change is primarily configuration parsing and model-group resolution, though it still carries routing and validation risks.
28125 requires deeper changes to deployment resolution and routing semantics, including nested expansion, inheritance, validation, and edge-case handling across existing configuration paths. 33371 is broader in API design and classification consistency, but can be implemented incrementally around existing cooldown and error-handling state.
29452 is harder because it requires a security-sensitive, cross-cutting authentication abstraction with storage, lifecycle, provider compatibility, migration, and extensive validation concerns. 28125 is narrower in scope, mainly involving configuration resolution, composition semantics, cycle handling, and routing tests.
31467 is harder because it requires a security-sensitive, cross-provider audit and coordinated changes to credential resolution, endpoint trust boundaries, compatibility behavior, and regression coverage. 28125 is a broader routing feature, but its behavior can be contained within configuration expansion and model-group selection mechanisms.
The left issue is harder because it requires cross-cutting changes to configuration resolution and routing semantics, including recursive composition, validation, compatibility handling, and broad integration testing. The right issue is narrower in scope, focused on database transaction behavior and concurrency validation.
#0 of 0 · 31d17h16m56s ago — current · #import:https:::github.com:berriai:litellm post #3574
Model-group composition requires cross-referencing and resolving nested configuration, defining routing and inheritance semantics, preventing cycles, and updating validation and broad test coverage. The other change is more localized to initialization readiness, retry handling, and recovery behavior.
discussed in #import:https:::github.com:berriai:litellm

ranked child groups

no voted pairs yet in this scope

cli
src
spread
search