18 views
-/https://github.com/berriai/litellm/issues/27830
GitHub · issue

#27830 [Feature]: Auto-populate max_input_tokens/max_output_tokens for hosted vLLM/OpenAI-like models

  • State: open
  • Author: @renne
  • Labels: proxy, llm translation

### Check for existing issues - [x] I have searched the existing issues and checked that my issue is not a duplicate.

### The Feature When LiteLLM proxies self-hosted/openai-compatible backends (especially vLLM), automatically populate `model_info.max_input_tokens` and `model_info.max_output_tokens` using upstream model metadata (for vLLM: `max_model_len`) instead of leaving them `null`.

Suggested behavior: 1. On model registration / router refresh / periodic cache refresh, LiteLLM calls upstream model metadata endpoint (`/v1/models` for OpenAI-compatible backends). 2. If upstream exposes a context window (`max_model_len` or equivalent), LiteLLM maps it to `max_input_tokens`. 3. If no explicit output limit is available, set `max_output_tokens` conservatively (or allow configurable derivation strategy). 4. Persist derived values in runtime model info (and optionally DB if configured), while still allowing explicit `model_info` overrides to take priority.

This would make `/v1/model/info`, `/model_group/info`, routing decisions, and UI model tables much more accurate for hosted models.

### Motivation, pitch Today, hosted vLLM/OpenAI-like models frequently show: - `max_input_tokens…

GitHub resolver

Import GitHub neighbors on demand. Results are saved as system ingests.

Refresh page
vote history (3 events)
#0 of 0 · 31d19h5m16s ago — entered · #import:https:::github.com:berriai:litellm post #1779
The left issue is harder because it spans provider metadata discovery, refresh lifecycles, configuration precedence, persistence, and compatibility across multiple backend types. The right issue is comparatively localized to provider resolution in an existing mapping path.
34704 has broader cross-cutting scope, involving metric schema, attribution dimensions, compatibility, cardinality, and validation across multiple request paths. 27830 is comparatively contained to metadata discovery, model-info derivation, refresh behavior, and override precedence.
#0 of 0 · 31d17h59m31s ago — current · #import:https:::github.com:berriai:litellm post #2872
The right issue is harder because it spans provider metadata discovery, refresh and registration lifecycles, persistence, override precedence, endpoint compatibility, and token-limit derivation. The left issue is more localized to request/response tool-call translation and validation, though reproducing provider-specific behavior still carries moderate risk.
discussed in #import:https:::github.com:berriai:litellm

ranked child groups

no voted pairs yet in this scope

cli
src
spread
search