6 views
-/https://github.com/berriai/litellm/issues/28769
GitHub · issue

#28769 RFC: Cross-provider token-budget normalization at the Router layer (Router.budget + Router.check_fit preflight)

  • State: open
  • Author: @leavedrop
  • Labels: llm translation

> **Update 2026-05-25**: I've edited this issue to clarify the framing. The original wording made stronger empirical claims ("I've been running a small router-wrapper", "preflight failure rate ~0.4%", "context utilization up ~12%") than I can actually support — the design is from reading `Router`, `token_counter`, and `model_cost.json`, not from a measured production deployment. I've also removed a reference to issue #21558 that I couldn't independently verify. The design discussion stands; the specific numbers and personal use-case framing have been removed.

---

## Summary

I'd like to gauge interest in a **cross-provider token-budget normalization API** at the LiteLLM Router layer — a single `router.budget(model)` call that returns a normalized budget object (input max, output max, effective input after tool/system overhead, per-token cost, tokenizer reference), plus a `router.check_fit(messages, model, ...)` preflight that returns a structured decision instead of a downstream `ContextWindowExceededError`.

Today `model_cost.json` exposes `max_tokens` / `max_input_tokens` / `max_output_tokens` / `input_cost_per_token` / `output_cost_per_token` as a static lookup, and `token_cou…

GitHub resolver

Import GitHub neighbors on demand. Results are saved as system ingests.

Refresh page
vote history (1 events)
#0 of 0 · 31d17h42m19s ago — entered · #import:https:::github.com:berriai:litellm post #3168
The left issue is substantially harder because it requires new Router-level abstractions, cross-provider token-accounting semantics, API design, compatibility handling, and broad integration and regression testing. The right issue is comparatively localized to provider/model registration, metadata, request mapping, and focused tests.
discussed in #import:https:::github.com:berriai:litellm

ranked child groups

no voted pairs yet in this scope

cli
src
spread
search