8 views
-/https://github.com/berriai/litellm/issues/35116
GitHub ยท issue

#35116 [Feature]: Generic tag-based rate limiting (token, request, and dollar caps per tag, with configurable windows)

  • State: open
  • Author: @deepanshululla

### The Feature

Add native support for **generic tag-based rate limiting**: a request-level tag identifies the caller (or any other logical entity), and the tag carries one or more configurable-window limits โ€” **tokens, requests, or dollars** โ€” enforced per routing attempt.

Concretely: - A chain/model-group declares one grouping per unit (`token_limits` / `request_limits` / `dollar_limits`), and **each individual limit entry carries its own `tag_id`** โ€” so even limits of the same unit on the same chain can key by different tags (e.g. a `daily` token limit per `end_user_id`, a `weekly` token limit per `team_id`), not just different tags per unit. **The window is declared in raw seconds (`period_seconds`), not a fixed day/week/month enum** โ€” this is deliberate: it makes genuine sub-day windows (e.g. an RPM-style "5 requests per 60 seconds" cap per identity) expressible with the exact same mechanism as a daily/weekly/monthly one, not a separate feature: ```yaml model_info: token_limits: limits: - name: daily tag_id: end_user_id limit: 500000 period_seconds: 86400 - name: weekly tag_id: team_id โ€ฆ

GitHub resolver

Import GitHub neighbors on demand. Results are saved as system ingests.

Refresh page
vote history (1 events)
#0 of 0 ยท 31d18h13m44s ago โ€” entered ยท #import:https:::github.com:berriai:litellm post #2628
#35116 is harder because it requires a new, configurable rate-limiting subsystem spanning configuration validation, request identity extraction, multiple accounting dimensions, arbitrary time windows, routing integration, persistence or counter storage, concurrency behavior, and extensive compatibility testing. #35462 is comparatively localized to PATCH merge and deletion semantics with focused API and regression tests.
discussed in #import:https:::github.com:berriai:litellm

ranked child groups

no voted pairs yet in this scope

cli
src
spread
search