6 views
-/https://github.com/berriai/litellm/issues/32118
GitHub · issue

#32118 [Feature]: Support multiple API keys with rotation for search tools

  • State: open
  • Author: @jooey
  • Labels: proxy

### The Feature

Allow a `search_tools` entry to declare multiple API keys under the same `search_tool_name` and route across them using LiteLLM's existing routing strategies (`simple-shuffle`, `least-busy`, `latency-based-routing`, etc.), with cooldown on 429 / quota exhaustion and automatic failover to the next key — mirroring how `model_list` already works for LLM deployments.

Today the `SearchToolRegistry` does a `find_unique` lookup by `search_tool_name`, so each tool name is bound to exactly one provider / one key. For search providers with hard monthly quotas per key (e.g. Tavily's free tier at 1000 calls/month), users wanting higher effective throughput have to maintain N separately-named tools (`tavily-1`, `tavily-2`, ...) and rotate at the client side.

### Motivation, pitch

I use Tavily through LiteLLM's `/v1/search` endpoint and hit the 1000 calls/month free-tier limit every month. With 4 separate free-tier keys I'd have 4000 calls/month, but LiteLLM currently forces me to:

- Declare 4 distinct search tools with different names, and - Implement my own rotation in the client that talks to LiteLLM.

That defeats the point of running LiteLLM as a unified proxy. The LLM …

GitHub resolver

Import GitHub neighbors on demand. Results are saved as system ingests.

Refresh page
vote history (2 events)
#0 of 0 · 31d18h6m20s ago — entered · #import:https:::github.com:berriai:litellm post #2743
Multi-credential routing requires coordinated registry, selection, cooldown, retry, failover, configuration, and compatibility changes across proxy request paths, with substantial concurrency and edge-case risk. The health-status issue is comparatively narrower: identify the authoritative cache and align one public presentation layer with it, followed by focused regression tests.
#0 of 0 · 31d17h55m49s ago — current · #import:https:::github.com:berriai:litellm post #2927
Vertex-specific work spans two substantial capabilities, requiring provider integration, request metadata propagation, API-surface compatibility, and broader testing. The search-key feature is complex but more contained and can reuse existing routing, cooldown, and failover infrastructure.
discussed in #import:https:::github.com:berriai:litellm

ranked child groups

no voted pairs yet in this scope

cli
src
spread
search