6 views
-/https://github.com/berriai/litellm/issues/28427
GitHub · issue

#28427 [Bug]: Prompt caching router affinity TTL is hardcoded to 5 minutes — breaks 1-hour ephemeral cache routing

  • State: open
  • Author: @nuernber
  • Labels: bug, proxy, llm translation

### Check for existing issues

- [x] I have searched the existing issues and checked that my issue is not a duplicate.

### What happened?

LiteLLM's prompt-caching-aware routing (`optional_pre_call_checks: ["prompt_caching"]`) stores the `model_id → cacheable_prefix_hash` binding with a hardcoded 5-minute TTL. This was correct when Anthropic only offered the 5-minute ephemeral cache, but it is now incorrect for the 1-hour cache (`cache_control: {"type": "ephemeral", "ttl": "1h"}`) supported by Anthropic and Bedrock.

In a multi-deployment setup (e.g., the same Bedrock model across multiple AWS accounts/regions, or multiple Anthropic API keys), this means:

1. Request at `t=0` lands on deployment A with `cache_control: {"type": "ephemeral", "ttl": "1h"}`. The provider caches the prefix for 60 minutes. LiteLLM stores `prefix_hash → A` for 5 minutes. 2. Request at `t=6 min` with the same prefix arrives. LiteLLM's affinity entry has expired, so the router falls back to its normal strategy (e.g., least-busy) and may pick deployment B. 3. Deployment B has no cache for the prefix → **provider-side cache miss**, full input token cost, even though deployment A is still holding the cache wa…

GitHub resolver

Import GitHub neighbors on demand. Results are saved as system ingests.

Refresh page
vote history (1 events)
#0 of 0 · 31d18h19m53s ago — entered · #import:https:::github.com:berriai:litellm post #2618
The left issue is harder because it affects shared routing state, cache-lifetime semantics, configuration behavior, and multi-deployment correctness, requiring broader regression coverage. The right issue is comparatively localized to parameter propagation and token-counting tests.
discussed in #import:https:::github.com:berriai:litellm

ranked child groups

no voted pairs yet in this scope

cli
src
spread
search