16 views
-/https://github.com/berriai/litellm/issues/24720
GitHub · issue

#24720 [Bug]: latency-based-routing degrades to random selection due to lost-update race condition in async_log_success_event

  • State: open
  • Author: @hassaan-sage
  • Labels: bug, proxy, llm translation

### Check for existing issues

- [x] I have searched the existing issues and checked that my issue is not a duplicate.

### What happened?

`latency-based-routing` degrades to random selection weighted by deployment count when handling concurrent requests. With 16 Anthropic keys + 4 OpenAI keys per tier, traffic distributes 80/20 matching the key ratio regardless of actual latency differences between providers (e.g., OpenAI GPT-4.1 at ~583ms vs Anthropic Claude Sonnet at ~1,588ms).

**Expected:** Router should favor the faster provider. **Actual:** Traffic distributes proportionally to deployment count — identical to `simple-shuffle`.

**Root cause:** `async_log_success_event()` in `litellm/router_strategy/lowest_latency.py` performs a non-atomic read-modify-write on a shared cache key (`{model_group}_map`). When concurrent requests complete simultaneously, the last writer overwrites all previous updates, causing latency data to be constantly lost. Deployments with lost data fall back to `latency: [0]` (treated as fastest) and are randomly selected, producing a distribution proportional to deployment count.

```python # READ — no lock request_count_dict = await self.router_cache.as…

GitHub resolver

Import GitHub neighbors on demand. Results are saved as system ingests.

Refresh page
vote history (1 events)
#0 of 0 · 31d18h23m25s ago — entered · #import:https:::github.com:berriai:litellm post #2452
The left issue requires diagnosing and safely fixing a concurrent state-update race in latency-aware routing, with cache consistency, async behavior, and regression testing across deployment-selection paths. The right issue is comparatively narrow request-input/header sanitization and error handling.
discussed in #import:https:::github.com:berriai:litellm

ranked child groups

no voted pairs yet in this scope

cli
src
spread
search