6 views
-/https://github.com/berriai/litellm/issues/29064
GitHub · issue

#29064 upsert_deployment does not clean up stale entries in team_pattern_routers / pattern_router on model update

  • State: open
  • Author: @dibyom
  • Labels: llm translation

### Check for existing issues

- [x] I have searched the existing issues and checked that my issue is not a duplicate.

### What happened?

When a wildcard deployment's `litellm_params` are updated (e.g. correcting the `model` field via `PATCH /model/{id}/update`), `upsert_deployment` removes the old deployment from `model_list` but does **not** remove the stale entry from `team_pattern_routers` or `pattern_router`. The subsequent `add_deployment` call appends a new entry alongside the stale one. The router then round-robins between the correct and stale deployments, causing **intermittent routing failures**.

We discovered this in production after patching a team BYOK wildcard model (`openai/*`) that had a double-prefixed `litellm_params.model` (`openai/openai/*`). After the DB fix via `PATCH /model/{id}/update`, ~50% of requests still failed because stale deployment entries persisted in the in-memory pattern router across all replicas. Only a full pod restart resolved the issue.

**Root Cause:**

`upsert_deployment` (router.py:7910) handles updates by: 1. Removing the old deployment from `self.model_list` (line 7945) 2. Calling `self.add_deployment(deployment)` (line 7953) → `_ad…

GitHub resolver

Import GitHub neighbors on demand. Results are saved as system ingests.

Refresh page
vote history (1 events)
#0 of 0 · 31d18h16m17s ago — entered · #import:https:::github.com:berriai:litellm post #2572
The right issue is harder because it requires tracing and preserving a protocol-level streaming lifecycle across proxy and provider translation paths, with ordering, compatibility, and end-to-end streaming tests. The left issue is more localized to deployment replacement and router-state cleanup, though it still carries consistency and regression risk.
discussed in #import:https:::github.com:berriai:litellm

ranked child groups

no voted pairs yet in this scope

cli
src
spread
search