18 views
-/https://github.com/berriai/litellm/issues/27852
GitHub · issue

#27852 [Bug]: Ghost models with --num_workers > 1 – Deleted models are not cleared from other workers' local cache (Redis Pub/Sub sync issue)

  • State: open
  • Author: @danbaierlacher
  • Labels: bug, proxy, llm translation

### Check for existing issues

- [x] I have searched the existing issues and checked that my issue is not a duplicate.

### What happened?

**What happened:** When running the LiteLLM Proxy with multiple workers (`--num_workers 4`) and a Redis cache configured, deleting a model added via API (`/model/new`) via the `/model/delete` API does not invalidate the local in-memory cache of the *other* workers.

The worker handling the delete request successfully removes the model from the Postgres DB and its own RAM. However, it seems no Pub/Sub broadcast is sent (or received) via Redis to tell the other workers to drop the model. As a result API calls to `/model/info` start "flapping" randomly, showing the deleted model or hiding it, depending on which worker handles the request.

*Note: This was tested with both Valkey 8.0.1 and official Redis 8.6.3. The issue persists regardless of the Redis distribution.*

**What did you expect to happen?** When `/model/delete` is called, the proxy should broadcast a cache invalidation event via Redis to all other Uvicorn workers immediately, purging the model from all local memories so the Proxy state remains consistent.

### Steps to Reproduce

**1. …

GitHub resolver

Import GitHub neighbors on demand. Results are saved as system ingests.

Refresh page
vote history (6 events)
#0 of 0 · 31d17h52m0s ago — entered · #import:https:::github.com:berriai:litellm post #3212
The left issue requires diagnosing and implementing reliable cross-process state propagation, including Redis pub/sub lifecycle, message handling, race conditions, and multi-worker integration tests. The right issue is primarily an endpoint-default security change with compatibility review and focused authorization/regression tests.
The right issue is harder because it spans distributed worker coordination, cache invalidation semantics, Redis messaging behavior, race conditions, and multi-process testing. The left issue is more localized to request-shape support and provider-specific image translation, with a narrower implementation surface.
Distributed accounting with timeout and retry behavior requires tracing cross-process state, preserving idempotency, and validating reconciliation under failure; the cache-invalidation change is narrower, mainly involving event publication, subscription handling, and worker synchronization.
The right issue spans configuration-resolution semantics, backend exposure, frontend presentation, provider coverage, secret-handling, and compatibility testing. The left issue is a narrower distributed-state synchronization problem, though it carries meaningful concurrency and infrastructure-debugging risk.
The cache-consistency fix is harder because it requires diagnosing and safely coordinating cross-process state invalidation, Redis Pub/Sub delivery, worker lifecycle, race conditions, and multi-worker testing. The provider addition is broader than a simple adapter but can mostly follow established provider, metadata, and pricing integration patterns.
#0 of 0 · 31d17h42m22s ago — current · #import:https:::github.com:berriai:litellm post #3372
Replacing a foundational database client is a broad, cross-cutting migration with compatibility, schema, deployment, and regression risks; the other issue is a more contained distributed-cache synchronization fix.
discussed in #import:https:::github.com:berriai:litellm

ranked child groups

no voted pairs yet in this scope

cli
src
spread
search