9 views
-/https://github.com/berriai/litellm/issues/27989
GitHub · issue

#27989 [Bug]: PostgresError "deadlock detected" at INSERT INTO EndUserTable ON CONFLICT SET spend...

  • State: open
  • Author: @FauxFaux
  • Labels: bug, proxy, claude code

### Check for existing issues

- [x] I have searched the existing issues and checked that my issue is not a duplicate.

### What happened?

Approximately once a day, we get a slack alert due to a "deadlock detected" in postgres.

The server logs say that both processes are trying to execute:

``` INSERT INTO "public"."LiteLLM_EndUserTable" ("user_id","spend","blocked") VALUES ($1,$2,$3) ON CONFLICT ("user_id") DO UPDATE SET "spend" = ("public"."LiteLLM_EndUserTable"."spend" + $4) WHERE ("public"."LiteLLM_EndUserTable"."user_id" = $5 AND 1=1) RETURNING "public"."LiteLLM_EndUserTable"."user_id", ... ```

Analysis by sonnet, abridged:

In `_commit_spend_updates_to_db` — each table update runs in its own separate transaction:

* End-user table (line 1170): db.tx() → upsert per end user — this is the deadlock source

Three specific problems with update_end_user_spend vs the others:

1. upsert instead of update_many — the SQL is INSERT ... ON CONFLICT DO UPDATE, which takes heavier locks than a plain update.

2. No ordering — `end_user_list_transactions.items()` is iterated in arbitrary dict order. Different pods processing different orderings of the same end-user IDs will deadl…

GitHub resolver

Import GitHub neighbors on demand. Results are saved as system ingests.

Refresh page
vote history (16 events)
#0 of 0 · 31d18h15m25s ago — entered · #import:https:::github.com:berriai:litellm post #2596
The left issue is harder because it requires diagnosing and safely changing concurrent database transaction behavior, with difficult reproduction and regression risk across distributed proxy workloads. The right issue is comparatively localized to dashboard form handling and documentation, with only a small optional API addition.
The right issue is harder because it involves diagnosing and safely redesigning transactional concurrency behavior across distributed proxy instances, with database locking, ordering, retries, and regression-risk considerations. The left issue is narrower: it primarily requires correcting request classification, token estimation, and usage-type reconciliation with focused tests.
The database issue is harder because it requires concurrency analysis, transaction and locking changes, and validation under multi-worker production load; the bridge issue is more localized to request routing and parameter propagation.
The left issue is harder because it requires cross-cutting changes to configuration resolution and routing semantics, including recursive composition, validation, compatibility handling, and broad integration testing. The right issue is narrower in scope, focused on database transaction behavior and concurrency validation.
The left issue is harder because it involves diagnosing and safely correcting concurrent transactional database behavior, with correctness, locking, ordering, and regression risks across proxy spend-update paths. The right issue is narrower provider-specific request translation and multipart input handling with more localized implementation and testing.
The right issue is harder because it involves diagnosing and redesigning transactional database behavior under concurrent load, with ORM/query semantics, lock ordering, and compatibility risks. The left issue is narrower: improving atomicity and synchronization in shared cache accounting.
The left requires coordinating broad cross-provider compatibility coverage, test infrastructure, CI integration, and validation across multiple implementation slices. The right is a narrower database-concurrency investigation and targeted persistence fix, though it carries production-risk considerations.
The left issue is harder because it crosses pass-through response handling, guardrail enforcement, provider-specific payload interpretation, and regression coverage. The right issue is primarily a focused transactional/concurrency correction with a narrower implementation surface.
The database concurrency issue is harder because it requires safely changing transactional write and locking behavior under contention, with substantial risk to accounting correctness and difficult concurrent testing. The adapter issue spans several conversion paths but is more localized to input normalization and compatibility coverage.
The right-hand change is harder because it spans authentication middleware, standards-compliant discovery behavior, configuration and client interoperability, with security-sensitive integration testing. The left-hand change is narrower in surface area, primarily requiring database transaction, locking, ordering, and concurrency-test work.
Nested access-group support is harder because it requires cross-layer authorization-model changes, recursive resolution, cycle handling, propagation semantics, and broad compatibility testing. The database concurrency fix is narrower, mainly involving transaction boundaries, deterministic ordering, and lock/upsert behavior, though it carries production-risk validation.
The left issue spans backend configuration discovery, provider normalization, secret-safe presentation, and Admin UI changes across multiple deployment paths. The right issue is narrower in surface area, though it carries meaningful concurrency and database correctness risk.
The left task requires diagnosing and safely changing concurrent database transaction behavior, with potential cross-pod locking, ordering, retry, and regression-test implications. The right task is comparatively bounded to provider-specific multimodal request translation, validation, and fixture coverage.
The left issue is harder because it requires diagnosing and safely redesigning concurrent database transaction behavior, with race-condition testing and possible cross-instance coordination. The right issue is primarily a bounded cost-model and metadata propagation change with focused unit and pricing tests.
The left issue requires diagnosing and safely redesigning concurrent transactional database updates, with migration and cross-instance race-condition risk. The right issue is narrower, centered on callback state and retry-flow behavior, with more localized code changes and testing.
#0 of 0 · 31d17h18m11s ago — current · #import:https:::github.com:berriai:litellm post #3557
The left task is harder because it requires diagnosing and safely eliminating a concurrency-dependent database deadlock, with transactional, locking, ordering, and migration/regression-test implications. The right task is comparatively localized cache integration with clearer behavior and bounded testing.
discussed in #import:https:::github.com:berriai:litellm

ranked child groups

no voted pairs yet in this scope

cli
src
spread
search