#27989 [Bug]: PostgresError "deadlock detected" at INSERT INTO EndUserTable ON CONFLICT SET spend...
### Check for existing issues
- [x] I have searched the existing issues and checked that my issue is not a duplicate.
### What happened?
Approximately once a day, we get a slack alert due to a "deadlock detected" in postgres.
The server logs say that both processes are trying to execute:
``` INSERT INTO "public"."LiteLLM_EndUserTable" ("user_id","spend","blocked") VALUES ($1,$2,$3) ON CONFLICT ("user_id") DO UPDATE SET "spend" = ("public"."LiteLLM_EndUserTable"."spend" + $4) WHERE ("public"."LiteLLM_EndUserTable"."user_id" = $5 AND 1=1) RETURNING "public"."LiteLLM_EndUserTable"."user_id", ... ```
Analysis by sonnet, abridged:
In `_commit_spend_updates_to_db` — each table update runs in its own separate transaction:
* End-user table (line 1170): db.tx() → upsert per end user — this is the deadlock source
Three specific problems with update_end_user_spend vs the others:
1. upsert instead of update_many — the SQL is INSERT ... ON CONFLICT DO UPDATE, which takes heavier locks than a plain update.
2. No ordering — `end_user_list_transactions.items()` is iterated in arbitrary dict order. Different pods processing different orderings of the same end-user IDs will deadl…