7 views
-/https://github.com/berriai/litellm/issues/29284
GitHub ยท issue

#29284 [Bug] Embedding cache merge corrupts `data[*].index` on partial cache hits

  • State: open
  • Author: @AFLexCom
  • Labels: llm translation

## What Happened

When an embedding request is partly served from the cache and partly forwarded to the upstream provider, the merged response returned to the client contains duplicate and wrong `index` values in `data[*]`. The duplicates cause downstream clients (e.g. our Java ETL using Apache HttpClient) to reject the response as malformed.

The cache write path keeps the indices correct, and the DB spend log re-enumerates them โ€” so this bug is invisible in `LiteLLM_SpendLogs.response`, only the client-visible response is broken.

## Affected Versions

Reproduced on **v1.86.2** and confirmed still present on `main` (HEAD `f27df8d` at time of writing).

## Root Cause

`litellm/caching/caching_handler.py`, function `_combine_cached_embedding_response_with_api_result` (lines 590โ€“640).

Cache hits are inserted with the **final position** as `index` (correct, see `_process_async_embedding_cached_response` at lines 444โ€“448):

```python final_embedding_cached_response.data[idx] = Embedding( embedding=embedding_data, index=idx, # position-based โ€” correct object="embedding", ) ```

But when merging the provider's response, items are taken verbatim with their **sub-โ€ฆ

GitHub resolver

Import GitHub neighbors on demand. Results are saved as system ingests.

Refresh page
vote history (1 events)
#0 of 0 ยท 31d18h43m44s ago โ€” entered ยท #import:https:::github.com:berriai:litellm post #2242
#31842 is harder because it likely requires tracing and correcting cross-cutting budget state, runtime enforcement, and customer/model scoping, with broader regression and integration testing. #29284 appears comparatively localized to a deterministic response-merging path with a focused fix and test surface.
discussed in #import:https:::github.com:berriai:litellm

ranked child groups

no voted pairs yet in this scope

cli
src
spread
search