6 views
-/https://github.com/berriai/litellm/issues/34165
GitHub · issue

#34165 [Bug]: NVIDIA NIM rerank: image documents flattened to text; top_n sent as unsupported top_k

  • State: open
  • Author: @atomic
  • Labels: llm translation

### What happened?

The NVIDIA NIM rerank transform (`litellm/llms/nvidia_nim/rerank/transformation.py`) has two defects that break reranking against NVIDIA's native `/v1/ranking` API, in particular for the vision-language reranker [`nvidia/llama-nemotron-rerank-vl-1b-v2`](https://build.nvidia.com/nvidia/llama-nemotron-rerank-vl-1b-v2):

1. **Structured image documents are serialized into text passages.** A document like `{"image": "data:image/jpeg;base64,..."}` is `json.dumps`-stringified into `{"text": "{\"image\": \"data:image/jpeg;base64,...\"}"}` before it reaches the provider, so the VL reranker scores a blob of JSON text instead of the image. The request still returns HTTP 200, which makes this a silent correctness bug — relevance scores track a text-control baseline rather than the image content. 2. **Cohere-compatible `top_n` is translated to `top_k`, which the native `/v1/ranking` endpoint rejects.** The native ranking API accepts only `model`, `query`, `passages`, and `truncate`; sending `top_k` fails the whole request with HTTP 400: `top_k: unknown field, expected one of model, query, passages, truncate`.

Both bugs live in the same transform. `NvidiaNimRankingConfig` (…

GitHub resolver

Import GitHub neighbors on demand. Results are saved as system ingests.

Refresh page
vote history (1 events)
#0 of 0 · 31d18h8m59s ago — entered · #import:https:::github.com:berriai:litellm post #2694
The left issue requires coordinated request-schema and multimodal serialization changes in a provider-specific transformation, with compatibility handling and regression coverage for both silent correctness and API rejection paths. The right issue is primarily a dependency constraint update with packaging validation, so the left issue carries substantially greater implementation and testing risk.
discussed in #import:https:::github.com:berriai:litellm

ranked child groups

no voted pairs yet in this scope

cli
src
spread
search