7 views
-/https://github.com/berriai/litellm/issues/30756
GitHub · issue

#30756 Rerank API drops instruction field (supported by vLLM/Qwen3-Reranker)

  • State: open
  • Author: @jhsmith409
  • Labels: llm translation

### What happened

LiteLLM's managed rerank route silently drops the top-level `instruction` field, so it never reaches providers that support it (e.g. vLLM serving Qwen3-Reranker).

vLLM's `/v1/rerank` and `/v1/score` accept an optional top-level `instruction` string. It's folded into `chat_template_kwargs` and consumed by the model's chat template: - `vllm/entrypoints/pooling/scoring/protocol.py` - (this dash should be an em-dash but using em-dashes often triggers AI warnins so I went back and edited it and added this comment) `instruction` on the request schema - `vllm/entrypoints/pooling/io_processor.py` - (this dash should be an em-dash but using em-dashes often triggers AI warnins so I went back and edited it and added this comment) folds it into `chat_template_kwargs`

In LiteLLM, `litellm/types/rerank.py` `RerankRequest` and `OptionalRerankParams` have no `instruction` field, and the hosted_vllm transformation rebuilds the outgoing body from those types - (this dash should be an em-dash but using em-dashes often triggers AI warnins so I went back and edited it and added this comment) so the field is dropped before the request leaves the proxy.

### Why this matters

`instru…

GitHub resolver

Import GitHub neighbors on demand. Results are saved as system ingests.

Refresh page
vote history (1 events)
#0 of 0 · 31d17h49m27s ago — entered · #import:https:::github.com:berriai:litellm post #3033
The right-hand task is harder because it likely requires coordinating proxy rate-limit accounting with fallback orchestration, retry semantics, and broad regression coverage. The left-hand task is a comparatively localized schema and provider-translation change with narrower testing scope.
discussed in #import:https:::github.com:berriai:litellm

ranked child groups

no voted pairs yet in this scope

cli
src
spread
search