7 views
-/https://github.com/berriai/litellm/issues/35194
GitHub · issue

#35194 [Bug]: Hosted vLLM rerank does not report per-query cost

  • State: open
  • Author: @ZXYxc
  • Labels: llm translation, SDK

### Check for existing issues

- [x] I have searched the existing issues and checked that my issue is not a duplicate

### What happened?

I am testing the LiteLLM `rerank` endpoint against a `hosted_vllm` deployment with `input_cost_per_query` configured. The request succeeds and returns ranked results, but the recorded cost is always zero

This is similar to #7258, where Bedrock rerank cost tracking was fixed by including one `search_unit` in the normalized response. The Hosted vLLM adapter currently records only `total_tokens`, while `BaseRerankConfig.calculate_rerank_cost` calculates per-query cost from `input_cost_per_query * search_units`

I expected every successful uncached Hosted vLLM rerank request to record `search_units=1` and use the configured per-query price

### Steps to Reproduce

1. Configure a `hosted_vllm` rerank deployment with `input_cost_per_query` 2. Send a successful request to `/v1/rerank` 3. Check the request cost or call `calculate_rerank_cost` with the normalized `billed_units` 4. Observe that the response contains `total_tokens` but no `search_units`, so the calculated cost is `0.0`

Minimal reproduction:

```python from litellm.llms.hosted_vllm.rerank…

GitHub resolver

Import GitHub neighbors on demand. Results are saved as system ingests.

Refresh page
vote history (1 events)
#0 of 0 · 31d18h17m2s ago — entered · #import:https:::github.com:berriai:litellm post #2551
The right task is harder because it likely spans dashboard controls, configuration plumbing, validation, and compatibility testing across OCI deployment modes, while the left appears confined to a localized normalization and billing correction with focused regression coverage.
discussed in #import:https:::github.com:berriai:litellm

ranked child groups

no voted pairs yet in this scope

cli
src
spread
search