16 views
-/https://github.com/berriai/litellm/issues/18801
GitHub · issue

#18801 [Bug]: Streaming + logprobs fails for vLLM-backed models (PydanticSerializationError)

  • State: open
  • Author: @asmaai
  • Labels: bug, llm translation

### What happened?

### Summary When using LiteLLM proxy with vLLM-backed models, enabling both `stream=True` and `logprobs=True` causes the stream to crash with a Pydantic v2 serialization error:

``` TypeError: 'MockValSer' object cannot be converted to 'SchemaSerializer' ```

This does not occur: - when `stream=False` - when `logprobs=False` - when using OpenAI-hosted models via LiteLLM

Direct calls to vLLM do support streaming + logprobs correctly, so the issue appears to be in LiteLLM’s streaming serialization logic.

- Crash happens after emitting the chunk, inside LiteLLM proxy: ``` File "litellm/proxy/proxy_server.py", line 4257 chunk = chunk.model_dump_json(exclude_none=True, exclude_unset=True)

pydantic_core.PydanticSerializationError: TypeError: 'MockValSer' object cannot be converted to 'SchemaSerializer' ```

### Expected Behavior

Streaming responses with `logprobs=True` should work the same for: - OpenAI models - vLLM-backed models Since vLLM itself supports streaming logprobs correctly, LiteLLM should be able to proxy them without crashing.

### Actual Behavior

- First streaming chunk arrives successfully - choice.logprobs.content is present and valid - LiteLLM c…

GitHub resolver

Import GitHub neighbors on demand. Results are saved as system ingests.

Refresh page
vote history (1 events)
#0 of 0 · 31d18h55m48s ago — entered · #import:https:::github.com:berriai:litellm post #2149
The left issue requires tracing and correcting shared streaming serialization across provider-specific response schemas, with compatibility and regression testing for multiple model modes. The right issue is a localized defensive handling and error-mapping change in one provider's streaming transformer.
discussed in #import:https:::github.com:berriai:litellm

ranked child groups

no voted pairs yet in this scope

cli
src
spread
search