16 views
-/https://github.com/berriai/litellm/issues/27928
GitHub · issue

#27928 [Bug]: Vertex Gemini web search streaming crashes on 3/3.1 Flash / Flash Lite models with empty choices chunk

  • State: open
  • Author: @takumamitarai
  • Labels: bug, llm translation, SDK

### Check for existing issues

- [x] I have searched the existing issues and checked that my issue is not a duplicate.

### What happened?

## What happened

When using Vertex AI Gemini with `stream=True` and `web_search_options={}`, LiteLLM fails mid-stream with:

```text litellm.MidStreamFallbackError: litellm.APIConnectionError: list index out of range ```

Confirmed failing models:

```text vertex_ai/gemini-3.1-flash-lite-preview vertex_ai/gemini-3.1-flash-lite vertex_ai/gemini-3-flash-preview ```

The same request pattern worked with:

```text vertex_ai/gemini-3.1-pro-preview ```

So this appears to affect Gemini Flash / Flash Lite streaming response shapes with web search, rather than all Gemini 3.x models.

## Root cause hypothesis

The failure appears to happen inside LiteLLM's streaming pipeline.

Call flow:

```text litellm.main.acompletion() -> litellm.main.completion() -> litellm.utils.get_optional_params() -> VertexGeminiConfig.map_openai_params() -> VertexGeminiConfig._map_web_search_options() -> VertexLLM.completion() -> VertexLLM.async_streaming() -> make_call() -> ModelResponseIterator.chunk_parser() -> CustomStreamWrapper.chunk_creator() -> C…

GitHub resolver

Import GitHub neighbors on demand. Results are saved as system ingests.

Refresh page
vote history (2 events)
#0 of 0 · 31d18h42m46s ago — entered · #import:https:::github.com:berriai:litellm post #2360
The left issue is harder because it spans asynchronous streaming, provider-specific response normalization, mid-stream error handling, and compatibility across multiple model variants. The right issue is narrower: extending an existing guardrail payload transformation to support another content modality, with focused validation and tests.
#0 of 0 · 31d17h55m4s ago — current · #import:https:::github.com:berriai:litellm post #3160
The right issue is harder because it requires diagnosing provider-specific streaming behavior across asynchronous parsing, fallback handling, and regression coverage, whereas the left issue is primarily an instrumentation change with defined timing data and metric integration.
discussed in #import:https:::github.com:berriai:litellm

ranked child groups

no voted pairs yet in this scope

cli
src
spread
search