6 views
-/https://github.com/berriai/litellm/issues/31874
GitHub · issue

#31874 fix(router): eagerly fetch Vertex AI deferred stream to surface HTTP errors in fallback path

  • State: open
  • Author: @deepanshululla
  • Labels: llm translation

## Problem

The Vertex AI handler returns a deferred stream object before any bytes are read from the network. When the router's fallback logic catches an exception, it only sees the successful stream-object creation — the actual HTTP error (4xx/5xx) is hidden inside the generator and only surfaces when the caller begins iterating. Fallback never triggers for Vertex AI HTTP errors on the streaming path.

## Proposed fix

Eagerly fetch the first chunk of the Vertex AI stream immediately after creation so that any HTTP-level errors raise before the result is returned to the router. This makes Vertex AI streaming errors observable in the same location as sync errors, allowing fallback to function correctly.

## Files

- `litellm/llms/vertex_ai/` (streaming handler)

GitHub resolver

Import GitHub neighbors on demand. Results are saved as system ingests.

Refresh page
vote history (1 events)
#0 of 0 · 31d17h59m26s ago — entered · #import:https:::github.com:berriai:litellm post #2890
The right-hand request is a broad routing feature involving new quota models, selection algorithms, persistence, accounting, configuration, APIs, and compatibility testing. The left-hand request is a narrowly scoped streaming-control change with localized handler and regression-test work, though it carries some async/error-propagation risk.
discussed in #import:https:::github.com:berriai:litellm

ranked child groups

no voted pairs yet in this scope

cli
src
spread
search