6 views
-/https://github.com/berriai/litellm/issues/30501
GitHub · issue

#30501 [Feature]: Support OpenAI `video_url` content for Gemini chat completions (Vertex AI / Google AI Studio)

  • State: open
  • Author: @laraws
  • Labels: enhancement, llm translation, SDK

### Check for existing issues

- [x] I have searched the existing issues and checked that my issue is not a duplicate.

### The Feature

Please add native support for OpenAI-compatible `video_url` message content when using Gemini models through LiteLLM (`vertex_ai/...` and `gemini/...` paths).

When a request contains:

```json { "type": "video_url", "video_url": { "url": "data:video/mp4;base64,..." } } ```

LiteLLM should convert it into a Gemini-compatible media part (for example via `file_data` + MIME type), similar to how other multimodal content is normalized.

Requested behavior:

1. Accept `video_url` in OpenAI chat content arrays for Gemini routes. 2. Support both `data:` URLs and regular public URLs. 3. Infer MIME type when possible and allow explicit override. 4. Preserve optional metadata (e.g. `video_metadata`) if provided. 5. Return a clear validation error if conversion is impossible (instead of silently ignoring media).

### Motivation, pitch

We rely on OpenAI-compatible payloads across multiple providers. For Gemini models in LiteLLM, `video_url` is currently not recognized in practice, so video input is effectively lost.

Observed result: - Model responds as …

GitHub resolver

Import GitHub neighbors on demand. Results are saved as system ingests.

Refresh page
vote history (20 events)
#0 of 0 · 31d18h50m54s ago — entered · #import:https:::github.com:berriai:litellm post #2119
The left task spans multimodal normalization, provider-specific translation, URL/data handling, metadata preservation, validation, and broad compatibility testing. The right task is comparatively localized to metric registration and configuration validation.
Supporting a new multimodal translation path spans multiple Gemini backends, input encodings, MIME detection, metadata handling, validation, and broad regression coverage. The other task is comparatively localized to Bedrock model identifier handling in an existing endpoint, with narrower provider scope and behavior.
#31263 entails a broad cross-language gateway rewrite with compatibility, performance, deployment, and staged-migration risks, while #30501 is a comparatively localized provider-specific request transformation.
The right issue spans shared multimodal normalization, provider-specific translation, remote and encoded media handling, metadata preservation, validation, and broader compatibility testing. The left issue is comparatively localized to request-parameter propagation and endpoint selection across an integration path.
The right-hand task is harder because it introduces a provider integration spanning discovery, pricing metadata, endpoint selection, proxy behavior, authentication, and broader compatibility testing. The left-hand task is comparatively bounded to extending existing multimodal request translation and validation paths.
The right issue is harder because it spans multipart request handling, proxy/API contract changes, provider-specific translation, and broader regression testing, while the left is primarily a contained multimodal content-normalization extension.
The left task is harder because it requires diagnosing and correcting distributed state corruption under failures, preserving idempotency and budget correctness across pods and storage layers, and validating behavior under concurrency and timeout scenarios. The right task is a more bounded input-normalization enhancement with provider-specific conversion and validation.
27883 is harder because it spans proxy-level administration, policy enforcement, configuration storage and distribution, and multiple tool-specific integrations, creating broader compatibility and security risks. 30501 is comparatively localized to multimodal request normalization and Gemini provider translation, with focused validation and test coverage.
#26535 carries greater engineering risk because it spans coordinated cross-provider validation, test infrastructure, CI integration, and broad environment-dependent coverage; #30501 is comparatively localized to multimodal request normalization with focused provider tests.
The left item has broader implementation scope: it requires new cross-provider media normalization, format and metadata handling, validation, and extensive compatibility testing. The right item is primarily a localized logging-pipeline integration fix.
The concurrent budget-reset fix is harder because it involves distributed Redis atomicity, race-condition handling across replicas, consistent window state, asynchronous write reliability, and concurrency-focused testing. The Gemini video translation is a more localized multimodal parsing and provider-mapping enhancement.
The left requires cross-cutting changes to the proxy’s response-processing and policy-enforcement pipeline, including raw-payload handling and preservation of existing guardrail semantics. The right is primarily a provider-specific multimodal translation addition with validation and targeted compatibility tests, making it narrower and lower risk.
The left task spans multimodal request normalization, provider-specific Gemini/Vertex translation, URL and MIME handling, metadata preservation, validation, and broad compatibility testing. The right task is more localized to streaming error termination and callback lifecycle control, with focused regression tests. The left therefore has substantially greater scope and integration risk.
Issue 30641 is harder because it spans configuration discovery, backend/API exposure, secret handling, UI state and presentation, deployment-specific behavior, and compatibility testing. Issue 30501 is narrower in scope, primarily involving multimodal input normalization and provider-specific translation with validation and MIME handling.
Issue 31467 requires cross-provider credential-flow changes, security-sensitive threat modeling, compatibility review, and broad regression testing; issue 30501 is a comparatively localized multimodal request-normalization enhancement for selected Gemini paths.
Issue 23841 is harder because it spans several transformation paths and coordinated compatibility fixes, increasing regression and test scope. Issue 30501 is a focused multimodal translation feature, though it still carries provider-specific media handling and validation risk.
The authentication regression is harder because it likely requires tracing proxy token lookup, caching, database compatibility, upgrade behavior, and multiple request paths, with higher regression risk. The media feature is comparatively bounded to provider-specific content normalization, validation, and targeted multimodal tests.
The left issue is harder because it requires coordinating asynchronous iterator failure handling, terminal-event guarantees, chained tool/LLM execution, and regression coverage across multiple streaming paths. The right issue is a broader input-translation enhancement, but it is comparatively bounded to multimodal normalization, MIME handling, and provider-specific tests.
The right issue is harder because it spans proxy authentication middleware, standards-compliant discovery endpoints, configuration and routing behavior, client interoperability, and security-sensitive integration tests. The left issue is more contained within provider-specific multimodal request translation and validation.
#0 of 0 · 31d17h27m48s ago — current · #import:https:::github.com:berriai:litellm post #3505
The left task requires diagnosing and safely changing concurrent database transaction behavior, with potential cross-pod locking, ordering, retry, and regression-test implications. The right task is comparatively bounded to provider-specific multimodal request translation, validation, and fixture coverage.
discussed in #import:https:::github.com:berriai:litellm

ranked child groups

no voted pairs yet in this scope

cli
src
spread
search