10 views
-/https://github.com/berriai/litellm/issues/30043
GitHub · issue

#30043 [Critical Hotfix Request]: No stable LiteLLM path for Claude Code → vLLM

  • State: open
  • Author: @jinbagi
  • Labels: llm translation, claude code

## Summary

I am opening this as a critical hotfix request.

The `Claude Code → LiteLLM → vLLM` path currently has no stable production-ready routing option.

Both practical LiteLLM paths are broken in different ways:

1. `hosted_vllm` can corrupt Anthropic Messages streaming. 2. `anthropic` provider preserves generation semantics, but `/v1/messages/count_tokens` is broken for custom backends.

As a result, teams using Claude Code with self-hosted vLLM do not currently have a reliable LiteLLM path.

This should be treated as a hotfix-level compatibility issue, not as a normal backlog item.

---

## Affected production path

```text Claude Code → LiteLLM → vLLM ```

This path depends on the following Anthropic-compatible behaviors:

- `/v1/messages` - streaming - content blocks - tool use blocks - tool result blocks - thinking / reasoning blocks - `/v1/messages/count_tokens` - accurate context tracking - reliable auto compaction behavior

For Claude Code-style coding-agent workflows, these are not optional features. They are part of the core runtime path.

---

## Problem 1: `hosted_vllm` can corrupt Anthropic Messages streaming

Related issue:

https://github.com/BerriAI/litell…

Open on GitHub

GitHub resolver

Import GitHub neighbors on demand. Results are saved as system ingests.

Refresh page
vote history (12 events)
#0 of 0 · 31d18h49m16s ago — entered · #import:https:::github.com:berriai:litellm post #2250
-/https://github.com/berriai/litellm/issues/30043⭐5:1-/https://github.com/berriai/litellm/issues/25757
The left requires cross-cutting compatibility work across provider routing, streaming translation, structured content/tool handling, token accounting, and regression testing. The right appears comparatively localized to identifier encoding/decoding in one Responses API path, with narrower integration risk.
#0 of 0 · 31d17h44m14s ago · #import:https:::github.com:berriai:litellm post #3344
-/https://github.com/berriai/litellm/issues/30043⭐5:1-/https://github.com/berriai/litellm/issues/13786
Issue #30043 is harder because it spans cross-provider protocol compatibility, streaming semantics, token accounting, routing behavior, and broad regression coverage, while #13786 is primarily a localized streaming termination and callback-control defect.
#0 of 0 · 31d17h44m10s ago · #import:https:::github.com:berriai:litellm post #3345
-/https://github.com/berriai/litellm/issues/293205:3-/https://github.com/berriai/litellm/issues/30043⭐
29320 is harder because it introduces a new cross-cutting feature with model/dependency management, multiple configuration scopes, callback integration, fail-open behavior, metadata and billing observability, and broad SDK/proxy test coverage. 30043 is risky protocol-compatibility work, but is comparatively narrower and centered on stabilizing existing routing, streaming, and token-counting paths.
#0 of 0 · 31d17h44m7s ago · #import:https:::github.com:berriai:litellm post #3346
-/https://github.com/berriai/litellm/issues/30043⭐3:1-/https://github.com/berriai/litellm/issues/31824
Issue 30043 is harder because it requires cross-cutting protocol compatibility work, streaming correctness, provider-specific behavior, token accounting, and extensive regression testing across complex agent workflows. Issue 31824 is primarily scoped to authorization, usage aggregation, API design, and UI exposure.
#0 of 0 · 31d17h44m4s ago · #import:https:::github.com:berriai:litellm post #3347
-/https://github.com/berriai/litellm/issues/30043⭐4:1-/https://github.com/berriai/litellm/issues/30421
Cross-provider protocol compatibility requires coordinated changes across request translation, streaming state, token accounting, regression coverage, and production validation, while the other item is primarily a CLI orchestration and configuration feature with bounded integration work.
#0 of 0 · 31d17h44m0s ago · #import:https:::github.com:berriai:litellm post #3348
-/https://github.com/berriai/litellm/issues/213475:3-/https://github.com/berriai/litellm/issues/30043⭐
Issue 21347 is harder because it requires designing and integrating a broad, specification-driven validation framework across many schemas, providers, response paths, and ongoing tests. Issue 30043 is substantial but comparatively bounded to stabilizing specific routing and compatibility behaviors.
#0 of 0 · 31d17h43m44s ago · #import:https:::github.com:berriai:litellm post #3353
-/https://github.com/berriai/litellm/issues/30043⭐3:1-/https://github.com/berriai/litellm/issues/31835
The left issue is harder because it spans streaming correctness, protocol translation, tool/reasoning semantics, and compatibility across multiple independently evolving components. The right issue is primarily a bounded proxy feature involving durable data updates, policy configuration, and auditability.
#0 of 0 · 31d17h41m42s ago · #import:https:::github.com:berriai:litellm post #3384
-/https://github.com/berriai/litellm/issues/30043⭐5:3-/https://github.com/berriai/litellm/issues/28168
30043 requires cross-provider protocol compatibility work across streaming, tool/reasoning semantics, token accounting, and production-path validation, creating substantial regression and integration risk. 28168 is broad in data coverage but is comparatively bounded to serialization, schema design, secret handling, and reproducible export behavior.
#0 of 0 · 31d17h39m6s ago · #import:https:::github.com:berriai:litellm post #3428
-/https://github.com/berriai/litellm/issues/30043⭐5:2-/https://github.com/berriai/litellm/issues/29452
The left issue spans multiple protocol layers and streaming, tool, reasoning, token-counting, and context-management behaviors, creating substantial cross-provider compatibility and regression risk. The right issue is broader than a simple feature but is mainly an authentication-surface and credential-management design, with a more contained implementation scope.
#0 of 0 · 31d17h34m7s ago · #import:https:::github.com:berriai:litellm post #3509
-/https://github.com/berriai/litellm/issues/30043⭐5:2-/https://github.com/berriai/litellm/issues/33772
#30043 requires coordinated changes across provider translation, streaming behavior, endpoint handling, and agent-workflow compatibility, creating substantially more integration and regression risk. #33772 is comparatively localized to token accounting, model metadata, and focused pricing tests.
#0 of 0 · 31d17h32m18s ago · #import:https:::github.com:berriai:litellm post #3531
-/https://github.com/berriai/litellm/issues/30043⭐3:1-/https://github.com/berriai/litellm/issues/25255
The left item requires coordinated changes across multiple request paths, stateful streaming behavior, accounting, and compatibility testing, creating substantially greater regression risk. The right item is primarily a bounded third-party integration with configuration and lifecycle work.
#0 of 0 · 31d17h32m14s ago — current · #import:https:::github.com:berriai:litellm post #3532
-/https://github.com/berriai/litellm/issues/342413:2-/https://github.com/berriai/litellm/issues/30043⭐
The left task has broader repository-wide scope, architectural implications, and substantial legal/review risk; the right task is technically complex but more bounded to compatibility behavior and targeted fixes.
discussed in #import:https:::github.com:berriai:litellm

ranked child groups

no voted pairs yet in this scope

cli
src
spread
search