6 views
-/https://github.com/berriai/litellm/issues/26535
GitHub · issue

#26535 [QA] Claude Code Compatibility Matrix — full-stack QA plan (PRD #26476)

  • State: open
  • Author: @mateo-berri
  • Labels: llm translation, stale, claude code

# QA Plan — Claude Code Compatibility Matrix (PRD #26476)

This is a step-by-step QA plan for the full stack landing on branch `sandcastle/compat-matrix-stack`, which implements PRD #26476.

The branch is a 5-commit stack (slices 1–5 of the PRD). Use the section headings below as a checklist; each section maps to a discrete piece of the implementation and can be QA'd independently.

**Branch under test:** `sandcastle/compat-matrix-stack` **Parent:** `litellm_internal_staging` **Related PRs:** #26477, #26478, #26479, #26480, #26481

## What landed on this branch (at a glance)

| Slice | PR | What it adds | |------|------|---| | 1 | #26477 | Tracer-bullet: `tests/claude_code/` skeleton, `manifest.yaml`, `cli_driver.py`, `conftest.py` (`compat_result` fixture + path-inference hook), `matrix_builder.py`, sample 1×5 JSON, driver/builder unit tests | | 2 | #26478 | First real feature row: `basic_messaging_non_streaming` × 5 providers (Anthropic, Bedrock Invoke, Bedrock Converse, Vertex AI, Azure-as-N/A), proxy `test_config.yaml` with model aliases | | 3 | #26479 | CircleCI PR-gate job `claude_code_compat_pr_gate`, `pr_gate_version_resolver.py` (latest-minus-3-days Claude Code), structura…

GitHub resolver

Import GitHub neighbors on demand. Results are saved as system ingests.

Refresh page
vote history (30 events)
#0 of 0 · 31d18h47m20s ago — entered · #import:https:::github.com:berriai:litellm post #2094
The left issue spans a multi-provider compatibility framework, CI integration, test infrastructure, and broad end-to-end validation, creating substantially greater coordination and regression risk. The right issue is a more localized MCP proxy pagination defect, likely requiring targeted forwarding logic and focused tests.
The compatibility-matrix work is harder because it spans broad test infrastructure, multiple provider paths, client-behavior coverage, fixtures, and CI integration, creating substantially more coordination and regression risk than adding one provider with metadata, routing, and cost integration.
#34241 is harder because it requires repository-wide architectural and licensing-boundary changes, disentangling shared enforcement from product implementations while preserving behavior and compliance. #26535 is substantial QA and CI work, but it is more bounded and primarily extends test coverage and automation around an existing implementation.
The right issue is harder because it involves diagnosing distributed state consistency, timeout behavior, idempotency, and multi-pod budget accounting, followed by reliable regression coverage. The left issue is broad but primarily structured QA, matrix validation, and CI coordination.
The managed-settings feature is harder because it requires new server-side configuration flows, policy enforcement, compatibility across multiple client ecosystems, secure administration, and broad integration testing. The compatibility-matrix item is primarily validation and release-gate work over an already implemented multi-commit stack, making it less implementation-intensive despite its testing breadth.
The left task spans a multi-provider compatibility matrix, test harness, proxy configuration, CI gating, version resolution, and substantial end-to-end validation. The right task is a focused SDK/bridge regression involving a narrower Azure request path, though it carries some integration and regression-testing risk.
Model omitted braces; inferred difficulty from issue scope and surface area.
The left item spans coordinated validation across multiple implementation slices, providers, proxy configuration, test infrastructure, and CI gating, creating substantially broader integration and maintenance risk. The right item is a narrower routing enhancement, though external-service availability, caching, security, and failure handling add moderate risk.
#26535 carries greater engineering risk because it spans coordinated cross-provider validation, test infrastructure, CI integration, and broad environment-dependent coverage; #30501 is comparatively localized to multimodal request normalization with focused provider tests.
26535 is harder because it spans broader test infrastructure, multiple provider integrations, CI orchestration, and version-dependent coverage; 30053 is comparatively localized to one streaming execution path with targeted regression testing.
The right issue is harder because it requires cross-cutting production changes in the request path, configuration, failure handling, and observability, while the left is primarily broad QA and CI validation of an existing implementation stack.
Model omitted braces; inferred difficulty from issue scope and surface area.
The right issue is harder because it spans authenticated API design, strict tenant-level authorization, usage aggregation and accounting semantics, backward compatibility, and dashboard/UI integration. The left issue is primarily a structured validation effort against an already-defined multi-commit compatibility stack, with substantial test coverage but lower product and architectural risk.
The left task is harder because it requires new stateful quota infrastructure, calendar-window semantics, concurrency-safe enforcement, persistence, API/UI changes, and broad regression coverage. The right task is primarily validation and test automation over an existing multi-provider stack, with less production-surface risk.
The left task spans a multi-provider compatibility test framework, CI integration, version handling, proxy configuration, and broad end-to-end validation, creating substantially greater coordination and environmental risk. The right task is a more contained backend-to-dashboard configuration visibility change with focused UI and regression testing.
The left issue spans a multi-provider compatibility test framework, CLI-driven execution, matrix generation, fixtures, proxy configuration, and CI gating, creating substantially broader integration and maintenance scope. The right issue is technically risky because it concerns distributed startup and recovery behavior, but is more concentrated in the worker/model-loading lifecycle.
The right issue is harder because it requires coordinated changes across data models, configuration handling, authentication-time enforcement, distributed spend counters, reservation logic, and scheduled resets, with substantial compatibility and migration risk. The left issue is primarily validation and automation of an already-defined multi-provider stack, with comparatively bounded implementation risk.
The left issue is harder because it requires a security-sensitive change to credential resolution and request routing across shared SDK behavior and provider integrations, with substantial regression and threat-model risk. The right issue is broad and operationally involved, but is primarily a structured compatibility test matrix and CI validation effort built around an existing implementation.
Cross-provider integration testing, proxy orchestration, version handling, and CI-gate maintenance create broader execution and environment risk for #26535; #33371 is a substantial API and router-contract change but has a more concentrated implementation scope.
First-class authentication surfaces require broader architecture, security-sensitive credential lifecycle work, provider integrations, compatibility considerations, and likely new APIs and persistence behavior. The other issue is primarily a bounded validation and CI effort over an already-defined implementation stack, so it carries less implementation scope and risk.
The left issue spans validation of a multi-provider compatibility stack, matrix generation, proxy configuration, CI gating, and independent test coverage, creating broader coordination and integration risk. The right issue is technically subtle but has a more contained remediation surface around bounded recursive-schema expansion, caller configuration, and regression tests.
#26535 is harder because it spans coordinated test infrastructure, provider coverage, CI integration, and release-compatible validation, whereas #32562 is a focused runtime error-propagation fix within one streaming subsystem.
The left requires coordinating broad cross-provider compatibility coverage, test infrastructure, CI integration, and validation across multiple implementation slices. The right is a narrower database-concurrency investigation and targeted persistence fix, though it carries production-risk considerations.
26535 requires broader cross-provider validation, matrix expansion, CI integration, and coordination across multiple implementation slices, creating substantially more engineering and maintenance risk than the focused authentication endpoint and metadata feature in 31296.
Nested access groups require changes to authorization semantics, persistence, propagation, cycle handling, and broad regression coverage; the compatibility-matrix work is broader in test execution but mainly validates an already-defined implementation stack.
Issue 26535 is harder because it spans multi-provider integration coverage, test infrastructure, version-sensitive execution, and CI gating, creating substantially broader coordination and validation risk. Issue 33772 is more localized to cost parsing, metadata registration, and focused regression tests.
Issue #26535 is substantially harder because it spans a multi-provider compatibility matrix, proxy configuration, test infrastructure, CI gating, version resolution, and broad end-to-end validation. Issue #32028 is a more narrowly scoped logging-path bug involving consistent identifier selection for S3 keys and stored request metadata.
The concurrency fix is harder because it requires designing atomic distributed admission control, handling reservation and failure semantics, and validating race conditions across replicas. The other issue is broad but primarily structured test coverage and CI validation over an existing implementation.
Issue 26535 is harder because it spans multiple implementation slices, provider integrations, compatibility behaviors, test infrastructure, and CI gating, creating substantially broader coordination and validation scope. Issue 33666 is a serious production-impacting defect, but its implementation is comparatively focused on query shaping, bounded reconstruction, and targeted regression coverage.
#0 of 0 · 31d17h17m30s ago — current · #import:https:::github.com:berriai:litellm post #3570
The right issue is harder because it requires multiple native platform integrations with provider-specific routing, parameter handling, metadata, and broad SDK/proxy test coverage. The left issue is primarily structured validation and CI coverage for an existing implementation, with more bounded engineering risk.
discussed in #import:https:::github.com:berriai:litellm

ranked child groups

no voted pairs yet in this scope

cli
src
spread
search