6 views
-/https://github.com/berriai/litellm/issues/34328
GitHub · issue

#34328 [Bug]: unpack_defs still hangs (unbounded) on recursive tool schemas — Bedrock & Vertex callers pass no byte budget

  • State: open
  • Author: @Jesse-Schultz-Relativity
  • Labels: llm translation

Related (closed): #19098, #13151, #11372 — this looks like an incomplete fix for #19098.

### What happened?

A `/chat/completions` request whose tool `parameters` is a **recursive, high-fan-in JSON Schema** (`$defs`/`$ref` where a type references itself and is reused by many other defs — the shape Pydantic emits for recursive models) causes `unpack_defs()` to expand **unboundedly**. It is CPU-bound, not just memory: a ~23 KB tool schema expands past **510 MB within 30 s and does not converge**, so the request never reaches the provider and the fronting gateway/LB times out (~80 s) → **502**. Adding memory does not help.

Reproduced on both: - **Bedrock** Converse (Claude), and - **Vertex AI / Gemini**.

This is closely related to the closed #19098. That fix added a `max_inlined_bytes` budget and a `ref_chain` cycle guard to `unpack_defs`, but the bug persists because:

1. **Neither caller passes `max_inlined_bytes`**, so it defaults to `None` (unbounded) on exactly the two paths that trigger the blow-up. 2. The `ref_chain` guard only prevents **path cycles** (a def already on the current resolution chain). It does **not** prevent **diamond fan-out** — the same def reused…

GitHub resolver

Import GitHub neighbors on demand. Results are saved as system ingests.

Refresh page
vote history (14 events)
#0 of 0 · 31d18h1m35s ago — entered · #import:https:::github.com:berriai:litellm post #3049
#34328 requires algorithmic changes to recursive schema expansion, safe budget semantics, multi-provider caller updates, and regression testing for pathological graphs; #30566 is comparatively localized timing instrumentation across two endpoint paths.
The recursive-schema failure requires redesigning bounded expansion semantics across multiple provider translation paths, with careful handling of cycles, fan-out, memory/CPU limits, compatibility, and regression testing. The budget issue is narrower: it mainly needs atomic/concurrency-safe counter-window updates and reliable async write coordination in one accounting path.
The left task is substantially harder because it spans a new opt-in integration, request-path behavior, configuration across multiple caller scopes, fail-open guarantees, and several observability surfaces. The right task is a focused safety fix across a small number of schema-expansion paths, though it carries meaningful performance and regression risk.
29452 is harder because it requires designing and integrating a broad authentication abstraction across providers, credential lifecycles, storage, CLI/API surfaces, and security-sensitive compatibility concerns. 34328 is technically subtle but comparatively bounded to schema-expansion safeguards and a small set of affected call paths.
The left issue spans validation of a multi-provider compatibility stack, matrix generation, proxy configuration, CI gating, and independent test coverage, creating broader coordination and integration risk. The right issue is technically subtle but has a more contained remediation surface around bounded recursive-schema expansion, caller configuration, and regression tests.
34328 is harder because it requires algorithmic safeguards, multiple integration-point changes, termination/resource guarantees, and cross-provider regression testing; 35524 is comparatively localized to budget-admission logic and its concurrency-focused tests.
26237 is harder because it requires coordinated changes across worker readiness, persistent-state recovery, refresh behavior, and request admission under partial infrastructure failure. 34328 is more localized to bounding and safely testing schema expansion across a small set of translation paths.
Recursive schema handling is harder because it requires safe, bounded expansion semantics across multiple translation paths, compatibility decisions, and broad regression testing; the streaming fix is comparatively localized to iterator error propagation and terminal-event behavior.
Issue 34328 is harder because it requires correcting recursive schema expansion safely, preserving translation behavior, enforcing budgets across multiple provider paths, and validating CPU/memory termination under adversarial schemas. Issue 31296 is a more bounded authentication-discovery feature involving endpoint behavior, metadata configuration, and response integration.
#34328 is harder because it requires correcting recursive schema expansion safeguards across multiple provider translation paths, addressing algorithmic termination and resource limits, and validating complex performance regressions. #32201 is narrower endpoint/response-processing integration work with a more localized enforcement fix.
The left requires correcting a recursive expansion algorithm, defining safe resource limits, preserving provider compatibility, and validating pathological schemas across multiple execution paths. The right is broader product work but can primarily be implemented through hierarchical data resolution, dependency handling, and associated API/test updates.
The left issue is harder because it spans public API design, cross-provider error normalization, router-state integration, compatibility, and downstream contract validation. The right issue is a narrower translation-layer safety fix, requiring bounded recursion behavior, caller updates, and regression tests, though it carries significant performance-risk validation.
#34328 requires algorithmic safeguards, cross-provider integration changes, and stress testing for pathological input, creating substantially higher correctness and performance risk; #28267 appears comparatively localized to proxy header-selection and forwarding logic.
#0 of 0 · 31d17h29m8s ago — current · #import:https:::github.com:berriai:litellm post #3571
The left issue is harder because it spans three provider integrations, cross-cutting model metadata, parameter translation, cost accounting, and compatibility across both SDK and proxy paths. The right issue has greater algorithmic risk but a narrower implementation surface focused on a shared expansion routine and its callers.
discussed in #import:https:::github.com:berriai:litellm

ranked child groups

no voted pairs yet in this scope

cli
src
spread
search