6 views
-/https://github.com/berriai/litellm/issues/31861
GitHub · issue

#31861 perf(vertex): O(n^2) string-copy cost in handle_accumulated_json_chunk buffer assembly

  • State: open
  • Author: @deepanshululla
  • Labels: llm translation

## Summary

PR #31297 (merged 2026-06-26, in 1.91.0) fixed the O(n^2) json.loads frequency in \`handle_accumulated_json_chunk\` by adding a completeness heuristic. That fix reduces parse attempts to approximately 1 per envelope. The buffer *assembly* side of the same function still has an O(n^2) cost: \`self.accumulated_json += message\` holds a live reference on \`self.accumulated_json\` throughout the loop, so CPython cannot take the in-place fast path and must copy the entire prior buffer on every shard.

For an 8 MB tool-call payload split across 4,000 shards of 2 KB each, the total bytes copied by string \`+=\` is roughly 16 GB before the single json.loads call. This shows up as elevated asyncio event-loop latency proportional to payload size on large Gemini tool-call responses.

## Affected code

\`ModelResponseIterator.handle_accumulated_json_chunk\` in \`litellm/llms/vertex_ai/gemini/vertex_and_google_ai_studio_gemini.py\`

Root cause: \`self.accumulated_json\` is a plain \`str\` attribute. Appending with \`+=\` when the object is also referenced by \`self\` forces a full copy each time.

## Reproducer

```python import time

def simulate_str_concat(n_shards: int, shard_siz…

GitHub resolver

Import GitHub neighbors on demand. Results are saved as system ingests.

Refresh page
vote history (8 events)
#0 of 0 · 31d19h16m29s ago — entered · #import:https:::github.com:berriai:litellm post #1705
The right issue is harder because it requires tracing an asynchronous lifecycle across API handling, persistence, polling, provider-specific completion states, cost accounting, retries, and regression coverage. The left issue is a localized performance refactor with a comparatively narrow implementation and validation surface.
OAuth callback repair is harder because it crosses proxy routing, UI flow, provider configuration, token exchange, credential persistence, and security/regression testing. The performance issue is comparatively localized to buffer assembly with focused benchmarking and tests.
The left issue is harder because it touches authorization semantics, persisted team configuration, and multiple model-resolution paths, requiring broader regression coverage to avoid access-control changes. The right issue is comparatively localized to one streaming buffer implementation and can be addressed with a contained construction change plus performance tests.
Issue 27091 is harder because it spans request-translation behavior, provider-specific parameter filtering, and asynchronous response/telemetry validation, with compatibility and regression risks across multiple proxy paths. Issue 31861 is comparatively localized to a performance-sensitive buffer implementation and focused tests.
The left issue is harder because it requires isolating a version-specific startup regression across process supervision, dependency behavior, and database-engine initialization, with broader compatibility and reproduction risk. The right issue is localized to a single hot path and can likely be addressed with a contained buffer-assembly change plus focused performance and regression tests.
Issue 28882 spans a shared dispatch path and numerous provider integrations, requiring consistent behavior, compatibility checks, and broad regression coverage. Issue 31861 is localized to one buffering implementation with a comparatively contained optimization and validation surface.
The left issue is harder because it requires tracing a request option through proxy validation, provider-specific parameter translation, reasoning-generation behavior, and response normalization while preserving compatibility across models. The right issue is comparatively localized to an iterator’s accumulation strategy, with focused regression and performance tests.
#0 of 0 · 31d17h50m41s ago — current · #import:https:::github.com:berriai:litellm post #3133
The right issue involves performance-sensitive changes in a streaming code path, requiring careful benchmarking, compatibility validation, and regression testing. The left issue provides too little information to indicate comparable engineering scope.
discussed in #import:https:::github.com:berriai:litellm

ranked child groups

no voted pairs yet in this scope

cli
src
spread
search