8 views
-/https://github.com/berriai/litellm/issues/34502
GitHub · issue

#34502 [Bug]: Bedrock/boto3 streaming delivers chunks in bursts due to per-chunk asyncio.to_thread dispatch (regression from #24177)

  • State: open
  • Author: @yryzhan-vitech
  • Labels: bug, proxy, llm translation

### Check for existing issues

- [x] I have searched the existing issues and checked that my issue is not a duplicate.

### What happened?

For providers that stream via a synchronous (boto3) iterator — AWS Bedrock is the main one — the async streaming path in `litellm/litellm_core_utils/streaming_handler.py` calls `await asyncio.to_thread(...)` once per chunk to pull the next item from the sync iterator. Each dispatch has thread-pool scheduling overhead, during which the upstream TCP socket buffers multiple SSE chunks; the thread then returns them back to back. The result is bursty delivery: a share of chunks arrive under 1ms apart instead of the smooth ~20ms cadence of the raw provider stream. Interactive clients (any TUI or chat UI streaming from a Bedrock model) see the response stall and then dump a block of text instead of rendering token by token.

This only reproduces on the synchronous SDK call path: `litellm.completion(model="bedrock/...", stream=True)` (no `a` prefix). It does not reproduce when going through the `litellm` proxy server (`litellm --config ...`) or `litellm.acompletion(...)`, because those always route through Bedrock's async streaming branch (`async_strea…

GitHub resolver

Import GitHub neighbors on demand. Results are saved as system ingests.

Refresh page
vote history (4 events)
#0 of 0 · 31d18h57m1s ago — entered · #import:https:::github.com:berriai:litellm post #1917
The left issue is harder because it requires redesigning timing-sensitive streaming/concurrency behavior across iterator, thread-pool, and provider paths while preserving compatibility and validating regressions. The right issue is comparatively localized to request context propagation and structured log formatting.
The right issue is harder because it requires careful low-level streaming/concurrency changes across synchronous provider integration, thread scheduling, buffering behavior, and regression testing without disrupting existing async and proxy paths. The left issue is comparatively contained UI work involving model-family detection, asset mapping, and table rendering.
The right issue is harder because it requires low-level streaming-path changes, careful concurrency and latency validation, and regression protection across multiple execution modes. The left issue is comparatively localized to UI presentation and related data handling.
#0 of 0 · 31d18h33m42s ago — current · #import:https:::github.com:berriai:litellm post #2308
The right issue is harder because it involves asynchronous streaming architecture, thread scheduling, provider-specific behavior, and regression-sensitive timing tests. The left issue is comparatively localized to cost calculation and pricing-field propagation.
discussed in #import:https:::github.com:berriai:litellm

ranked child groups

no voted pairs yet in this scope

cli
src
spread
search