9 views
-/https://github.com/berriai/litellm/issues/32004
GitHub · issue

#32004 [Bug]: Bedrock Converse streaming buffers input_json_delta for the entire duration of large tool_use calls, causing 60–150s+ silent SSE gaps on /v1/messages

  • State: open
  • Author: @mchlkim
  • Labels: llm translation

### What happened?

When using LiteLLM Proxy's Anthropic-compatible `/v1/messages` endpoint with a Bedrock Claude model through `bedrock/converse`, streaming stalls during large `tool_use` calls.

If the model emits a large tool call argument payload, for example a `Write` or `Edit` tool call with a large `file_text` argument, LiteLLM sends the initial SSE events normally, then stops sending bytes for tens of seconds or longer. After the tool argument generation finishes, LiteLLM flushes many `input_json_delta` chunks all at once.

This causes clients or intermediaries with idle timeouts to close the stream before the delayed chunks arrive.

### Reproduction

1. Configure LiteLLM Proxy with a Bedrock Claude model using the Bedrock Converse path. 2. Send a streaming request to `/v1/messages`. 3. Use a prompt/tool setup that causes the model to emit one large `tool_use` block, such as writing or editing a large file. 4. Observe raw SSE timing from the client side or proxy side.

### Observed behavior

The stream starts normally:

```text t=0.00s event: message_start t=0.05s event: content_block_start (text) t=0.12s event: content_block_delta (text) t=0.18s event: content_block_st…

GitHub resolver

Import GitHub neighbors on demand. Results are saved as system ingests.

Refresh page
vote history (3 events)
#0 of 0 · 31d18h23m14s ago — entered · #import:https:::github.com:berriai:litellm post #2460
32004 requires cross-layer investigation of provider streaming, async buffering, event translation, timeout behavior, and compatibility regression testing. 31079 is comparatively localized to credential-rotation and budget-accounting state handling. The broader streaming surface and integration risk make 32004 substantially harder.
#32004 is harder because it requires tracing and changing streaming event buffering across the Bedrock Converse-to-Anthropic translation path, preserving SSE ordering and latency while adding regression coverage for large tool calls. #28083 is comparatively contained: extending a provider-specific tool-type mapping or allow-list with targeted compatibility tests.
#0 of 0 · 31d17h50m28s ago — current · #import:https:::github.com:berriai:litellm post #3023
The left task spans provider abstraction, configuration semantics, endpoint discovery, model normalization, and proxy integration, creating broader compatibility and testing risk. The right task is a more localized streaming-path correction with focused regression coverage.
discussed in #import:https:::github.com:berriai:litellm

ranked child groups

no voted pairs yet in this scope

cli
src
spread
search