7 views
-/https://github.com/berriai/litellm/issues/35329
GitHub · issue

#35329 [Bug]: Azure async handler converts asyncio.CancelledError into AzureOpenAIError(500) — cancellation is swallowed and Router fallbacks run to completion

  • State: open
  • Author: @shouenlee
  • Labels: llm translation

### What happened?

The Azure async chat-completion handler catches `asyncio.CancelledError` and re-raises it as `AzureOpenAIError(status_code=500)`:

https://github.com/BerriAI/litellm/blob/main/litellm/llms/azure/azure.py#L462

```python except asyncio.CancelledError as e: ## LOGGING logging_obj.post_call(...) raise AzureOpenAIError(status_code=500, message=str(e)) ```

`CancelledError` is a `BaseException` (Python 3.8+) specifically so libraries don't treat task cancellation as an operational failure. Converting it into a provider 500 breaks cancellation semantics:

1. **Plain `litellm.acompletion`**: cancelling the task raises `litellm.APIError: AzureException APIError` instead of propagating `CancelledError`. 2. **Router with `fallbacks`**: it's worse — the fake 500 triggers the fallback deployment, which **runs to completion** (Python delivers cancellation only once, and it was consumed). The "cancelled" call returns a successful response from the fallback deployment several seconds later. The caller cannot cancel the request at all, and pays for a completion it explicitly abandoned. 3. **Router cooldowns**: every cancellation counts as an azure deployment failure…

GitHub resolver

Import GitHub neighbors on demand. Results are saved as system ingests.

Refresh page
vote history (4 events)
#0 of 0 · 31d19h11m46s ago — entered · #import:https:::github.com:berriai:litellm post #1685
The launch-agent feature is substantially harder: it requires a new CLI workflow, remote authentication and model discovery, subprocess/environment integration, and compatibility across multiple external agent tools and platforms. The cancellation bug is narrower, though it still needs careful async exception, fallback, cooldown, and regression-test handling.
The left issue is harder because it spans MCP discovery, Responses API event sequencing, streaming lifecycle, HTTP status handling, and provider-error propagation across multiple layers. The right issue is more localized to preserving cancellation semantics and adding targeted regression coverage, although router fallback behavior requires care.
The left issue spans passthrough logging, response-shape detection, transformation safeguards, metric persistence, and regression coverage across multiple endpoint variants. The right issue is comparatively localized to preserving cancellation semantics in one async provider path and validating router behavior. The left therefore carries greater integration and compatibility risk.
#0 of 0 · 31d18h11m17s ago — current · #import:https:::github.com:berriai:litellm post #2667
The right issue is harder because it affects cross-cutting async control flow, exception propagation, fallback orchestration, and regression coverage. The left issue is comparatively localized to request-signing/header handling with a narrower validation surface.
discussed in #import:https:::github.com:berriai:litellm

ranked child groups

no voted pairs yet in this scope

cli
src
spread
search