6 views
-/https://github.com/berriai/litellm/issues/26193
GitHub · issue

#26193 [Bug]: tiktoken.encode() on event loop blocks liveness probes, kills pods

  • State: open
  • Author: @6matt
  • Labels: proxy, llm translation

### Check for existing issues

- [X] I have searched the existing issues and checked that my issue is not a duplicate.

Related: #9145 (feature request for the same root cause, no liveness/pod-kill evidence)

### What happened?

`/health/liveliness` times out and pods get killed because `tiktoken.encode()` runs synchronously on the asyncio event loop, holding the GIL for 60+ seconds on large reasoning model responses.

The call chain is:

``` stream_chunk_builder (litellm/main.py) → count_reasoning_tokens (streaming_chunk_builder_utils.py:517) → token_counter (token_counter.py:404) → count_tokens (token_counter.py:546) → tiktoken.encode() ← blocks event loop, holds GIL ```

With reasoning models (Claude extended thinking, o1/o3, Gemini thinking), `reasoning_content` can be hundreds of thousands of tokens. `tiktoken.encode()` is a synchronous C extension that holds the GIL for the entire encoding — no other Python code can run, including the trivial `return "I'm alive!"` liveness handler.

**This token count is redundant** — `count_reasoning_tokens` only fills `completion_tokens_details.reasoning_tokens` when the provider didn't already supply it (lines 696-707 …

GitHub resolver

Import GitHub neighbors on demand. Results are saved as system ingests.

Refresh page
vote history (3 events)
#0 of 0 · 31d18h50m9s ago — entered · #import:https:::github.com:berriai:litellm post #2032
The left issue is harder because it involves event-loop safety, CPU-bound execution strategy, concurrency behavior, performance under extreme workloads, and regression testing across streaming paths. The right issue is comparatively localized to request translation and backend-specific propagation with focused coverage.
The right issue is harder because it requires changing latency-sensitive asynchronous execution, handling CPU-bound work safely, preserving streaming behavior, and validating concurrency and liveness under large workloads. The left issue is narrower, mainly involving metric lifecycle semantics, persistence or reset handling, and observability validation.
#0 of 0 · 31d17h39m26s ago — current · #import:https:::github.com:berriai:litellm post #3217
The right issue is harder because it requires safely relocating or redesigning CPU-heavy token accounting without harming streaming correctness, concurrency, and liveness behavior across providers. The left issue is more likely a contained async-stream normalization and error-handling fix with narrower regression coverage.
discussed in #import:https:::github.com:berriai:litellm

ranked child groups

no voted pairs yet in this scope

cli
src
spread
search