18 views
-/https://github.com/berriai/litellm/issues/26699
GitHub · issue

#26699 [Bug]: LiteLLM triton provider adds ~5s delay to time-to-first-token (TTFT)

  • State: open
  • Author: @djangodesmet
  • Labels: bug, proxy, llm translation

### Check for existing issues

- [x] I have searched the existing issues and checked that my issue is not a duplicate.

### What happened?

When using LiteLLM with the Triton provider to proxy requests to a Triton server (vLLM backend), every request shows an extra ~5.0s delay to TTFT compared to calling Triton directly.

**What I see:** - Request via LiteLLM → Triton: TTFT 5.10s - Same request directly to Triton: TTFT 0.10s - The ~5s delay occurs for every model served by Triton when proxied through LiteLLM. - Using LiteLLM → vLLM (with openai provider) shows no meaningful extra delay versus direct requests.

**Environment** - LiteLLM version: v1.83.7-stable - Triton with vLLM backend - Both LiteLLM and Triton running in same Kubernetes namespace - Fully air-gapped (no Internet access)

**Debugging done** - Checked LiteLLM logs: no obvious errors or warnings pointing to the delay. - Observed a repeated internal API call that includes header Authorization: Bearer <invalid JWT>; unclear whether this call is related. - Timing is consistently ~5.0s, suggesting a timeout or retry behavior.

**Impact**

This consistent 5s TTFT addition prevents using LiteLLM + Triton in production.

**R…

GitHub resolver

Import GitHub neighbors on demand. Results are saved as system ingests.

Refresh page
vote history (1 events)
#0 of 0 · 31d17h42m15s ago — entered · #import:https:::github.com:berriai:litellm post #3186
The left task is harder because it requires diagnosing an elusive latency regression across provider translation, streaming behavior, authentication, retries, and deployment boundaries. The right task is a broader feature integration, but its interfaces and validation flow can be designed and tested more directly.
discussed in #import:https:::github.com:berriai:litellm

ranked child groups

no voted pairs yet in this scope

cli
src
spread
search