8 views
-/https://github.com/berriai/litellm/issues/31092
GitHub · issue

#31092 [Bug]: Triton models have no means to control chat template parameters (e.g. disable thinking)

  • State: open
  • Author: @vladmirtxrx
  • Labels: bug, proxy

### Check for existing issues

- [x] I have searched the existing issues and checked that my issue is not a duplicate.

### What happened?

LiteLLM uses prompt_factory function (https://github.com/BerriAI/litellm/blob/69b0dd2da072c2ae0dd0c41ed969cbebf8e36828/litellm/llms/triton/completion/transformation.py#L199) in triton model requests handler.

Looks like all extra params are send to Triton instead, and triton by itself doesnt do any chat templating.

Problem is, it looks like there is no way to pass in parameters related to chat template when making a request. For example: user might want to disable model thinking to use less tokens in the output.

### Steps to Reproduce

```py import requests

API_KEY="sk-..." LLM_MODEL="qwen3.6-27b" query_text = "test?" LLM_REQUEST_TIMEOUT=1000 url = "http://localhost:4000/project-api/litellmqwe/litellm-webui/chat/completions"

r = requests.post( url, headers={ "Content-Type": "application/json", "Authorization": f"Bearer {API_KEY}", }, json={ "model": LLM_MODEL, "messages": [{"role": "system", "content": query_text}], "max_tokens": 10000, …

GitHub resolver

Import GitHub neighbors on demand. Results are saved as system ingests.

Refresh page
vote history (2 events)
#0 of 0 · 31d18h37m14s ago — entered · #import:https:::github.com:berriai:litellm post #2248
The left task spans request parsing, provider-specific transformation, parameter validation, and regression coverage across proxy and Triton integration boundaries. The right task is likely concentrated in dashboard export aggregation and query validation, with a narrower change surface.
#0 of 0 · 31d17h44m46s ago — current · #import:https:::github.com:berriai:litellm post #3129
The Triton change likely requires provider-specific parameter plumbing, compatibility decisions, and targeted integration tests across request transformation paths. The authentication-message correction is comparatively localized to exception classification and response handling, with limited behavioral scope.
discussed in #import:https:::github.com:berriai:litellm

ranked child groups

no voted pairs yet in this scope

cli
src
spread
search