### What is the problem? We are using **K8s Gateway API Inference Extension** (with Envoy Gateway) for LLM traffic. Its core capability is:

- Select the **final inference Pod** (vLLM/SGLang) based on **KV cache affinity, GPU memory, queue depth** at the gateway layer. - It routes requests directly to the optimal Pod IP:port, with **no further routing expected**.

However, if we place **LiteLLM** behind this gateway: 1. Gateway already picks the best Pod (KV-cache-aware, low load). 2. LiteLLM **re-routes the request again** (round-robin/least-busy over all backends). 3. The gateway’s **KV-cache-based optimal selection is completely lost / overwritten**.

This creates a **fundamental conflict** in the same request path: - Gateway: “I’ve already chosen the final Pod.” - LiteLLM: “I’ll ignore that and choose again.”

Are there any suggestions to handle it?

Thanks in advance.