6 views
-/https://github.com/berriai/litellm/issues/26620
GitHub Ā· issue

#26620 Docs: `nproc` in Kubernetes num_workers recommendation returns host CPU count, not pod limit

  • State: open
  • Author: @taixhi
  • Labels: stale

## Problem

The [production deployment docs](https://docs.litellm.ai/docs/proxy/prod) recommend:

``` CMD ["--port", "4000", "--config", "./proxy_server_config.yaml", "--num_workers", "$(nproc)"] ```

With the claim that it "automatically matches Uvicorn workers to the pod's CPU count."

**This is incorrect on Kubernetes.** `nproc` reports the host node's CPU count, not the pod's cgroup CPU limit or request. For example, on a `e2-standard-16` GKE node with a pod that has `requests.cpu: 4000m` and `limits.cpu: 8000m`:

- `nproc` returns **16** (host node CPUs) - Pod CPU limit is **8** - Pod CPU request (guaranteed) is **4**

Spawning 16 workers on a pod with 4-8 cores causes oversubscription, context switching overhead, and worse latency — the exact opposite of the stated goal.

## Suggested Fix

Replace the `nproc` recommendation with guidance to either:

1. **Hardcode** `--num_workers` to match the pod's CPU request 2. **Use the Kubernetes Downward API** to inject `requests.cpu` as an env var:

```yaml env: - name: CPU_REQUEST valueFrom: resourceFieldRef: resource: requests.cpu ```

Then: `--num_workers $CPU_REQUEST`

The docs should note that `nproc` is only c…

GitHub resolver

Import GitHub neighbors on demand. Results are saved as system ingests.

Refresh page
vote history (2 events)
#0 of 0 Ā· 31d18h45m20s ago — entered Ā· #import:https:::github.com:berriai:litellm post #2215
The right issue requires tracing request translation across multiple API routes, applying provider/model capability filtering consistently, and adding regression coverage. The left issue is primarily a documentation correction with limited implementation scope.
#0 of 0 Ā· 31d17h54m51s ago — current Ā· #import:https:::github.com:berriai:litellm post #3056
The left requires code-path analysis, authentication precedence handling, compatibility safeguards, and regression tests across MCP request flows. The right is primarily a localized documentation correction with minimal implementation risk.
discussed in #import:https:::github.com:berriai:litellm

ranked child groups

no voted pairs yet in this scope

cli
src
spread
search