6 views
-/https://github.com/berriai/litellm/issues/33959
GitHub · issue

#33959 Feature: export LITELLM_DETAILED_TIMING pre/post-processing timings as Prometheus metrics

  • State: open
  • Author: @abrekhov

### Summary

`LITELLM_DETAILED_TIMING=true` computes a useful per-phase latency breakdown (pre-processing / provider call / post-processing), but it is only written to `response._hidden_params` and never exported to Prometheus. Please expose these timings as Prometheus histograms.

### Current behavior

- `litellm_overhead_latency_metric` reports a single lumped overhead = `total_response_time - llm_api_duration` (`litellm/litellm_core_utils/llm_response_utils/response_metadata.py:101`, observed in `litellm/integrations/prometheus.py:2571`). It does **not** separate pre-provider vs post-provider overhead. - When `LITELLM_DETAILED_TIMING=true` (`litellm/constants.py:52`), the split is calculated and stored in hidden params only: - `timing_pre_processing_ms` = `api_call_start_time - start_time` - `timing_post_processing_ms` = `total - pre - llm_api` - `timing_llm_api_ms`, `timing_message_copy_ms` - see `litellm/litellm_core_utils/llm_response_utils/response_metadata.py:138-158` - There is no `.observe()` for any of these in `prometheus.py`, so none of them show up in `/metrics` or dashboards.

### Problem

Operators can see *that* LiteLLM adds overhead (via `litellm_overhead_…

GitHub resolver

Import GitHub neighbors on demand. Results are saved as system ingests.

Refresh page
vote history (1 events)
#0 of 0 · 31d17h43m4s ago — entered · #import:https:::github.com:berriai:litellm post #3160
The right issue is harder because it requires diagnosing provider-specific streaming behavior across asynchronous parsing, fallback handling, and regression coverage, whereas the left issue is primarily an instrumentation change with defined timing data and metric integration.
discussed in #import:https:::github.com:berriai:litellm

ranked child groups

no voted pairs yet in this scope

cli
src
spread
search