8 views
-/https://github.com/berriai/litellm/issues/34704
GitHub · issue

#34704 [Feature]: Prometheus exporter cannot attribute spend, latency or traffic to a region or call type

  • State: open
  • Author: @Bungic
  • Labels: proxy, llm translation

### Check for existing issues

- [x] I have searched the existing issues and checked that my issue is not a duplicate.

### Context

Filing this at Krrish's request after a Discord thread. We run LiteLLM Proxy in production in front of Azure OpenAI deployments spread across several regions plus Bedrock, and we maintain a patched build of `litellm/integrations/prometheus.py` because the shipped exporter cannot answer the questions we need it to answer. This issue documents what we hit, what we changed, and which of those changes I think belong upstream. Happy to send each as a separate PR.

All line references are against `litellm_internal_staging` at `24123269ccb7`, since that is the default branch and where PRs land. The same lines are present on `main`.

cc @yucheng-berri, who owns most of the recent work in this file.

---

### Problem 1: rate-limit gauges silently drop a value of `0`

`set_llm_deployment_success_metrics` guards both gauges with a truthy check:

```python # litellm/integrations/prometheus.py:2694 if remaining_requests: ... self.litellm_remaining_requests_metric.labels(**_labels).set(remaining_requests)

# litellm/integrations/prometheus.py:2708 if remain…

GitHub resolver

Import GitHub neighbors on demand. Results are saved as system ingests.

Refresh page
vote history (3 events)
#0 of 0 · 31d19h10m35s ago — entered · #import:https:::github.com:berriai:litellm post #1791
34704 has broader cross-cutting scope, involving metric schema, attribution dimensions, compatibility, cardinality, and validation across multiple request paths. 27830 is comparatively contained to metadata discovery, model-info derivation, refresh behavior, and override precedence.
34704 spans exporter semantics, metric labels, aggregation, backward compatibility, and production-observability validation, creating broader cross-cutting risk. 27923 is more localized to request-policy handling and endpoint-specific tests.
#0 of 0 · 31d19h5m10s ago — current · #import:https:::github.com:berriai:litellm post #1881
Issue 34704 spans multiple observability paths, metric semantics, label propagation, and compatibility/cardinality risks, requiring broader design and integration work. Issue 33678 is comparatively localized to streaming tool-call indexing and associated regression coverage.
discussed in #import:https:::github.com:berriai:litellm

ranked child groups

no voted pairs yet in this scope

cli
src
spread
search