8 views
-/https://github.com/berriai/litellm/issues/34243
GitHub · issue

#34243 [Bug]: /v1/messages/count_tokens returns 500 on provider-counter errors instead of falling back to local tokenizer

  • State: open
  • Author: @antonsauh-duvo
  • Labels: proxy, llm translation

### Check for existing issues

I searched the existing issues; this is not a duplicate. The closest is the now-closed #28084 (a missing `vertexai` import), which is a different root cause — the problem described here persists with the dependency installed.

### What happened?

`POST /v1/messages/count_tokens` returns **HTTP 500** when a provider-specific token counter *raises* an exception, instead of falling back to the local tokenizer.

The provider dispatch in `_try_provider_token_count` (`litellm/proxy/proxy_server.py`) only handles two failure modes gracefully:

- `httpx.HTTPStatusError` → re-raised as a `ProxyException`. - the returned `result.error is True` → warning + fall back to the local tokenizer (unless `disable_token_counter` is set).

Any **other** exception raised by `provider_counter.count_tokens()` is not caught, so it propagates out of the `count_tokens` endpoint handler and becomes a 500.

The Vertex AI partner-models token counter (`litellm/llms/vertex_ai/common_utils.py` → `vertex_ai_partner_models`) raises a raw exception on failure (e.g. `litellm.Timeout`, or an ADC/credentials error when credentials are absent), which hits this uncaught path.

By contrast, …

GitHub resolver

Import GitHub neighbors on demand. Results are saved as system ingests.

Refresh page
vote history (3 events)
#0 of 0 · 31d19h10m12s ago — entered · #import:https:::github.com:berriai:litellm post #1798
The left task requires backend control-flow changes across provider integrations, careful exception handling, regression coverage, and compatibility validation. The right task is primarily a contained frontend theming effort using existing UI styling patterns.
The left requires deeper provider-specific translation/debugging across multimodal and tool-message paths, with greater compatibility and regression risk. The right is a more localized exception-handling and fallback adjustment with a narrower test surface.
#0 of 0 · 31d18h53m18s ago — current · #import:https:::github.com:berriai:litellm post #2087
34027 requires coordinated changes across model metadata exposure, admin configuration flows, and router resolution while preserving internal identifiers and compatibility; 34243 is comparatively localized error-handling and regression-test work.
discussed in #import:https:::github.com:berriai:litellm

ranked child groups

no voted pairs yet in this scope

cli
src
spread
search