8 views
-/https://github.com/berriai/litellm/issues/27122
GitHub · issue

#27122 test: unify model_cost map source for tests (LITELLM_LOCAL_MODEL_COST_MAP=True globally)

  • State: open
  • Author: @mateo-berri
  • Labels: testing, llm translation

## Background

`litellm.model_cost` is loaded once at `litellm.__init__` time. By default it fetches `model_prices_and_context_window.json` from `https://raw.githubusercontent.com/BerriAI/litellm/main/model_prices_and_context_window.json` (the **main branch** on GitHub) — *not* from the PR-branch JSON in the working tree. Setting `LITELLM_LOCAL_MODEL_COST_MAP=True` overrides this and uses the local file.

This default is correct for production (operators get new models without redeploying litellm), but **wrong for tests**, which need to validate the code in front of them against the data in front of them. It creates a chicken-and-egg: PR adds a flag + a test asserting the flag → CI fetches main without the flag → test fails → PR can't merge → the flag never lands on main.

## Current state (inconsistent)

The codebase tacitly admits tests should use the local file, but does it ad-hoc:

- ~10 test files self-set the env var inline, e.g. [`tests/litellm_utils_tests/test_utils.py:565`](https://github.com/BerriAI/litellm/blob/main/tests/litellm_utils_tests/test_utils.py#L565), `tests/search_tests/base_search_unit_tests.py:42`, `tests/code_coverage_tests/bedrock_pricing.py:194`, `tests/…

GitHub resolver

Import GitHub neighbors on demand. Results are saved as system ingests.

Refresh page
vote history (2 events)
#0 of 0 · 31d18h45m48s ago — entered · #import:https:::github.com:berriai:litellm post #2312
Issue 27122 is harder because it requires coordinating test-environment initialization and potentially many existing test paths, with CI and import-order risks. Issue 35577 appears more localized to router upsert handling, validation, and regression coverage.
#0 of 0 · 31d18h31m35s ago — current · #import:https:::github.com:berriai:litellm post #2527
OAuth lifecycle work spans token refresh behavior, credential persistence, request integration, expiry and concurrency handling, and end-to-end MCP validation; the testing change is comparatively localized to shared test configuration and cleanup of overrides.
discussed in #import:https:::github.com:berriai:litellm

ranked child groups

no voted pairs yet in this scope

cli
src
spread
search