8 views
-/https://github.com/berriai/litellm/issues/34797
GitHub · issue

#34797 [SAP provider] cache_control is stripped from messages — Anthropic prompt caching unusable

  • State: open
  • Author: @eduardomagnussap
  • Labels: llm translation

When routing Anthropic models through the `sap/` provider (SAP AI Core Orchestration), `cache_control` markers placed on message content blocks are silently dropped during request transformation. As a result, Anthropic prompt caching never activates — responses come back with `cache_read` / `cache_creation` at 0.

### Why

The SAP chat message models (`SAPMessage`, `SAPUserMessage`, `TextContent`) do not define a `cache_control` field, and content is flattened to a plain string during validation, so the marker is discarded before the request is built. By contrast, litellm's native `anthropic` provider supports `cache_control` on messages and tools.

### Minimal repro

```python from litellm.llms.sap.chat.models import SAPMessage m = SAPMessage(role="system", content=[ {"type": "text", "text": "…large static prompt…", "cache_control": {"type": "ephemeral"}} ]) print(m.model_dump(by_alias=True)) # -> {"role": "system", "content": "…"} # cache_control gone ```

### Expected

`cache_control` on content blocks is preserved through `transform_request` so it can reach the model, consistent with Anthropic's prompt-caching API (https://docs.anthropic.com/en/docs/build-with-claude…

GitHub resolver

Import GitHub neighbors on demand. Results are saved as system ingests.

Refresh page
vote history (3 events)
#0 of 0 · 31d18h11m43s ago — entered · #import:https:::github.com:berriai:litellm post #2651
The right-side fix is harder because it touches shared streaming response assembly, asynchronous logging, usage normalization, and billing correctness, with broader regression risk across proxy endpoints. The left-side fix is comparatively contained to SAP-specific structured-message models and request transformation, with narrower testing scope.
The left issue is harder because it involves an unclear compatibility regression across client behavior, proxy routing, and provider translation, requiring broader diagnosis and validation. The right issue is comparatively localized to preserving a field through one provider's models and request transformation, with a more direct implementation and test path.
#0 of 0 · 31d17h56m14s ago — current · #import:https:::github.com:berriai:litellm post #2920
Provider-specific request-model and transformation changes carry broader compatibility and testing risk than a localized routing/auth condition fix.
discussed in #import:https:::github.com:berriai:litellm

ranked child groups

no voted pairs yet in this scope

cli
src
spread
search