Prompt caching

Which providers cache

• Claude models (anthropic/...) cache a prompt only where the request carries a cache marker, on every provider. A request without one, such as a plain OpenAI SDK or Codex call, pays the full input price on every turn.
• OpenAI models, and most other models, cache a repeated prefix on their own, without a marker. Some providers do not cache at all.

GET https://api.llm.buzz/api/v1/models/{author}/{slug}/endpoints says what each provider of a model does. pricing.input_cache_read is its cached-input price, and supports_implicit_caching is true when it caches a request without markers at a lower price. The Models page marks the models that cache.

Mark a prompt for Claude

From the OpenAI SDK, send Anthropic's cache_control at the top level of the request. The provider puts the cache breakpoint on the last block it can cache, so the breakpoint moves with the conversation and each turn reads what the turn before it wrote. On Chat Completions, a marker on the last function tool also caches the tool definitions.

reply = client.chat.completions.create(
    model="anthropic/claude-sonnet-5.5",
    messages=messages,
    extra_body={"cache_control": {"type": "ephemeral"}},
)

message = client.messages.create(
    model="anthropic/claude-sonnet-5.5",
    max_tokens=1024,
    system=[{"type": "text", "text": instructions, "cache_control": {"type": "ephemeral"}}],
    messages=messages,
)

A marker is {"type": "ephemeral"}, with an optional ttl of "5m", the default, or "1h". null carries nothing, and any other value is a 400 that names where it was, such as cache_control or tools.1.cache_control.

A marker reaches every provider of the model that takes Claude's markers. A provider that caches on its own, such as OpenAI, receives none. x-sgr-ignored-parameters names a marker that stayed behind, as cache_control->not_forwarded(provider_decides_caching; ...): a top-level or message marker when no provider of the model can take it, and a tool marker when any provider cannot.

Claude takes at most 4 breakpoints in a request, and the top-level marker counts as one unless the last block of the last message already carries a marker. A one-hour breakpoint must come before every five-minute one, in the order the provider reads them: tools, then system, then messages, then the top-level marker. A provider that refuses such a layout is skipped and the model's other providers serve the request; when every provider refuses it, the answer is a 400 invalid_parameter with param: cache_control (Anthropic's invalid_request_error on Messages), and its message says what to remove or move.

On Claude Sonnet 5.5, whose cached input costs a twentieth of its input, a 40-turn session that grows from 20,000 to 120,000 tokens pays about $5.60 for its input without a marker, and under $1 with one top-level marker.

What a cached read costs

Each provider lists its cache prices beside its input and output prices, in US dollars per token on the endpoints route and per million tokens on the model pages:

A request bills its cached input at the cached price, the input written to the cache at the write price, and the rest of its input at the input price, and usage.cost includes all three. Claude Sonnet 5.5, for example, lists $2 per million input tokens, $0.10 cached, $2.50 for a five-minute write and $4 for a one-hour write.

Read the cache in usage

On Chat Completions and Responses, the reads and the writes are separate parts of the prompt's token count. cache_write_1h_tokens appears only when the provider reports the split. Chat Completions shows a field only when the provider reported it; Responses always shows cached_tokens and cache_write_tokens, with 0 when the provider reported none.

Keep a conversation on its cache

A cache lives at one provider. Name the conversation with session_id in the body or the x-session-id header, on every turn, and its turns stay on the provider that holds its warm cache, ahead of a cheaper or faster provider. On Chat Completions and Responses, prompt_cache_key names it too. A client that sends none of these still gets a session from the conversation's opening: its first system text and first message.

The conversation moves when that provider fails or your request can no longer use it, such as a provider.only that leaves it out, and a provider.order in the request turns this off.

Providers keep their caches per account, and llm.buzz calls each provider from its own account for every organization. So two organizations that send the same prompt prefix to the same provider can share its cache entry, and how fast an answer comes back can show that someone sent that prefix recently.

OpenAI's explicit caching

OpenAI's GPT-5.6 and later models, such as openai/gpt-5.6-luna and openai/gpt-6-sol, take OpenAI's explicit caching controls on Chat Completions and Responses, as the OpenAI SDK sends them:

reply = client.responses.create(
    model="openai/gpt-5.6-luna",
    input=[{"role": "user", "content": [
        {"type": "input_text", "text": long_context, "prompt_cache_breakpoint": {"mode": "explicit"}},
        {"type": "input_text", "text": question},
    ]}],
    extra_body={"prompt_cache_options": {"mode": "explicit"}},
)

• prompt_cache_breakpoint: {"mode": "explicit"} marks the end of a prefix to reuse, on a text part: on Chat Completions a text part of a system, developer, user or tool message, and on Responses an input_text part of a system, developer or user message or of a function_call_output. On any other part, or on an assistant message, it is a 400 that names where it was.
• prompt_cache_options takes mode, implicit (OpenAI's default) or explicit (OpenAI then writes the cache only at breakpoints), and ttl 30m.
• prompt_cache_retention is in_memory or 24h. 24h keeps prefixes for up to 24 hours, so on a request with provider.zdr: true, provider.data_collection: "deny" or your organization's no-retention policy it is a 400 invalid_parameter on prompt_cache_retention. OpenAI may refuse in_memory on GPT-5.5 and later, so on such a request to those models send no retention at all.

OpenAI receives these controls as you send them. Sent to a Claude model, a breakpoint becomes a five-minute Claude marker, named in x-sgr-ignored-parameters as prompt_cache_breakpoint->translated(cache_control), and the options and retention are not sent. A provider that cannot take them receives none of them, and when no provider of the model can, x-sgr-ignored-parameters names them. Your cache_control never becomes an OpenAI breakpoint, so you never pay for an OpenAI cache write you did not ask for. OpenAI reports its writes in cache_write_tokens, billed at the five-minute write price.