Prompt caching

Prompt caching lets a provider reuse the tokens it already processed for a repeated prompt prefix instead of reading them again. Use it when many requests share a large, stable head, such as a long system prompt, a tool schema, a document you ask several questions about, or the growing history of a chat or coding-agent session. The cached prefix is billed at a lower rate and skips re-processing, so repeated requests are cheaper and start returning tokens sooner.

Gateway treats caching as a vendor-route capability. The same canonical model can cache differently depending on the vendor that serves the request, so the caching behavior follows the route, not the model name alone.

How prompt caching works

Every route falls into one of three caching families. The family decides what Gateway does with your request.

The split exists because providers disagree on how caching is triggered. Anthropic models need you to mark the cacheable prefix. Most other providers cache repeated prefixes automatically and reject the marker if you send it, so Gateway strips it for you rather than letting the request break.

Two things follow from this. Send cache_control markers only on explicit routes. On every other route, rely on automatic caching and skip the markers.

Which providers use which family

Routes not listed here are none today. A none route silently drops cache markers and bills every input token at the standard rate. Automatic caching is best-effort by the provider, so a given prompt may not produce a cache hit even on an automatic route, usually because the shared prefix is shorter than the provider’s minimum cacheable length.

Inputs: explicit caching with cache markers

On explicit routes, add a cache_control marker to the last content block of the prefix you want to reuse. Everything up to and including that block becomes the cached prefix. Put your stable content first (system instructions, tool definitions, a shared document), then the marker, then the part that changes per request.

$
curl https://api-gateway.merge.dev/v1/responses \
>
  -H "Authorization: Bearer YOUR_API_KEY" \
>
  -H "Content-Type: application/json" \
>
  -d '{
>
    "model": "anthropic/claude-haiku-4-5-20251001",
>
    "vendor": "anthropic",
>
    "max_tokens": 1024,
>
    "input": [
>
      {
>
        "type": "message",
>
        "role": "user",
>
        "content": [
>
          {
>
            "type": "text",
>
            "text": "Here is the full onboarding handbook for Acme Corp: ...long stable document...",
>
            "cache_control": { "type": "ephemeral" }
>
          },
>
          {
>
            "type": "text",
>
            "text": "What is the PTO policy for a new hire in their first 90 days?"
>
          }
>
        ]
>
      }
>
    ]
>
  }'

The first request with a new prefix writes it to the cache and costs a small write premium. Every later request that repeats the same prefix reads it back at the discounted cache-read rate. The cache entry is short-lived, so caching pays off when repeat requests arrive close together, as they do in an agent loop or a multi-turn conversation.

You can place the marker on a text, image, document, or audio block. The block you mark, and everything before it, forms the cached prefix.

Let Anthropic place the marker for you

On Anthropic routes you can skip per-block markers entirely and send a single cache_control at the top level of the request body. Anthropic places the breakpoint on the last cacheable block and moves it forward as the conversation grows, so you do not rewrite markers on every turn. Gateway forwards the top-level marker unchanged.

$
curl https://api-gateway.merge.dev/v1/responses \
>
  -H "Authorization: Bearer YOUR_API_KEY" \
>
  -H "Content-Type: application/json" \
>
  -d '{
>
    "model": "anthropic/claude-sonnet-5",
>
    "vendor": "anthropic",
>
    "max_tokens": 1024,
>
    "cache_control": { "type": "ephemeral" },
>
    "input": [
>
      {
>
        "type": "message",
>
        "role": "system",
>
        "content": "Here is the full onboarding handbook for Acme Corp: ...long stable document..."
>
      },
>
      {
>
        "type": "message",
>
        "role": "user",
>
        "content": "What is the PTO policy for a new hire in their first 90 days?"
>
      }
>
    ]
>
  }'

This is still an explicit route, and the top-level field is still a marker. Anthropic does not cache anything you have not asked it to, so a request with no cache_control at all produces no cache hit no matter how long the shared prefix is. Do not confuse this with the automatic family below, where the provider caches on its own and Gateway strips markers.

Both styles compose. A top-level marker uses one of the four available breakpoints, so you can add per-block markers alongside it when you want to cache sections that change at different rates. Two constraints apply:

• A per-block marker with a different ttl than the top-level marker, on the block Anthropic would target, is rejected by Anthropic. Keep the TTLs the same or move the per-block marker earlier in the prompt.
• Claude on Bedrock accepts a top-level marker on current model generations. Opus 4.6 and earlier on Bedrock reject it, so use per-block markers if your routing policy can land on one of those.

Inputs: automatic caching

On automatic routes, send the request as usual. The provider detects the repeated prefix and caches it without a marker. Keep the stable content at the front of the prompt so the reused prefix is as long as possible.

To help the provider group related requests onto the same cache, pass a session_id. Gateway namespaces it per organization and derives the provider cache key, so two of your requests with the same session_id land on the same cached prefix, and a different tenant’s identical value never collides with yours.

$
curl https://api-gateway.merge.dev/v1/responses \
>
  -H "Authorization: Bearer YOUR_API_KEY" \
>
  -H "Content-Type: application/json" \
>
  -H "X-Session-Id: chat_9f2a7c" \
>
  -d '{
>
    "model": "openai/gpt-4o",
>
    "vendor": "openai",
>
    "max_tokens": 1024,
>
    "input": [
>
      {
>
        "type": "message",
>
        "role": "system",
>
        "content": "You are a support agent for Acme Corp. Follow this 3000-word policy: ...long stable document..."
>
      },
>
      {
>
        "type": "message",
>
        "role": "user",
>
        "content": "How do I reset a customer'\''s API key?"
>
      }
>
    ]
>
  }'

You can pass the session hint either as the X-Session-Id header or the session_id body field. The header wins when both are present. The value is capped at 256 characters and truncated rather than rejected, so a caching hint never fails a request.

Caching across API specs

Gateway accepts several request specs so you can point an existing SDK at it by changing the base URL. Each spec expresses caching in its own idiom, and Gateway normalizes them into the same caching contract. The X-Session-Id header works on every spec, since it rides on the HTTP request rather than the body.

Top-level markers are specific to the Anthropic request shape, so they are available on the Merge Responses API and the Anthropic SDK. The OpenAI and Vercel specs have no top-level field of their own, so place markers on a message or content block there.

The Merge Responses API examples are shown above. The rest of this section shows the same caching on the other specs.

Anthropic SDK

Point the Anthropic SDK at the /v1/anthropic base URL and mark the cacheable prefix with cache_control, the same way you would against Anthropic directly. Use this on explicit routes.

1
from anthropic import Anthropic
2
3
client = Anthropic(
4
    api_key="YOUR_API_KEY",
5
    base_url="https://api-gateway.merge.dev/v1/anthropic",
6
)
7
8
message = client.messages.create(
9
    model="anthropic/claude-haiku-4-5-20251001",
10
    max_tokens=1024,
11
    system=[
12
        {
13
            "type": "text",
14
            "text": "Here is the full onboarding handbook for Acme Corp: ...long stable document...",
15
            "cache_control": {"type": "ephemeral"},
16
        }
17
    ],
18
    messages=[
19
        {"role": "user", "content": "What is the PTO policy for a new hire in their first 90 days?"}
20
    ],
21
)

To let Anthropic place the breakpoint instead, pass a single top-level cache_control and leave the blocks unmarked:

1
message = client.messages.create(
2
    model="anthropic/claude-sonnet-5",
3
    max_tokens=1024,
4
    cache_control={"type": "ephemeral"},
5
    system=[
6
        {
7
            "type": "text",
8
            "text": "Here is the full onboarding handbook for Acme Corp: ...long stable document...",
9
        }
10
    ],
11
    messages=[
12
        {"role": "user", "content": "What is the PTO policy for a new hire in their first 90 days?"}
13
    ],
14
)

If your SDK version does not accept cache_control as a named argument, pass it as extra_body={"cache_control": {"type": "ephemeral"}} instead. The SDK merges extra_body into the request body, so the field arrives at the top level either way.

OpenAI SDK

Point the OpenAI SDK at the /v1/openai base URL. The OpenAI request shape has no place for a session field in the body, so pass the caching session hint as the X-Session-Id header. This drives automatic caching on automatic routes.

1
from openai import OpenAI
2
3
client = OpenAI(
4
    api_key="YOUR_API_KEY",
5
    base_url="https://api-gateway.merge.dev/v1/openai",
6
    default_headers={"X-Session-Id": "chat_9f2a7c"},
7
)
8
9
response = client.chat.completions.create(
10
    model="openai/gpt-4o",
11
    messages=[
12
        {
13
            "role": "system",
14
            "content": "You are a support agent for Acme Corp. Follow this policy: ...long stable document...",
15
        },
16
        {"role": "user", "content": "How do I reset a customer's API key?"},
17
    ],
18
)

Vercel AI SDK

Point the AI SDK at the /v1/ai-sdk base URL and attach the marker through providerOptions on the content part you want cached. Use this on explicit routes.

1
import { createOpenAICompatible } from "@ai-sdk/openai-compatible";
2
import { generateText } from "ai";
3
4
const merge = createOpenAICompatible({
5
  name: "merge",
6
  apiKey: "YOUR_API_KEY",
7
  baseURL: "https://api-gateway.merge.dev/v1/ai-sdk",
8
});
9
10
const { text } = await generateText({
11
  model: merge("anthropic/claude-haiku-4-5-20251001"),
12
  messages: [
13
    {
14
      role: "user",
15
      content: [
16
        {
17
          type: "text",
18
          text: "Here is the full onboarding handbook for Acme Corp: ...long stable document...",
19
          providerOptions: { anthropic: { cacheControl: { type: "ephemeral" } } },
20
        },
21
        {
22
          type: "text",
23
          text: "What is the PTO policy for a new hire in their first 90 days?",
24
        },
25
      ],
26
    },
27
  ],
28
});

Whichever spec you use, the response reports cache activity the same way, covered next.

Outputs: reading cache token usage

Gateway reports cache activity in the response usage object, normalized across providers so you read the same two fields regardless of the route.

1
{
2
  "usage": {
3
    "input_tokens": 5620,
4
    "output_tokens": 180,
5
    "total_tokens": 5800,
6
    "cache_creation_input_tokens": 0,
7
    "cache_read_input_tokens": 5533
8
  }
9
}

A first request that writes the prefix reports a positive cache_creation_input_tokens and a cache_read_input_tokens of 0. A later request that hits the cache reports a positive cache_read_input_tokens. On automatic routes the provider may report only reads.

1
usage = response.usage
2
cached = usage.cache_read_input_tokens or 0
3
total_input = usage.input_tokens or 0
4
5
if cached:
6
    print(f"Cache hit: {cached} of {total_input} input tokens served from cache")
7
else:
8
    print("No cache hit on this request")

On the OpenAI-shaped surfaces

/v1/openai/chat/completions, /v1/ai-sdk/chat/completions, and the LangChain surface report cache reads using OpenAI’s own field, so cache-aware clients and observability tooling read it without special-casing Gateway.

1
{
2
  "usage": {
3
    "prompt_tokens": 6042,
4
    "completion_tokens": 118,
5
    "total_tokens": 6160,
6
    "prompt_tokens_details": {
7
      "cached_tokens": 5533
8
    }
9
  }
10
}

prompt_tokens_details is absent when the route did not serve anything from cache, so a missing field means “not cached” rather than “cached zero tokens”. It is reported on both streaming and non-streaming responses; on a stream it arrives on the final chunk with the rest of usage.

Cache-read tokens are billed at the route’s reduced cache-read rate instead of the standard input rate, which is where the savings come from. Track cache_read_input_tokens across your traffic to measure the hit rate and the spend you avoid. For budgets and spend reporting, see Cost governance and savings.

Get the most out of caching

• Put stable content first. Order the prompt so the unchanging part (system instructions, tool schemas, shared documents) comes before the part that changes each request. The cached prefix ends at the first token that differs.
• Reuse prefixes while they are warm. Cache entries are short-lived. Caching pays off most in agent loops and multi-turn chats where repeat requests arrive within seconds.
• Keep the prefix long enough. Providers only cache prefixes above a minimum length. A short shared prefix produces no cache hit even on a caching-capable route.
• Pin model and vendor when caching matters. Caching follows the route. A routing policy that can pick a none route for some requests will not cache those, so pin the route when you depend on cache hits.

Common errors

Next steps