Prompt caching

Prompt caching lets a vendor reuse a prompt prefix it already processed, so cached tokens bill at a lower rate and skip reprocessing. Use it when requests share a long system prompt, tool schemas, a document, or a growing agent conversation.

Caching follows the vendor route, not the model, and never fails a request: if a route can’t honor a cache marker, Gateway removes it.

How caching works

Each route belongs to one caching family.

On an explicit route with no marker, Gateway adds one when the request has an earlier assistant turn, tool definitions, or a session hint: top-level on anthropic, and on the latest user, system, or tool message elsewhere, so the cached prefix grows with the conversation. OpenAI-shaped clients such as Codex, Cursor, and the OpenAI SDK cache against Claude without client changes. A single-turn request with no tools or session hint goes as-is, since a cache write only pays off on reuse.

/v1/models doesn’t list the family; check the route’s pricing.cache_read_per_million and pricing.cache_write_per_million. A route with no cache-read price bills cached reads at the full input rate. Automatic caching is best-effort, and vendors only cache prefixes above a minimum length.

Mark a cached prefix

On explicit routes, put cache_control on the last block of the stable prefix. That block and everything before it are cached, so put unchanging content (instructions, tool definitions, shared documents) first.

curl https://api-gateway.merge.dev/v1/responses \
  -H "Authorization: Bearer $MERGE_GATEWAY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "anthropic/claude-haiku-4-5-20251001",
    "max_tokens": 1024,
    "input": [
      {
        "type": "message",
        "role": "user",
        "content": [
          {"type": "text", "text": "Acme Corp onboarding handbook: ...long stable document...", "cache_control": {"type": "ephemeral"}},
          {"type": "text", "text": "What is the PTO policy for a new hire in their first 90 days?"}
        ]
      }
    ]
  }'

Markers work on text, image, document, audio, and video blocks, and on a whole message. Or send one top-level cache_control: on anthropic, Gateway forwards it and Anthropic moves the breakpoint forward as the conversation grows; on other explicit routes, Gateway converts it to a marker on the latest message.

Markers accept Anthropic’s ttl values, 5m (default) or 1h, passed through unchanged. Anthropic rejects a request whose top-level marker and targeted block carry different ttl values. Every cache write bills at the route’s single write rate, whatever the ttl.

Send a session hint

On automatic routes, keep the stable content first. To group related requests, add a session hint with the X-Session-Id header or session_id body field. The header wins when both are sent, and values over 256 characters are truncated, not rejected.

curl https://api-gateway.merge.dev/v1/responses \
  -H "Authorization: Bearer $MERGE_GATEWAY_API_KEY" \
  -H "Content-Type: application/json" \
  -H "X-Session-Id: chat_9f2a7c" \
  -d '{
    "model": "openai/gpt-5.5",
    "input": [
      {"type": "message", "role": "system", "content": "You are a support agent for Acme Corp. Follow this policy: ...long stable document..."},
      {"type": "message", "role": "user", "content": "How do I reset a customer'\''s API key?"}
    ]
  }'

On OpenAI and Azure, Gateway sends the hint as the vendor’s cache key, scoped to your organization so another tenant’s identical value never shares your cache. On explicit routes it signals reuse, so Gateway places a marker. Cache-aware routing uses it to keep a session on the vendor whose cache is warm. On /v1/openai, a body prompt_cache_key also works as the hint.

Read cache usage

On /v1/responses and /v1/anthropic, input_tokens counts the full prompt, including cached reads and writes.

{
  "usage": {
    "input_tokens": 5620,
    "output_tokens": 180,
    "total_tokens": 5800,
    "cache_creation_input_tokens": 0,
    "cache_read_input_tokens": 5533
  }
}

A new prefix reports a write in cache_creation_input_tokens, which is null on automatic routes whose vendor doesn’t count writes. Reuse reports reads in cache_read_input_tokens. Track that for hit rate, and usage.cost for the dollar effect.

Reads bill at cache_read_per_million. Writes bill at cache_write_per_million (1.25x the input rate on Anthropic and Claude on Bedrock), or the input rate on routes that don’t price writes.

Reference

Streams carry the usage fields on the final chunk.

No cache tokens? Check the route’s family (markers on automatic and none routes are stripped), that the marked prefix is byte-for-byte identical across requests, that repeats arrive before the entry expires, and that the prefix meets the vendor’s minimum length. A routing policy or alias spanning families caches only on some requests, so pin vendor or keep its routes in one family when you depend on hits.

Next steps