Cache-aware routing

When a model has more than one vendor route, consecutive requests in the same conversation can land on different vendors. Each vendor keeps its own prompt cache, so every switch means the prefix gets written again at the new vendor instead of read from the old one. Cache-aware routing remembers which vendor served a session and keeps follow-up requests there while that vendor’s cache is still warm.

How it works

Gateway identifies a session by checking, in order, and using the first match:

1. A session_id in the request body, or the X-Session-Id header (the header wins if both are sent). This is the same session hint used for automatic caching. On the OpenAI-compatible Codex route, a prompt_cache_key maps to this.
2. A session header from your harness, such as x-claude-code-session-id, x-session-id, or x-session-affinity.
3. When neither is present, a hash of the conversation opener (the system prompt plus the first user message).

The model is part of the identity, so a session that mixes models keeps a separate affinity per model. The routing policy is too: switching policies starts a fresh affinity.

When affinity is dropped

Affinity only changes the order Gateway considers routes in. If the remembered vendor is unavailable, its cache entry has expired, or starting cold somewhere else is cheaper than reading its cache, the request routes normally. It can’t make a request fail and it doesn’t add latency.

The price check matters in practice: Gateway compares the remembered route’s cache-read rate against full input price on the best alternative, and drops the affinity as soon as it stops winning that comparison.

Get the benefit

Affinity only helps if the provider cache is in use to begin with, which takes two things:

• Send an identity on every request in the session: a session_id (or the X-Session-Id header), or a harness session header like x-claude-code-session-id.
• On providers in the explicit caching family (Anthropic, Claude on Bedrock, Amazon Nova on Bedrock), put the cache_control breakpoint on the stable prefix, meaning your system prompt or earlier turns. A marker at the end of a prompt that changes every turn never gets a cache hit. See Prompt caching for the per-provider details.

Requests routed through a routing policy work the same way; the resolved policy is part of the session identity.

$
curl https://api-gateway.merge.dev/v1/responses \
>
  -H "Authorization: Bearer YOUR_API_KEY" \
>
  -H "Content-Type: application/json" \
>
  -H "X-Session-Id: chat_7f3a9c" \
>
  -d '{
>
    "model": "anthropic/claude-sonnet-5",
>
    "max_tokens": 1024,
>
    "input": [
>
      {
>
        "type": "message",
>
        "role": "system",
>
        "content": "You are a support agent for Acme Corp. Follow this policy: ...long stable document...",
>
        "cache_control": { "type": "ephemeral" }
>
      },
>
      {
>
        "type": "message",
>
        "role": "user",
>
        "content": "How do I reset a customer'\''s API key?"
>
      }
>
    ]
>
  }'

The X-Session-Id header carries the session identity across turns. A body session_id field works the same way, and the header wins if you send both.

When it does nothing

• The model has only one vendor route, so there’s no routing decision to influence
• Caching never engaged: no session identity on the request, or no cache_control marker on an explicit route
• The feature hasn’t reached your organization yet, in which case requests route the same as before

Next steps