Cache-aware routing

When a conversation’s turns land on different vendor routes of a model, each switch rewrites the prompt cache instead of reading it. Cache-aware routing keeps a session’s follow-up requests on the vendor that served it while that cache is warm. It’s on for every organization, with no configuration.

How a session is identified

First match wins:

1. Explicit session. The X-Session-Id header (wins) or body session_id. On the /v1/openai chat completions and responses endpoints, prompt_cache_key also counts, the same hint automatic caching uses.
2. Thread or harness session. An X-Merge-Thread-Id header, or else a coding agent’s own session header: x-claude-code-session-id, session-id or session_id (Codex CLI), or x-session-affinity.
3. Conversation opener. A hash of the first system and first user messages.

Affinity is tracked per requested model or policy, so a session that mixes models tracks each one and switching policies starts fresh. It applies on /v1/responses and every compatible endpoint.

When affinity applies

Affinity only moves the remembered vendor to the front of the routing order. It never pins, never fails a request, and adds negligible latency. Ordinary vendor selection takes over when:

• The remembered vendor is unavailable or filtered out for the request
• Its cache window (per route, 10 minutes by default) has passed
• The session now resolves to a different model
• Reading that vendor’s cache is no cheaper than its own input price or the cheapest input price on the model’s other routes
• The route’s cache pricing isn’t known

It has no effect on a model with one vendor route, or when caching never engaged. On explicit caching-family routes, such as Anthropic and Claude on Bedrock, caching needs a cache_control marker on the stable prefix (the system prompt or earlier turns). Prompt caching lists each vendor’s family.

curl https://api-gateway.merge.dev/v1/responses \
  -H "Authorization: Bearer $MERGE_GATEWAY_API_KEY" \
  -H "Content-Type: application/json" \
  -H "X-Session-Id: chat_7f3a9c" \
  -d '{
    "model": "anthropic/claude-sonnet-5",
    "input": [
      {
        "type": "message",
        "role": "system",
        "content": "You are a support agent for Acme Corp. Follow this policy: ...long stable document...",
        "cache_control": {"type": "ephemeral"}
      },
      {"type": "message", "role": "user", "content": "How do I reset a customer API key?"}
    ]
  }'

Next steps