Rate limits

Gateway’s limiting model is spend-first: it does not impose request-per-minute or token-per-minute caps on your API keys by default. Your throughput is governed by the providers’ own capacity, and your protection against runaway usage comes from budgets and spend limits. This page covers the limits that do exist and how to handle them.

Limits Gateway enforces

Requests served on your own BYOK credentials skip the managed-credential limits entirely; you get whatever capacity your provider account has.

Provider rate limits

When an upstream provider throttles a request, what happens depends on your routing:

• With a routing policy, a provider 429 counts as a provider failure and triggers failover to the next candidate, so most throttles never reach you.
• Without a policy (direct model calls), the 429 passes through with type: "rate_limit_error" and source: "provider". No Retry-After header is forwarded; back off and retry.
• On the flex service tier, set service_tier_fallback: true to retry once at standard when flex capacity is throttled. See Service tiers.

Handling a 429

Retry with exponential backoff, and honor Retry-After when present (Gateway-issued 429s carry it):

1
import time, random
2
import requests
3
4
def create_response(payload, max_attempts=5):
5
    for attempt in range(max_attempts):
6
        resp = requests.post(
7
            "https://api-gateway.merge.dev/v1/responses",
8
            headers={"Authorization": "Bearer YOUR_API_KEY"},
9
            json=payload,
10
        )
11
        if resp.status_code != 429:
12
            return resp
13
        retry_after = float(resp.headers.get("Retry-After", 2 ** attempt))
14
        time.sleep(retry_after + random.uniform(0, 1))
15
    return resp

The stronger fix is structural: a Priority routing policy turns provider throttles into automatic failover instead of client-side retries.

Rate-limit headers

Responses carry X-RateLimit-Limit, X-RateLimit-Remaining, and X-RateLimit-Reset. When no per-key limit is configured (the default), all three read 0; they become meaningful only for keys that have explicit limits set. Retry-After appears only on Gateway-issued 429s.

What Gateway does not limit

There is no cap on request body size beyond the inline-media limits, no max_tokens ceiling beyond the model’s own, and no concurrency cap. If you need a hard ceiling on what a workload can consume, use a project budget or per-key spend limit; those are enforced before any provider is called.

Next steps