Service tiers

A service tier picks how a request is processed. flex trades latency for a lower price and is a good fit for background jobs, evals, and batch work that tolerates variable speed. standard is the default and is used when you send nothing.

Gateway treats a tier as a vendor-route capability, the same way it treats caching. The same canonical model can offer flex on one vendor and only standard on another, so the tier follows the route, not the model name alone.

How service tiers work

There are two tiers. A route supports a tier only when that route is priced for it.

Support is defined by pricing presence: Gateway advertises a tier on a route only when that route has a price for it. Ask for a tier the route does not price and Gateway returns 400 rather than serving at the wrong tier. This keeps billing honest, at the cost of requiring the route to be explicitly priced for each tier it offers.

Which vendors support flex

Every other route is standard only today, and a flex request against it fails closed. Two details worth knowing:

• Gemini flex is served by both the direct google vendor and the ZDR-compliant vertexai vendor, so a flex request reaches Gemini flex on whichever route your policy or ZDR setting selects.
• Older OpenAI models (gpt-4o, o3, o4-mini, gpt-4.1, and gpt-5 / 5.1 / 5.2 / 5.3) do not offer a flex tier.

Discover support programmatically rather than hardcoding it: GET /models reports a service_tiers list and per-tier pricing on each vendor route (see Outputs).

Inputs: requesting a tier

Send service_tier on the request. Optionally set service_tier_fallback to true so a throttled flex request retries once at standard instead of returning an error.

$
curl https://api-gateway.merge.dev/v1/responses \
>
  -H "Authorization: Bearer YOUR_API_KEY" \
>
  -H "Content-Type: application/json" \
>
  -d '{
>
    "model": "openai/gpt-5.4",
>
    "vendor": "openai",
>
    "service_tier": "flex",
>
    "service_tier_fallback": true,
>
    "input": [
>
      {
>
        "type": "message",
>
        "role": "user",
>
        "content": [
>
          { "type": "input_text", "text": "Summarize this transcript: ..." }
>
        ]
>
      }
>
    ]
>
  }'

service_tier accepts standard or flex. Any other value is rejected with 422.

Fallback on throttling

flex is best-effort at the provider, so a busy provider can throttle it (429 / 503). What happens next depends on service_tier_fallback:

• false (default) — the throttle surfaces to you as the provider’s 429 / 503, and nothing is billed. Retry later, or resend at standard.
• true — Gateway retries the request once on the same route at standard, billed at the standard rate. The service_tier in the response shows the tier that actually served, so a fallback shows standard even though you asked for flex.

Set service_tier_fallback: true when you would rather get a standard-priced answer than an error under load, and leave it off when a flex request must either run at the flex price or not at all.

Outputs: reading the served tier

Every response carries a top-level service_tier — the tier that actually served the request, and the tier you were billed at. On a fallback this differs from what you requested.

1
{
2
  "model": "openai/gpt-5.4",
3
  "vendor": "openai",
4
  "service_tier": "flex",
5
  "usage": {
6
    "input_tokens": 812,
7
    "output_tokens": 180,
8
    "total_tokens": 992
9
  }
10
}

Bill off service_tier, not the tier you sent: it reflects the rate charged, including the standard you land on after a throttle fallback. Track it across your traffic to measure flex savings.

To discover which routes support which tiers, read GET /models. Each vendor route reports a service_tiers list and per-tier pricing:

1
{
2
  "model": "openai/gpt-5.4",
3
  "vendors": {
4
    "openai": {
5
      "service_tiers": ["standard", "flex"],
6
      "pricing": {
7
        "input_per_million": 2.5,
8
        "output_per_million": 15,
9
        "flex": { "input_per_million": 1.25, "output_per_million": 7.5 }
10
      }
11
    }
12
  }
13
}

A tier missing from service_tiers isn’t offered on that route — request it and you get the fail-closed 400.

Best practices

• Use flex for latency-tolerant work. Batch scoring, evals, offline enrichment, and background agents are ideal; interactive, user-facing calls usually are not.
• Add service_tier_fallback when an answer beats an error. For flex traffic that must still complete under load, fallback trades the discount for reliability on the throttled subset.
• Pin model and vendor when the tier matters. A routing policy can land on a route that does not price the tier, which fails closed. Pin the route, or configure the policy with routes that all support the tier.
• Discover support at runtime. Read service_tiers from GET /models instead of hardcoding which models offer flex, since the catalog changes.

Common errors

Next steps