Batch inference

Batch inference runs up to 10,000 requests as one job on the vendor’s batch API at the route’s batch price (typically 50% of standard), finishing or timing out within 24 hours. Use it for work that can wait: evals, backfills, classification, and embeddings.

Run a batch

Submit with POST /v1/batches, poll until it finishes, then read the results. Each line has a custom_id and a body shaped for endpoint.

# 1. Submit (returns 202 and the batch object)
curl https://api-gateway.merge.dev/v1/batches \
  -H "Authorization: Bearer $MERGE_GATEWAY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "endpoint": "/v1/chat/completions",
    "model": "anthropic/claude-sonnet-4-6",
    "metadata": { "job": "ticket-triage" },
    "requests": [
      { "custom_id": "ticket-1041", "body": { "messages": [{ "role": "user", "content": "Classify: My invoice is wrong." }], "max_tokens": 50 } },
      { "custom_id": "ticket-1042", "body": { "messages": [{ "role": "user", "content": "Classify: I cannot log in." }], "max_tokens": 50 } }
    ]
  }'
# 2. Poll until status is completed, failed, or cancelled
curl https://api-gateway.merge.dev/v1/batches/batch_2f0c8a91d4e64b7c9a1e0b3d5f7a9c21 \
  -H "Authorization: Bearer $MERGE_GATEWAY_API_KEY"
# 3. Read results (newline-delimited JSON, one line per request)
curl https://api-gateway.merge.dev/v1/batches/batch_2f0c8a91d4e64b7c9a1e0b3d5f7a9c21/results \
  -H "Authorization: Bearer $MERGE_GATEWAY_API_KEY"

The batch object reports status, per-outcome request_counts, expires_at, your metadata, and usage.cost once the batch is billed. Match result lines by custom_id, since they arrive in any order. Each line’s result.type is succeeded (with a response in the endpoint’s normal shape), errored, expired, or canceled (one L, unlike the batch status cancelled).

Submit validation is all or nothing: one invalid line rejects the whole batch. Once running, each line succeeds or fails on its own.

The dashboard lists batches under Batches.

How Gateway picks the vendor

A batch runs on one vendor for its whole life. After applying your organization’s vendor, region, and zero data retention rules, Gateway keeps routes that can run the endpoint and have a batch price, then picks the cheapest (or your BYOK route, if your organization prefers its own keys). "provider": {"only": ["anthropic"]} narrows the choice but can’t widen your rules.

In GET /v1/models, a batch-capable route lists batch in service_tiers, a pricing.batch price, and batch_endpoints. The batch tier applies only to /v1/batches, not to service_tier on normal requests.

Billing

Each line is priced like a normal request (prompt caching, long-context, and time-of-day pricing included), then discounted. The discount ratio is fixed at submit and applied to the route’s standard rates at billing time, so a mid-run standard price change still counts. Lines are charged once, when results are collected; errored, expired, and cancelled lines are free. On a BYOK key the vendor bills you and Gateway charges only its fee. On a prepaid balance, a batch whose estimate plus your other unbilled batches exceeds the balance returns 402 batch_budget_exceeded.

Reference

Endpoints

Every line: custom_id is unique, 1 to 64 characters of letters, digits, _, or -. A line model must match the batch model. stream: true, web search tools, and search models are rejected. On Anthropic chat routes, parameters Anthropic batch can’t honor (such as response_format, n above 1, logprobs, reasoning_effort, seed, thinking) return 400 unsupported_params. OpenAI routes get the line body as sent, with model set to the vendor’s model name.

Media on chat lines: images and files go by public http or https URL, and the vendor fetches them. Anthropic accepts images and PDFs ({"type": "file", "file": {"file_data": "https://..."}}), OpenAI accepts images, and Parasail accepts neither. data: URIs, base64, private or localhost addresses, and vendor file IDs return 400 unsupported_batch_media. If your organization has DLP rules, any batch with media returns 422 batch_media_blocked_by_dlp, since DLP can’t scan media.

Statuses

Limits

Errors

On OpenAI, a full vendor queue instead shows as a failed batch with error.code provider_batch_queue_full. It wasn’t run or billed; resubmit later or split it.

Next steps