Batch inference
Batch inference runs many requests as one job. You submit up to 10,000 requests in a single call, Gateway hands them to the provider’s batch API, and you collect the results when the job finishes. Most batches finish within minutes, and every batch finishes or expires within 24 hours. In exchange for waiting, you pay the route’s batch price, typically 50% of standard.
Use batch for work that doesn’t need an answer right away: evals, backfills, classification, document processing, and embeddings. For interactive traffic, send normal requests.
How batch works
- You send
POST /v1/batcheswith anendpoint, amodel, and arequestsarray. Each request has acustom_idand abodyin the same shape you’d send to that endpoint. - Gateway validates every line, screens it with your DLP rules, checks your balance, and submits the whole batch to one provider.
- You poll
GET /v1/batches/{id}until the batch reaches a finished status. - You read the results from
GET /v1/batches/{id}/results, one line per request, matched bycustom_id. - Gateway bills each request line once, when its result comes back. Lines that errored, expired, or were cancelled cost nothing.
One bad request doesn’t fail the batch. Each line succeeds or fails on its own.
Supported endpoints and providers
A route supports batch when its provider can run the batch’s endpoint and the route has a batch price. To see which models qualify, check GET /v1/models: a batch-capable vendor route lists batch in service_tiers, a pricing.batch price, and the batch endpoints it runs in batch_endpoints.
batch in service_tiers only applies to POST /v1/batches; you can’t send service_tier: "batch" on a normal request.
If no route you’re allowed to use can batch the model, the submit fails closed with 403 batch_not_supported and nothing is sent to the provider.
Choosing a provider
Each batch runs on one provider for its whole life. Gateway picks it the same way it picks a vendor for a normal request: your organization’s vendor, region, and zero data retention rules apply first. It then keeps only routes that can batch the endpoint and chooses the cheapest. If your organization prefers its own provider keys, an eligible BYOK route wins over a cheaper managed one.
To limit a batch to specific providers, add provider.only:
provider.only can narrow your organization’s rules but never widen them. If none of the listed providers can run the batch, Gateway returns 400 vendor_unavailable. only is the only supported key in provider.
Quickstart
This script submits a two-line batch, waits for it to finish, and prints each result.
The sections below cover each step in detail.
Submit a batch
Gateway responds with 202 and the batch object:
Rules for each line:
custom_idis required, unique within the batch, 1 to 64 characters, and uses only letters, digits,_, and-bodyis what you’d send to the batch’sendpoint. If a line setsmodel, it must match the batch’smodel.- Chat lines need a non-empty
messagesarray.stream: trueisn’t supported in batch. - Web search isn’t supported in batch, because search calls are billed separately from tokens. Lines with web search tools or
web_search_options, and search models such as those ending in:online, are rejected with422 web_search_unsupported. - Audio and video aren’t supported. Images and files are supported by public URL; see Images and files.
On Anthropic routes, Gateway forwards messages (including system messages), max_tokens, temperature, top_p, stop, tools, and tool_choice. Parameters that would change the output but that Anthropic batch can’t honor, such as response_format, n greater than 1, logprobs, reasoning_effort, and seed, are rejected with 400 unsupported_params instead of being ignored. For native Anthropic features, send a Messages batch. OpenAI routes receive the line body unchanged.
Track a batch
Poll the batch until its status is completed, failed, expired, or cancelled. Every few minutes is often enough.
You can also follow your organization’s batches on the Batches page in the Gateway dashboard.
request_counts shows how many requests completed, failed, expired, or were cancelled. After the batch is billed, usage reports its token totals, cost (what Gateway billed for the batch), and is_byok (whether it ran on your own provider key).
Read the results
Results are available once the batch is completed, or cancelled after some lines finished, and they stay available for 29 days. The response is newline-delimited JSON with one line per request. Match lines to your requests by custom_id, because they can arrive in any order.
result.type is succeeded, errored, expired, or canceled. A succeeded line has a response in the endpoint’s normal response shape. Every other type has an error and isn’t billed.
Asking for results before any are ready returns 409 batch_results_not_ready.
Cancel a batch
The batch moves to cancelling, then cancelled. Requests that finished before the cancel are kept, returned in the results, and billed. The rest are marked canceled and cost nothing. Cancelling a batch that already finished returns 409 batch_not_cancellable.
List batches
GET /v1/batches returns your organization’s batches, newest first. Keys scoped to a customer see only that customer’s batches.
The response has data, first_id, last_id, and has_more. A page only comes back short when nothing else matches.
Delete a batch
Delete a finished batch to remove it from Gateway and from the provider.
deletion.upstream.status is deleted, failed, unsupported, or not_applicable. If the provider cleanup fails, the batch is still deleted from Gateway and the response says so. Deleting doesn’t remove the batch’s usage and billing history.
A batch can be deleted only once it has finished and been billed. Before that, the request returns 409 batch_not_deletable. After a delete, the batch’s status, results, and delete endpoints all return 404.
Deleting a batch is permanent. Its results can’t be recovered afterwards.
Embeddings batches
Set endpoint to /v1/embeddings and give each line an input: a string, an array of strings, a token array, or an array of token arrays. You can also pass dimensions, encoding_format, and user.
Embeddings batches run on OpenAI, are billed on input tokens only, and allow up to 50,000 inputs per batch.
Messages batches
Set endpoint to /v1/messages to send native Anthropic Messages requests. Each line’s body is a Messages request, and the results are Anthropic message objects rather than chat completions. Messages batches run on Anthropic.
max_tokensis required (400 max_tokens_required), andmessagesmust be non-empty- Message content must be text. Tool definitions and
tool_choiceare supported, buttool_useandtool_resultblocks in the conversation history aren’t yet systemis screened by your DLP rules like the rest of the request
Messages lines are billed exactly like the equivalent chat line, including cache reads and writes.
Responses batches
Set endpoint to /v1/responses to send OpenAI Responses requests. Each line’s body is a Responses request, and the results are OpenAI Response objects. Responses batches run on OpenAI.
inputis required and must be non-empty (400 empty_input)- Each line runs on its own, so
background,previous_response_id, andconversationare rejected with400 unsupported_params instructionsis screened by your DLP rules like the rest of the request
Reasoning tokens are billed as output and cached input as cache reads, the same as the equivalent chat line.
Images and files
Chat lines can include images and files by public http or https URL. The provider downloads the file; Gateway never fetches it.
For a PDF on Anthropic, use a file part: { "type": "file", "file": { "file_data": "https://assets.acme.com/contracts/msa.pdf" } }.
These are rejected with 400 unsupported_batch_media: data: URIs and base64 content, non-http(s) URLs, private or localhost addresses, provider file IDs, and media the chosen provider doesn’t accept. Media is only allowed in user messages.
DLP rules can’t scan images or files. If your organization has DLP rules, batches with images or files are rejected with 422 batch_media_blocked_by_dlp, so no unscreened content reaches a provider.
Billing
- Price: each line is priced like a normal request, including prompt caching, long-context, and time-of-day pricing, and then the route’s batch discount is applied to the whole line. Cached tokens get the batch discount too.
- Locked at submit: the discount is fixed when you submit the batch, so a price change while it runs doesn’t change its bill.
- When you’re charged: once per line, when results come back. Errored, expired, and cancelled lines are free.
- Your own keys: on a BYOK key, the provider bills you directly and Gateway charges only its usual fee.
- Prepaid balances: a new batch is refused with
402 batch_budget_exceededif its estimated cost, plus the estimates of your batches that haven’t been billed yet, is more than your remaining balance.
See Service tiers for the interactive flex tier, the other way to trade latency for price.
Limits
On OpenAI, Gateway deletes the uploaded input file when the batch ends, and all batch files expire after 30 days.
Errors
My batch failed with a queue-full error
On a managed key, the provider limits how much batch work can be queued at once. The batch wasn’t run and wasn’t billed. Resubmit it later, or split it into smaller batches.
My batch has been in_progress for hours
Providers finish most batches within minutes, but they can take up to 24 hours. Anything unfinished at 24 hours expires, and expired lines aren’t billed.
403 batch_not_supported for a model I use every day
The model’s route doesn’t have a batch price yet, or its provider doesn’t run this endpoint in batch. Try another supported model, or contact your Merge account team.