> This page is for Gateway. > For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.merge.dev/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.merge.dev/_mcp/server. # Batch inference > Submit, track, and collect batches on Gateway across chat, embeddings, Messages, and Responses, plus how batch lines are billed and the limits and errors to plan for. Batch inference runs up to 10,000 requests as one job on the vendor's batch API at the route's batch price (typically 50% of standard), finishing or timing out within 24 hours. Use it for work that can wait: evals, backfills, classification, and embeddings. ## Run a batch Submit with `POST /v1/batches`, poll until it finishes, then read the results. Each line has a `custom_id` and a `body` shaped for `endpoint`. **`cURL`** ```bash title="cURL" # 1. Submit (returns 202 and the batch object) curl https://api-gateway.merge.dev/v1/batches \ -H "Authorization: Bearer $MERGE_GATEWAY_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "endpoint": "/v1/chat/completions", "model": "anthropic/claude-sonnet-4-6", "metadata": { "job": "ticket-triage" }, "requests": [ { "custom_id": "ticket-1041", "body": { "messages": [{ "role": "user", "content": "Classify: My invoice is wrong." }], "max_tokens": 50 } }, { "custom_id": "ticket-1042", "body": { "messages": [{ "role": "user", "content": "Classify: I cannot log in." }], "max_tokens": 50 } } ] }' # 2. Poll until status is completed, failed, or cancelled curl https://api-gateway.merge.dev/v1/batches/batch_2f0c8a91d4e64b7c9a1e0b3d5f7a9c21 \ -H "Authorization: Bearer $MERGE_GATEWAY_API_KEY" # 3. Read results (newline-delimited JSON, one line per request) curl https://api-gateway.merge.dev/v1/batches/batch_2f0c8a91d4e64b7c9a1e0b3d5f7a9c21/results \ -H "Authorization: Bearer $MERGE_GATEWAY_API_KEY" ``` **`Python`** ```python title="Python" # pip install requests import json, os, time import requests BASE = "https://api-gateway.merge.dev/v1/batches" HEADERS = {"Authorization": f"Bearer {os.environ['MERGE_GATEWAY_API_KEY']}"} batch = requests.post(BASE, headers=HEADERS, json={ "endpoint": "/v1/chat/completions", "model": "anthropic/claude-sonnet-4-6", "requests": [ {"custom_id": "ticket-1041", "body": {"messages": [{"role": "user", "content": "Classify: My invoice is wrong."}], "max_tokens": 50}}, {"custom_id": "ticket-1042", "body": {"messages": [{"role": "user", "content": "Classify: I cannot log in."}], "max_tokens": 50}}, ], }).json() while batch["status"] not in {"completed", "failed", "cancelled"}: time.sleep(60) batch = requests.get(f"{BASE}/{batch['id']}", headers=HEADERS).json() if batch["status"] != "failed": for line in requests.get(f"{BASE}/{batch['id']}/results", headers=HEADERS).text.splitlines(): row = json.loads(line) result = row["result"] if result["type"] == "succeeded": print(row["custom_id"], result["response"]["choices"][0]["message"]["content"]) else: print(row["custom_id"], result["type"], result.get("error")) ``` The batch object reports `status`, per-outcome `request_counts`, `expires_at`, your `metadata`, and `usage.cost` once the batch is billed. Match result lines by `custom_id`, since they arrive in any order. Each line's `result.type` is `succeeded` (with a `response` in the endpoint's normal shape), `errored`, `expired`, or `canceled` (one L, unlike the batch status `cancelled`). Submit validation is all or nothing: one invalid line rejects the whole batch. Once running, each line succeeds or fails on its own. | Request | Behavior | | ------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `POST /v1/batches/{id}/cancel` | Stops the batch. Finished lines are kept and billed, and the rest come back `canceled`. | | `GET /v1/batches` | Lists batches newest first. A key scoped to a customer sees only that customer's. Takes `limit` (1 to 100, default 20), `after` (the previous page's `last_id`), a repeatable `status`, and `created_after` or `created_before` as unix seconds or ISO 8601. | | `DELETE /v1/batches/{id}` | Permanently removes a finished batch from Gateway and the vendor but keeps its billing history | The dashboard lists batches under **Batches**. ## How Gateway picks the vendor A batch runs on one vendor for its whole life. After applying your organization's vendor, region, and zero data retention rules, Gateway keeps routes that can run the endpoint and have a batch price, then picks the cheapest (or your BYOK route, if your organization prefers its own keys). `"provider": {"only": ["anthropic"]}` narrows the choice but can't widen your rules. In `GET /v1/models`, a batch-capable route lists `batch` in `service_tiers`, a `pricing.batch` price, and `batch_endpoints`. The `batch` tier applies only to `/v1/batches`, not to `service_tier` on normal requests. ## Billing Each line is priced like a normal request (prompt caching, long-context, and time-of-day pricing included), then discounted. The discount ratio is fixed at submit and applied to the route's standard rates at billing time, so a mid-run standard price change still counts. Lines are charged once, when results are collected; errored, expired, and cancelled lines are free. On a BYOK key the vendor bills you and Gateway charges only its fee. On a prepaid balance, a batch whose estimate plus your other unbilled batches exceeds the balance returns `402 batch_budget_exceeded`. ## Reference ### Endpoints | `endpoint` | Vendors | Line rules | | ---------------------- | --------------------------- | ----------------------------------------------------------------------------------------------------------------------- | | `/v1/chat/completions` | Anthropic, OpenAI, Parasail | Non-empty `messages`. Images and PDFs by public URL (see below) | | `/v1/embeddings` | OpenAI, Parasail | `input` as a string, string array, or token arrays. Billed on input only | | `/v1/messages` | Anthropic | `max_tokens` required. Text content only. `tools` and `tool_choice` allowed, `tool_use` and `tool_result` blocks aren't | | `/v1/responses` | OpenAI | Non-empty `input`. `background`, `previous_response_id`, and `conversation` are rejected | Every line: `custom_id` is unique, 1 to 64 characters of letters, digits, `_`, or `-`. A line `model` must match the batch `model`. `stream: true`, web search tools, and search models are rejected. On Anthropic chat routes, parameters Anthropic batch can't honor (such as `response_format`, `n` above 1, `logprobs`, `reasoning_effort`, `seed`, `thinking`) return `400 unsupported_params`. OpenAI routes get the line body as sent, with `model` set to the vendor's model name. **Media on chat lines:** images and files go by public `http` or `https` URL, and the vendor fetches them. Anthropic accepts images and PDFs (`{"type": "file", "file": {"file_data": "https://..."}}`), OpenAI accepts images, and Parasail accepts neither. `data:` URIs, base64, private or `localhost` addresses, and vendor file IDs return `400 unsupported_batch_media`. If your organization has DLP rules, any batch with media returns `422 batch_media_blocked_by_dlp`, since DLP can't scan media. ### Statuses | Status | Meaning | | ------------- | ------------------------------------------------------------------------------- | | `validating` | Accepted, being submitted to the vendor | | `in_progress` | The vendor is running the requests | | `finalizing` | The vendor finished and results aren't published yet | | `completed` | Results are ready. Lines that ran out of time count in `request_counts.expired` | | `failed` | The batch didn't run. `error` says why | | `cancelling` | A cancel is in progress | | `cancelled` | Cancelled. Lines that finished first are kept and billed | ### Limits | Limit | Value | | ----------------------------- | -------- | | Requests per batch | 10,000 | | Request body | 50 MB | | Embedding inputs per batch | 50,000 | | Open batches per organization | 25 | | Completion window | 24 hours | | Results retention | 29 days | ### Errors | Status | Code | Cause | | ------ | ----------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------ | | `400` | `unsupported_batch_endpoint`, `empty_batch`, `invalid_custom_id`, `duplicate_custom_id`, `batch_model_mismatch` | The batch or a `custom_id` is malformed | | `400` | `stream_unsupported`, `empty_messages`, `empty_input`, `max_tokens_required`, `invalid_request_body`, `unsupported_params`, `unsupported_batch_media` | A line can't run in a batch. The message names the `custom_id` | | `400` | `vendor_unavailable`, `unsupported_parameter` | `provider.only` matches no eligible vendor, or `provider` has another key | | `402` | `batch_budget_exceeded` | The batch doesn't fit your prepaid balance | | `403` | `batch_not_supported` | No route you can use has a batch price for this model and endpoint | | `404` | `batch_not_found` | Unknown or deleted batch | | `409` | `batch_results_not_ready`, `batch_not_cancellable`, `batch_not_deletable` | Wrong status for the action. Delete needs a finished batch with nothing left to bill | | `413` | `batch_body_too_large`, `batch_too_large` | Over a size limit | | `422` | `blocked_by_dlp_policy`, `batch_media_blocked_by_dlp` | DLP blocked a line, or the batch has media under DLP | | `422` | `web_search_unsupported` | A line or the model uses web search | | `429` | `too_many_open_batches`, `batch_submit_busy`, `provider_batch_queue_full`, `provider_rate_limited` | At capacity. Retry later, honoring `Retry-After` when present | On OpenAI, a full vendor queue instead shows as a `failed` batch with `error.code` `provider_batch_queue_full`. It wasn't run or billed; resubmit later or split it. ## Next steps #### [Service tiers](/merge-gateway/capabilities/service-tiers) Flex-tier discounts for interactive requests that can wait a little #### [Prompt caching](/merge-gateway/capabilities/prompt-caching) Cut the cost of batch lines that share a system prompt