> This page is for Gateway. > For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.merge.dev/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.merge.dev/_mcp/server. # Evals API > The /v1/evals and /v1/migrations endpoints: gate CI on a suite, sync datasets from your repo, manage schedules, and automate migrations. The Evals API runs suites and migrations from automation, such as a CI job that blocks a deploy on a failing suite. Schemas are in the [Evals API](/merge-gateway/evals-api) and [Migrations API](/merge-gateway/migrations-api) reference. Authenticate with your organization-level `mg_` key (the one you use for `/v1/responses`); [customer API keys](/merge-gateway/embedded-routing-stack/customer-api-keys) are rejected. ## Gate CI on a suite Start a run, poll until status leaves `pending` or `running`, then branch on `passed` (there's no blocking mode). **`cURL`** ```bash title="cURL" BASE="https://api-gateway.merge.dev" AUTH="Authorization: Bearer $MERGE_GATEWAY_API_KEY" SUITE_ID="11a4c3f0-8b2e-4c1d-9f6a-2e7b5d3c9a01" RUN_ID=$(curl -s -X POST "$BASE/v1/evals/suites/$SUITE_ID/runs" \ -H "$AUTH" -H "Content-Type: application/json" \ -d '{"target_model": "anthropic/claude-opus-5", "trials": 3, "idempotency_key": "ci-8412"}' | jq -r .id) while true; do RUN=$(curl -s "$BASE/v1/evals/runs/$RUN_ID" -H "$AUTH") STATUS=$(jq -r .status <<<"$RUN") [ "$STATUS" = "pending" ] || [ "$STATUS" = "running" ] || break sleep 10 done jq -r '"\(.status): \(.pass_count)/\(.pass_count + .fail_count + .error_count) passed"' <<<"$RUN" [ "$(jq -r .passed <<<"$RUN")" = "true" ] # nonzero exit fails the build ``` **`Python`** ```python title="Python" import os import time import httpx BASE = "https://api-gateway.merge.dev" HEADERS = {"Authorization": f"Bearer {os.environ['MERGE_GATEWAY_API_KEY']}"} SUITE_ID = "11a4c3f0-8b2e-4c1d-9f6a-2e7b5d3c9a01" run = httpx.post( f"{BASE}/v1/evals/suites/{SUITE_ID}/runs", headers=HEADERS, json={"target_model": "anthropic/claude-opus-5", "trials": 3, "idempotency_key": "ci-8412"}, ).raise_for_status().json() while run["status"] in ("pending", "running"): time.sleep(10) run = httpx.get(f"{BASE}/v1/evals/runs/{run['id']}", headers=HEADERS).json() graded = run["pass_count"] + run["fail_count"] + run["error_count"] print(f"{run['status']}: {run['pass_count']}/{graded} passed") assert run["passed"], "eval suite failed" ``` A run ends `completed`, `failed` (couldn't execute), or `cancelled`; `passed` is `null` unless it completed. Completed runs also carry `pass_rate`, its 95% interval (`ci_low`, `ci_high`), `total_cost_usd`, and `trace_id`. ### Trigger fields | Field | Description | | ------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `target_model` | A catalog model ID | | `trials` | 1 to 5, default 1. A case passes only when every trial passes. | | `idempotency_key` | Up to 128 characters. A repeat returns the original run with `200`, so use your CI job ID. | | `metadata` | Up to 16 string pairs (keys up to 64 characters, values up to 256), such as git SHA. Echoed on every read. | | `request_overrides` | `system` prepends a system prompt to every case, and `params` overrides `temperature`, `max_output_tokens`, `response_format`, `tools`, or `tool_choice`. Use it to test a new prompt on the same cases. | | `mode`, `outputs` | Grade your own outputs. See [External runs](#external-runs). | ## Dataset as code Keep cases in your repo so a prompt change ships with its cases. `PUT /v1/evals/suites` creates (`201`) or updates (`200`) a suite by name. `PUT /v1/evals/suites/{suite_id}/cases` syncs cases by `external_id`: it updates matches, creates new ones, and deletes synced cases missing from the payload unless you send `"prune": false`. **`cURL`** ```bash title="cURL" curl -s -X PUT "https://api-gateway.merge.dev/v1/evals/suites/$SUITE_ID/cases" \ -H "Authorization: Bearer $MERGE_GATEWAY_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "cases": [ { "external_id": "refund-policy", "name": "Refund policy answer", "input_messages": [{"role": "user", "content": "What is your refund window?"}], "graders": [{"type": "contains", "value": "30 days"}] } ] }' # {"created": 0, "updated": 1, "deleted": 0, "unchanged": 0} ``` Syncs never touch dashboard-authored cases (no `external_id`), and overwrite dashboard edits to a **Synced** case. ## External runs When your pipeline runs the model itself (a local checkpoint or your full agent loop), send `"mode": "external"` with one `outputs` entry per case, matched by `external_id` or case name and carrying `output_text`, `output_json`, or `tool_calls`. ```json { "target_model": "support-agent-v2", "mode": "external", "outputs": [ {"external_id": "refund-policy", "output_text": "Refunds are accepted within 30 days."}, {"name": "Greeting tone", "output_json": {"tone": "friendly"}} ] } ``` `target_model` is a free-text label, trials are always 1, LLM judge graders still bill, and an enabled case with no output fails. > **External runs can set a migration's eval gate** > > The gate uses the latest completed run of either mode whose `target_model` matches the candidate. An external run labeled with the candidate's exact model ID sets the verdict without the candidate running through Gateway, so pick another label unless you intend that. ## Schedules `GET`, `PUT`, and `DELETE` on `/v1/evals/suites/{suite_id}/schedule` manage the schedule. `PUT` takes `target_model`, `interval_hours` (6, 12, 24, or 168), `trials`, and `is_enabled`, and restarts the clock. ## Migrations `POST /v1/migrations` creates a [migration](/merge-gateway/evals/model-migrations) draft from `baseline_model`, `candidate_model`, and `experiment_suite_id`, with optional `shadow_sample_rate` (above 0 up to 1, default 1) and `shadow_daily_budget_usd`. `PATCH` takes exactly one change: a `status` (such as `shadowing` or `paused`), an `experiment_suite_id` (`null` unlinks), or a `shadow_sample_rate`. To cut over, `POST /v1/migrations/{migration_id}/complete` with up to 50 `policy_ids` from `GET /v1/migrations/{migration_id}/affected-policies` (an empty list changes no traffic). Revert with `PATCH` `{"status": "reverted"}`. > **Warning** > > Any organization-level Gateway key can complete a migration and change production routing, so protect automation keys like a deploy credential. ## Webhook alerts Add one webhook per organization under **Evals → Alerts**. Each new [alert](/merge-gateway/evals/running-evals#alerts) POSTs JSON with `kind` (`run_failed`, `run_error`, or `regression`), `title`, `suite_id`, `run_id`, `migration_id`, and `detail`. `X-Merge-Signature: sha256=` is an HMAC-SHA256 of the raw body under the signing secret, shown once when you create or rotate the webhook. **`Python`** ```python title="Python" import hashlib import hmac def verify(secret: str, body: bytes, signature_header: str) -> bool: expected = "sha256=" + hmac.new(secret.encode(), body, hashlib.sha256).hexdigest() return hmac.compare_digest(expected, signature_header) ``` Any `2xx` acknowledges; failures retry four times over about 75 minutes. ## Errors Eval errors return `{"detail": {"code": "...", "message": "..."}}`. Most migration errors return `detail` as a plain string, so branch on status. | Status | Code | When | | ------ | ------------------------------------------- | ------------------------------------------------------------------------ | | `400` | `unknown_model` | `target_model` isn't a catalog model | | `400` | `no_cases` | The suite has no enabled cases | | `400` | `duplicate_external_id`, `duplicate_output` | A sync or external run repeats a key | | `400` | `unknown_case` | An external output matches no enabled case | | `400` | Plain string | A completion names an unaffected policy, or the candidate isn't routable | | `403` | `customer_scoped_key_forbidden` | A customer API key was used | | `404` | | The ID doesn't exist in your organization | | `409` | `run_in_flight` | The suite already has 3 runs in progress | | `409` | `not_cancellable` | The run already finished | | `409` | Plain string | The baseline already has an active migration | | `409` | `baseline_has_override` | A completed, unreverted migration exists for the baseline | | `422` | | Schema validation, such as an unsupported `interval_hours` | | `429` | | More than 10 run triggers per minute | ## Next steps #### [Evals](/merge-gateway/evals/overview) Author the suites this API runs #### [Model migrations](/merge-gateway/evals/model-migrations) The migration flow this API automates #### [Evals API reference](/merge-gateway/evals-api) Full request and response schemas