> This page is for Gateway.

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.merge.dev/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.merge.dev/_mcp/server.

# Evals API

> Reference for the Evals API: suite and case sync, gateway and external runs, per-case results, and recurring schedules.

The Evals API runs eval suites from CI: sync cases from your repo, run them against a catalog model or grade your pipeline's own outputs, and fail the build on a red suite. See the [Evals guide](/merge-gateway/evals/overview) for concepts and graders, and [running evals from CI](/merge-gateway/evals/evals-api) for a walkthrough.

## Base URL and authentication

```text
https://api-gateway.merge.dev
```

Authenticate with the `mg_` API key you send `/v1/responses` with. The organization comes from the key; customer API keys are rejected.

```http
Authorization: Bearer mg_...
```

## Reference

| Endpoints                                                           | What they do                                                                                                                                                                                                        |
| ------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `GET`, `PUT /v1/evals/suites`, `GET /v1/evals/suites/{suite_id}`    | List suites, or create or update one by name (`201` created, `200` updated)                                                                                                                                         |
| `PUT /v1/evals/suites/{suite_id}/cases`                             | Sync cases by `external_id`. With `prune` (default `true`), synced cases missing from the payload are deleted. Dashboard-authored cases are never touched. Returns created, updated, deleted, and unchanged counts. |
| `GET`, `POST /v1/evals/suites/{suite_id}/runs`                      | List or trigger runs                                                                                                                                                                                                |
| `GET /v1/evals/runs/{run_id}`, `GET .../results`, `POST .../cancel` | Poll a run, read per-case results, or cancel a pending or running run                                                                                                                                               |
| `GET`, `PUT`, `DELETE /v1/evals/suites/{suite_id}/schedule`         | A recurring run every 6, 12, 24, or 168 hours. Pair it with [alerts](/merge-gateway/evals/running-evals#alerts).                                                                                                    |

A trigger takes `target_model`, `mode` (`gateway`, or `external` with your own `outputs`), `trials` (1 to 5, and a case passes only if every trial passes), `metadata` (up to 16 string pairs), `idempotency_key`, and `request_overrides` for the system prompt and params.

**The CI contract.** A trigger returns `201` immediately and runs asynchronously. Poll `GET /v1/evals/runs/{run_id}` until `status` is `completed`, `failed`, or `cancelled`, then branch on `passed`. Retrying a trigger with the same `idempotency_key` returns the original run with `200`. Each organization can trigger 10 runs per minute (`429` beyond that), and each suite can have 3 runs in progress (`409 run_in_flight`). See [Errors](/merge-gateway/errors#management-and-platform-apis) for other codes.

## Next steps

#### [Run evaluations](/merge-gateway/evals/running-evals)

Compare model responses against your cases

#### [Model migrations](/merge-gateway/evals/model-migrations)

Measure a proposed model change