Evals API

The Evals API runs your suites from your own automation: a CI job that gates a deploy on your evals, a cron that drives cadence from your infrastructure, or a script that walks a migration through its lifecycle. You can author suites and cases in the dashboard, or keep the dataset in your repo and sync it declaratively on every push; the two coexist on the same suite.

All requests go to https://api-gateway.merge.dev authenticated with your gateway API key, the same Authorization: Bearer key you use for /v1/responses. Add Content-Type: application/json on any request with a body. The organization is derived from the key.

The CI pattern

The contract is trigger-then-poll: start a run, poll until the status is terminal, then branch on passed. There is no blocking mode, so a long suite never holds an HTTP connection open.

BASE="https://api-gateway.merge.dev"
AUTH="Authorization: Bearer $GATEWAY_API_KEY"
SUITE_ID="11a4c3f0-8b2e-4c1d-9f6a-2e7b5d3c9a01"
RUN_ID=$(curl -s -X POST "$BASE/v1/evals/suites/$SUITE_ID/runs" \
  -H "$AUTH" -H "Content-Type: application/json" \
  -d '{"target_model": "anthropic/claude-opus-5", "trials": 3}' | jq -r .id)
while true; do
  RUN=$(curl -s "$BASE/v1/evals/runs/$RUN_ID" -H "$AUTH")
  STATUS=$(jq -r .status <<<"$RUN")
  [ "$STATUS" = "pending" ] || [ "$STATUS" = "running" ] || break
  sleep 10
done
jq -r '"\(.status): \(.pass_count)/\(.pass_count + .fail_count + .error_count) passed"' <<<"$RUN"
[ "$(jq -r .passed <<<"$RUN")" = "true" ]   # nonzero exit fails the build

The trigger returns 201 with the run, or 409 with code run_in_flight once the suite already has 3 runs in progress, so parallel CI branches don’t serialize but a runaway loop can’t stack unbounded work. trials (1 to 5) executes each case that many times; a case passes only when every trial passes. Run triggers are also rate limited per organization.

Three optional trigger fields exist for pipelines:

• idempotency_key: a retried trigger with a key the suite has already seen returns the original run with 200 instead of starting a duplicate. Use your CI job id.
• metadata: up to 16 short key-value strings (git SHA, branch, pipeline URL) echoed back on every read of the run and shown in the dashboard.
• request_overrides: run-level request configuration applied to every case. {"system": "..."} prepends a system prompt to each case, and {"params": {"temperature": 0}} overrides per-case request params. This is how you eval a new prompt version against the same dataset without editing any case. Overrides are frozen into the run’s config snapshot.

{
  "target_model": "anthropic/claude-opus-5",
  "trials": 3,
  "idempotency_key": "ci-8412",
  "metadata": {"git_sha": "3fc9a17", "branch": "main"},
  "request_overrides": {"system": "You are the support agent, prompt v14."}
}

Dataset as code

Keeping the eval dataset in your repo means a prompt change and its test cases review and ship in the same PR. Two idempotent endpoints make the API the source of truth:

PUT /v1/evals/suites creates or updates a suite by name: 201 when it created one, 200 when it updated the existing live suite with that name. Only the fields you send change.

PUT /v1/evals/suites/{suite_id}/cases syncs the suite’s cases from your payload. Each case carries an external_id (its stable identity in your repo); matched cases are updated in place, new ones created, and synced cases missing from the payload deleted. The response reads like a plan apply:

curl -s -X PUT "$BASE/v1/evals/suites/$SUITE_ID/cases" \
  -H "$AUTH" -H "Content-Type: application/json" \
  -d '{
    "cases": [
      {
        "external_id": "refund-policy",
        "name": "Refund policy answer",
        "input_messages": [{"role": "user", "content": "What is your refund window?"}],
        "graders": [{"type": "contains", "value": "30 days"}]
      }
    ]
  }'
# {"created": 0, "updated": 1, "deleted": 0, "unchanged": 0}

Cases authored in the dashboard have no external_id and are never touched by a sync, so hand-written cases and repo-synced cases coexist. Pass "prune": false to add and update without deleting. In the dashboard, synced cases carry a Synced chip and editing one warns that the next sync overwrites manual changes. Runs snapshot their cases at trigger time, so a sync never disturbs a run already in flight.

External runs (offline evals)

When your pipeline runs the model itself (a local checkpoint, a candidate that is not on Gateway yet, or your full agent loop), submit the outputs and let Gateway grade them. Trigger with mode: "external" and one output per case:

{
  "target_model": "my-agent-v2",
  "mode": "external",
  "outputs": [
    {"external_id": "refund-policy", "output_text": "Refunds are accepted within 30 days."},
    {"name": "Greeting tone", "output_json": {"tone": "friendly"}}
  ]
}

Outputs match cases by external_id, falling back to the case name; each output carries output_text, output_json, or tool_calls. In external mode target_model is a free-text label for the system under test, so it doesn’t need to be a catalog model. Grading runs the same pipeline as any other run (deterministic graders run locally; LLM judge graders still execute through Gateway and bill normally), and the run’s verdict, statistics, comparison, and alerts behave identically. An enabled case you submit no output for counts as failed. External runs never feed a migration’s eval gate, whose promise is that the candidate ran through your real Gateway path.

Endpoints

Suites and runs

A completed run carries everything a pipeline needs to report on:

{
  "id": "3f6f0d1a-...",
  "status": "completed",
  "passed": false,
  "pass_count": 18,
  "fail_count": 2,
  "error_count": 0,
  "pass_rate": 0.9,
  "ci_low": 0.699,
  "ci_high": 0.972,
  "trials": 3,
  "target_model": "anthropic/claude-opus-5",
  "total_cost_usd": 0.42,
  "trace_id": "exp-9b8c7d6e5f4a3210",
  "completed_at": "2026-09-03T21:14:09Z"
}

pass_rate comes with its 95% confidence interval (ci_low, ci_high), and trace_id links the run to its trace in the dashboard. passed is null until the run completes.

Schedules

Schedules configure the same recurring runs as the suite’s Configuration tab. interval_hours accepts 6, 12, 24, or 168.

Every PUT restarts the clock: the next run lands one interval from the save.

Migrations

The migration lifecycle is fully drivable over the API. State transitions go through PATCH with exactly one action per request: a status change, an eval suite link, or a shadow_sample_rate change.

Completion rewrites the policies in policy_ids, which must come from the affected-policies response; an empty list completes without touching any traffic. Reverting is PATCH to {"status": "reverted"}, which restores the rewritten policies and skips any edited since the cutover.

Webhook alerts

Failing runs should reach you without anyone watching a dashboard. Configure a webhook endpoint under Evals → Alerts and every new alert (failed run, errored run, or regression) is POSTed to it as JSON:

{
  "id": "9c1e...",
  "kind": "run_failed",
  "title": "Evals failed: golden-set on anthropic/claude-opus-5 (18/20 passed)",
  "status": "open",
  "created_at": "2026-09-08T09:30:00Z",
  "suite_id": "11a4...",
  "run_id": "3f6f...",
  "migration_id": null,
  "detail": {"target_model": "anthropic/claude-opus-5", "pass_count": 18, "fail_count": 2}
}

Each request carries X-Merge-Event: experiment_alert and X-Merge-Signature: sha256=<hex>, an HMAC-SHA256 of the raw body under your signing secret. The secret is shown once when you create or rotate the webhook; verify before trusting:

import hashlib
import hmac
def verify(secret: str, body: bytes, signature_header: str) -> bool:
    expected = "sha256=" + hmac.new(secret.encode(), body, hashlib.sha256).hexdigest()
    return hmac.compare_digest(expected, signature_header)

Respond with any 2xx to acknowledge. Failed deliveries retry with exponential backoff for about an hour before giving up; the alert always remains visible in the dashboard regardless of delivery.

Errors

Validation failures return 400 with a structured body, {"code": "...", "message": "..."}: for example unknown_model for a target outside the catalog, no_cases for an empty suite, duplicate_external_id for a sync payload that repeats a key, unknown_case for an external output that matches no enabled case, or unknown_policy for a completion referencing a policy outside the affected set. Conflicts return 409 (run_in_flight, not_cancellable, duplicate_active_migration). IDs that do not exist in your organization return 404.

Next steps