Evals API

The Evals API runs suites and migrations from automation, such as a CI job that blocks a deploy on a failing suite. Schemas are in the Evals API and Migrations API reference. Authenticate with your organization-level mg_ key (the one you use for /v1/responses); customer API keys are rejected.

Gate CI on a suite

Start a run, poll until status leaves pending or running, then branch on passed (there’s no blocking mode).

BASE="https://api-gateway.merge.dev"
AUTH="Authorization: Bearer $MERGE_GATEWAY_API_KEY"
SUITE_ID="11a4c3f0-8b2e-4c1d-9f6a-2e7b5d3c9a01"
RUN_ID=$(curl -s -X POST "$BASE/v1/evals/suites/$SUITE_ID/runs" \
  -H "$AUTH" -H "Content-Type: application/json" \
  -d '{"target_model": "anthropic/claude-opus-5", "trials": 3, "idempotency_key": "ci-8412"}' | jq -r .id)
while true; do
  RUN=$(curl -s "$BASE/v1/evals/runs/$RUN_ID" -H "$AUTH")
  STATUS=$(jq -r .status <<<"$RUN")
  [ "$STATUS" = "pending" ] || [ "$STATUS" = "running" ] || break
  sleep 10
done
jq -r '"\(.status): \(.pass_count)/\(.pass_count + .fail_count + .error_count) passed"' <<<"$RUN"
[ "$(jq -r .passed <<<"$RUN")" = "true" ]   # nonzero exit fails the build

A run ends completed, failed (couldn’t execute), or cancelled; passed is null unless it completed. Completed runs also carry pass_rate, its 95% interval (ci_low, ci_high), total_cost_usd, and trace_id.

Trigger fields

Dataset as code

Keep cases in your repo so a prompt change ships with its cases. PUT /v1/evals/suites creates (201) or updates (200) a suite by name. PUT /v1/evals/suites/{suite_id}/cases syncs cases by external_id: it updates matches, creates new ones, and deletes synced cases missing from the payload unless you send "prune": false.

curl -s -X PUT "https://api-gateway.merge.dev/v1/evals/suites/$SUITE_ID/cases" \
  -H "Authorization: Bearer $MERGE_GATEWAY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "cases": [
      {
        "external_id": "refund-policy",
        "name": "Refund policy answer",
        "input_messages": [{"role": "user", "content": "What is your refund window?"}],
        "graders": [{"type": "contains", "value": "30 days"}]
      }
    ]
  }'
# {"created": 0, "updated": 1, "deleted": 0, "unchanged": 0}

Syncs never touch dashboard-authored cases (no external_id), and overwrite dashboard edits to a Synced case.

External runs

When your pipeline runs the model itself (a local checkpoint or your full agent loop), send "mode": "external" with one outputs entry per case, matched by external_id or case name and carrying output_text, output_json, or tool_calls.

{
  "target_model": "support-agent-v2",
  "mode": "external",
  "outputs": [
    {"external_id": "refund-policy", "output_text": "Refunds are accepted within 30 days."},
    {"name": "Greeting tone", "output_json": {"tone": "friendly"}}
  ]
}

target_model is a free-text label, trials are always 1, LLM judge graders still bill, and an enabled case with no output fails.

Schedules

GET, PUT, and DELETE on /v1/evals/suites/{suite_id}/schedule manage the schedule. PUT takes target_model, interval_hours (6, 12, 24, or 168), trials, and is_enabled, and restarts the clock.

Migrations

POST /v1/migrations creates a migration draft from baseline_model, candidate_model, and experiment_suite_id, with optional shadow_sample_rate (above 0 up to 1, default 1) and shadow_daily_budget_usd. PATCH takes exactly one change: a status (such as shadowing or paused), an experiment_suite_id (null unlinks), or a shadow_sample_rate.

To cut over, POST /v1/migrations/{migration_id}/complete with up to 50 policy_ids from GET /v1/migrations/{migration_id}/affected-policies (an empty list changes no traffic). Revert with PATCH {"status": "reverted"}.

Webhook alerts

Add one webhook per organization under Evals → Alerts. Each new alert POSTs JSON with kind (run_failed, run_error, or regression), title, suite_id, run_id, migration_id, and detail. X-Merge-Signature: sha256=<hex> is an HMAC-SHA256 of the raw body under the signing secret, shown once when you create or rotate the webhook.

import hashlib
import hmac
def verify(secret: str, body: bytes, signature_header: str) -> bool:
    expected = "sha256=" + hmac.new(secret.encode(), body, hashlib.sha256).hexdigest()
    return hmac.compare_digest(expected, signature_header)

Any 2xx acknowledges; failures retry four times over about 75 minutes.

Errors

Eval errors return {"detail": {"code": "...", "message": "..."}}. Most migration errors return detail as a plain string, so branch on status.

Next steps