Evals API

Run eval suites from CI: sync cases from your repo, trigger runs, poll verdicts

The Evals API runs your eval suites from CI and automation. Sync test cases from your repo, trigger a run against any catalog model (or submit outputs your own pipeline produced and let Gateway grade them), poll until the verdict is terminal, and fail the build on a red suite. Concepts, graders, and the full CI walkthrough live in the Evals guide.

  • Suites and cases: create or update a suite by name, and declaratively sync its cases by external_id; cases authored in the dashboard coexist untouched
  • Runs: single-call trigger with idempotency keys, CI metadata, request overrides, and an external mode for outputs you produced yourself
  • Schedules: recurring runs on a fixed cadence, the other half of regression monitoring with alerts

Base URL

https://api-gateway.merge.dev

Authentication

Authenticate with your gateway API key as a bearer token, the same mg_ key you use for /v1/responses. The organization is derived from the key.

Authorization: Bearer mg_<your_api_key>

The CI contract

Triggering a run returns 201 immediately; execution is asynchronous. Poll GET /v1/evals/runs/{run_id} until status is terminal, then branch on passed. A trigger retried with the same idempotency_key returns the original run with 200. See running evals from CI for complete examples.