> This page is for Gateway.

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.merge.dev/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.merge.dev/_mcp/server.

# Runs and monitoring

> Eval runs: trials for flaky cases, confidence intervals, paired comparison, alerts, and schedules.

A run grades every enabled case in a suite against one model. Choose **Run evals** on the suite page, review the estimated cases, judge calls, and spend, and confirm. Runs continue if you leave; a suite can have 3 in progress.

## Trials

Each case runs 1, 3, or 5 times and passes only if every trial passes. Use 1 while iterating and 3 or 5 for real decisions; cost scales with trials.

## Reading a verdict

Pass rates show a 95% confidence interval: 8/10 reads "80%, CI 49-94%", which is weak evidence. Runs with fewer than 20 graded cases are flagged **Low evidence**, with a hint of how many cases would tighten the interval.

A case result shows the full response, each grader's outcome (with the judge's score and rationale), latency, and cost. Each run is a [trace](/merge-gateway/observability/tracing) with one span per case.

## Comparing two runs

Run the suite against both models, then choose **Compare** on either completed run. Gateway pairs results by case, counts cases fixed and broken, tests significance, and lists regressions first.

## Alerts

Alerts appear at **Evals → Alerts**, with an open count in the Evals header. Acknowledging records who and when. Cancelled runs never alert.

| Alert            | Raised when                                                                                                                                               |
| ---------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Evals failed** | A run completed without meeting the pass criteria                                                                                                         |
| **Run error**    | A run couldn't execute, for example because every case errored                                                                                            |
| **Regression**   | A scheduled run passed, but more cases regressed than improved against the previous completed run of the same suite and model. Errored cases don't count. |

Route alerts to Slack or a pager with a [webhook](/merge-gateway/evals/evals-api#webhook-alerts) on the alerts page.

## Scheduled runs

On the **Configuration** tab (or [over the API](/merge-gateway/evals/evals-api#schedules)), schedule a suite against one model every 6 hours, 12 hours, day, or week. Saving restarts the clock, so a re-enabled schedule never fires a backlog; a run due while another is in progress is skipped.

## Next steps

#### [Evals API](/merge-gateway/evals/evals-api)

Trigger runs from CI and fail the build on a red suite

#### [Model migrations](/merge-gateway/evals/model-migrations)

Put a suite's verdict in front of every cutover

#### [Tracing](/merge-gateway/observability/tracing)

Inspect any eval case's request as a span