Runs and monitoring
A run executes every enabled case in a suite against one model you pick and returns a verdict against the suite’s pass criteria. This page covers what happens during a run and the machinery around it: repeated trials, the statistics on the results, comparing two runs, alerts, and schedules.
Start a run from the suite with Run evals. Before anything executes, Gateway shows a cost estimate (case count, judge calls, estimated spend) so a large suite never surprises you. Runs continue server-side; you can leave the page and come back.
Trials
LLMs are nondeterministic, and a case that passes once is not the same as a case that passes reliably. A run can execute each case 1, 3, or 5 times, and a case passes only when every trial passes. A case that flakes across trials is a failure, not a coin worth flipping in production.
One trial is the fast default for iterating on a suite. Use 3 or 5 when the verdict feeds a real decision, such as a migration cutover. Trials multiply cost proportionally, and the estimate reflects that before you confirm.
Reading a verdict
A pass rate over a sample of cases carries sampling uncertainty, so Gateway always shows the 95% confidence interval next to the rate. A 10-case suite that scores 8/10 reads as “80%, CI 49-94%”: consistent with anything from a coin flip to near-perfect. The same 80% on 100 cases is a much stronger claim.
Small suites are additionally flagged Low evidence, with a hint of how many cases would resolve the verdict to a useful precision. The flag never blocks anything; it exists so a lucky green on a handful of cases is not mistaken for certainty.
Each case result opens into a detail view with the model’s full response, every grader’s outcome (including the judge’s score and rationale), latency, and cost. Runs are also traces: one trace per run, one span per case, so you can inspect any case’s request in Logs.
Comparing two runs
To compare two models, run the same suite against each and use Compare on either completed run. The comparison pairs each case’s outcome across the two runs, because only the cases that changed carry information about which model is better: a case both models pass says nothing.
The verdict line reports how many cases the second run fixed and broke, with a statistical significance test on the changed cases and a plain-language reading: a difference that could be sampling noise says so explicitly. The case-by-case table defaults to changed cases only, regressions first.
Alerts
A run that needs attention raises an alert in the dashboard. Alerts live at Evals → Alerts, newest first, with an open-count badge on the Evals page header. Each alert links to its run, and acknowledging it records who and when.
Three kinds exist:
- Evals failed: the run completed and the suite’s pass criteria were not met
- Run error: the run could not execute (for example, every case errored)
- Regression: a scheduled run passed, but cases the previous run of the same suite and model passed now fail. A passing verdict can hide slippage when the pass criteria allow headroom; this alert names the newly failing cases so a slow slide never goes quiet.
Cancelling a run yourself never raises an alert.
To get alerts out of the dashboard, add a webhook on the alerts page: every new alert is POSTed to your endpoint with an HMAC signature you can verify, which is how a failed nightly run reaches your Slack or pager. The wire contract and verification snippet are in the Evals API.
Scheduled runs
A schedule runs a suite on a fixed cadence against one model: every 6 hours, every 12 hours, daily, or weekly. Configure it on the suite’s Configuration tab with a target model and trials per case. A failing scheduled run raises an alert, and a passing one that loses cases the previous run passed raises a regression alert, so a schedule plus alerts (or a webhook) is regression monitoring end to end: the suite you trust runs nightly, and you hear about it only when something breaks or slips.
Saving a schedule restarts its clock, so re-enabling a long-paused schedule never fires a backlog. If a run is already in flight when the schedule comes due, that tick is skipped rather than stacked.
Schedules are also manageable over the API, which suits driving cadence from your own infrastructure instead; see the Evals API.