Evals
Evals let you encode what “working” means for your application as a suite of test cases, then grade any model against that suite. Use them when you need evidence for a model decision: adopting a new model, migrating off an old one, or catching a regression before your customers do.
Runs execute through your real Gateway setup, not a sandbox. Each case is sent through the same path as production traffic, so your credentials, routing restrictions, and guardrails all apply, and the run appears in your logs and spend like any other traffic. What you validate is what you ship.
Evals live in the dashboard under Evals. A suite groups your test cases and defines what passing means; a run executes every enabled case against one model and returns a verdict.
Suites and test cases
A suite is your source of truth for one behavior you care about: a support agent’s tone, an extraction pipeline’s JSON contract, a router’s tool selection. Each test case in the suite carries:
- Input messages: a single prompt or a multi-turn conversation (system, user, and assistant turns)
- Graders: one or more assertions that score the model’s response
- Required flag: required cases must pass for the suite to pass, regardless of the suite’s pass criteria
Add cases one at a time in the dashboard, import them in bulk as JSONL or CSV, or save a real production request as a case directly from its log entry (open a request in Logs and choose Save as test case). Saving from logs is the fastest way to turn an interesting production interaction, good or bad, into a permanent regression test.
Teams that keep their eval dataset in the repo can also sync it over the API: a prompt change and its test cases then review and ship in the same PR, and dashboard-authored cases coexist untouched alongside the synced ones.
Graders
Each case is scored by its graders. Deterministic graders are exact and free; the LLM judge handles open-ended quality at the cost of one extra model call per case.
A case can combine graders and requires either all of them or any of them to pass. Grader errors count against the case, never for it, so infrastructure noise cannot inflate a pass rate.
For structured-output applications, prefer the deterministic graders: they are exact, instant, and add no cost. Reserve the judge for questions determinism cannot answer, such as tone, helpfulness, or factual quality.
The LLM judge
The judge is a model call that scores a response from 0 to 1 against your rubric, with a pass threshold you set. Configure suite-level defaults (judge model, rubric, threshold) on the suite’s Configuration tab; individual cases inherit them unless they override. Gateway ships a sensible default rubric and judge model so a new suite works before you have tuned anything.
Judge calls run through your organization like any other traffic: they appear in logs, bill to you, and are itemized in each run’s cost estimate before you confirm.
Pass criteria
The suite’s verdict policy is customer-defined:
- All required cases pass (the default): strict, best for contract-style suites
- Minimum pass rate: a percentage of cases must pass, while required cases must still all pass
The verdict is always advisory. Evals inform decisions like completing a migration; they never block one.