Guardrail tester
The tester runs one sample input through the Security Gateway exactly as a real tool call would go through it, and streams back what happened at each stage. Use it whenever you add a rule or a guardrail, and whenever you change one: most rule mistakes look correct in the editor and only show up on live traffic. Change a rule anywhere in the Security section and a bar appears offering to try it here, because your edits are live the moment you save them.
Open Security → Guardrail tester. It covers both halves of the Gateway, the entity and regex rules and the AI Guardrails, and it uses your organization’s live configuration: every active guardrail is included automatically, so there is nothing to select. The tab appears once guardrails are enabled for your organization.
How to use it
- Paste a sample of the content you want to test, or pick one of the built-in sample prompts. A real tool argument or tool result copied out of Tool call logs is the best source you have.
- Click Test guardrails.
- Watch the pipeline and read the feed.
The pipeline above the input shows the path a call takes: your organization, the pre-run lane holding the entity scan and every pre-run guardrail, the tool call itself, the post-run lane, and the response. Each stage lights up as it runs and carries how long it took, so a lane that stays dark tells you as much as one that fires. A blocked pre-run verdict halts the connector into the tool call, which is the visual answer to “would this call have reached the third party.”
Reading the feed
The feed prints one timestamped line per event, in the order the Gateway produced them, each with the action it would take:
- The entity scan. Either “No sensitive data detected” or “Sensitive data detected: N entities”, followed by one line per entity type or custom rule that matched, badged Redact, Log, or Block.
- Pre-run guardrail verdicts, one per guardrail, naming the guardrail and the content the evaluator pointed at, badged Pass, Log, or Block.
- The tool call, either “Tool call ran” or “Tool call skipped” when something upstream blocked it.
- Post-run guardrail verdicts, in the same shape, or “Post-run screening skipped” if the call never ran.
- The outcome, “Response delivered” or the guardrail that blocked it.
A verdict badged Evaluation failed means the evaluator returned nothing usable for that guardrail rather than clearing it. Rerun the sample before reading anything into that row.
What to test
Run every new rule through three cases.
Positive cases
The content the rule exists to catch. Confirm it matches and that the badge shows the action you intended.
Negative cases
Content that looks similar but should not match. Confirm nothing fires.
Edge cases
The fuzzy boundaries: matches sitting near your context keywords and matches sitting far from them, different casings, whitespace variants.
If a rule fires on your negative case, or misses your positive one, the fix is in the rule rather than in the sample: a word boundary, a missing alternative, or context keywords that are too generic. See Custom Regex Rules for how the keywords shift a match’s confidence.
Testing a guardrail instruction
A guardrail is where the tester earns the most time, because an instruction is prose and prose is easy to over-read. Write one sample that clearly violates the instruction and one that touches the same subject innocently, then run both. An instruction that blocks the innocent sample is too broad, and one that passes the violating sample names something the evaluator cannot see in the text. Both are a second draft rather than a model-class change.
After you deploy
The tester is pre-flight. Violations and alerts is the flight: after a rule goes live, watch the breakdown for a few hours, and if the volume looks wrong come back here with a real sample pulled out of the violations you got.
Next
Watch what your rules catch on real traffic in Violations and alerts.