Configure evaluators
Choose checks that match the output and control which cases they score.
An evaluator defines a rule for judging a generated output. Evaluators belong to the prompt and can apply to all cases or selected cases. A case passes only when all of its applicable evaluators pass.
Choose the check for the task
| Output requirement | Evaluator | Example |
|---|---|---|
| One known answer | Exact match | A ticket must return billing |
| Required or forbidden wording | Contains | A reply must include a support contact |
| A repeatable text pattern | Regex | An identifier must look like BUG-1234 |
| Valid structured data | JSON schema | A bug report must contain a severity enum |
| A size bound | Length | A summary must be at most 60 words |
| Quality on a defined scale | LLM score | A support answer must meet a groundedness rubric |
| Several qualitative requirements | LLM checklist | A reply must answer the question and follow policy |
Prefer a deterministic rule when the requirement is mechanical. Use a judge when the requirement needs interpretation. You can combine rules: a support reply may need both a maximum length and a policy-following checklist.
Set scope deliberately
Create and name the evaluator, configure its rule, and select the cases it should cover. A rule created from a row starts with that case as its scope; a prompt-wide rule can cover all cases. Inspect scope whenever you add cases or reuse an evaluator.
For initial setup, every case needs an applicable evaluator. A case without a reference cannot use exact matching meaningfully; add the expected value or choose a different rule.
Try a worked example
Use ticket routing for exact matching, bug extraction for a JSON schema, and support replies for score and checklist judges. The evaluator reference gives the exact matching behavior and configuration fields for all seven types.
Configured dashboard evaluators and the management interfaces' scoringMethod are different controls. The API, CLI, and MCP tutorials use expected with a reference for every routing case; do not substitute an evaluator type name into that field.