Evaluator types and matching rules
Reference the seven configured evaluator types and their passing rules.
These types describe prompt-scoped dashboard evaluators. The management API's scoringMethod accepts expected, human_review, or llm_judge; it does not accept these configuration objects as scoring-method values.
Exact match
Type exact compares the actual output with the case's reference as text. trimWhitespace trims leading and trailing whitespace; ignoreCase controls case sensitivity. Neither option removes punctuation, rewrites prose, or makes JSON comparison structural.
For ticket routing, enable trimming and keep case sensitivity: billing\n passes against billing, while Billing and billing. fail. See the worked lesson.
Contains
Type contains takes texts, match (all or any), ignoreCase, and forbidden. In normal mode, the selected matching rule must find the configured text. Forbidden mode checks that each configured text is absent, then combines those absence checks with All or Any.
To require a support address, use texts: ["support@example.com"] with forbidden: false. To reject any mention of either forbidden phrase, use match: "all" and forbidden: true: every phrase must be absent. With match: "any" in forbidden mode, the output passes when at least one configured phrase is absent.
Regex
Type regex takes pattern, flags, and shouldMatch. Use ^BUG-[0-9]{4}$ to require a complete identifier such as BUG-1234. Without anchors, a match can occur inside longer text. Set shouldMatch to false to reject matching outputs.
Patterns have a 100 ms execution budget. Prefer simple bounded patterns and inspect evaluator errors rather than interpreting a timeout as a failed content check.
JSON schema
Type json_schema takes schema, a JSON-encoded Draft 7 schema string. The actual output must parse as JSON and validate against the schema. Markdown fences are not JSON. Structural validity does not establish factual correctness. Use the bug extraction example.
Length
Type length takes optional inclusive min and max bounds and a unit of characters or words. Characters count Unicode code points; words are split on whitespace. These are not token limits.
A release-note summary can use max: 60 with unit: "words". The rule constrains size but does not decide whether the summary contains the right facts.
LLM score
Type llm_score takes criterion, levels with a score and description for each level, and a passing threshold. A valid selected level at or above the threshold passes. Define levels with observable differences so the judge can choose consistently.
LLM checklist
Type llm_checklist takes items and match (all or any). All requires every item to pass; Any requires at least one. Use All for independent mandatory policy requirements.
Both judge types explicitly configure modelRouteId, temperature, and maxTokens. Provide provider access or credits for the judge route as well as the generation route. Judge execution errors differ from negative judgments. See the complete support-reply checklist and rubric.