Free template
LLM-as-a-judge rubric template
A strict, reviewable template for comparing actual model output with expected behavior and returning a structured verdict.
- Updated
- July 17, 2026
- Reading time
- 6 minutes
This template is designed for regression testing: one input, one expected output, one actual output, and a binary verdict with a short reason. It avoids vague numeric scoring and keeps the evidence visible to the reviewer.
Adapt the rules to the specific workflow, then calibrate the judge against human-reviewed examples before using it to block releases.
Key takeaways
Judge against an explicit expectation
Do not ask whether an answer is generally good; define what this case requires.
Make formatting requirements binding
A downstream contract can fail even when the prose contains the right idea.
Return one concise reason
The reason should identify the missing, incorrect, or malformed behavior a reviewer can act on.
Judge system instruction
Use this as the stable judge instruction. Keep it unchanged when comparing a baseline with a candidate so the scoring conditions remain comparable.
You are an exacting evaluation judge for AI outputs.
You will receive a test input, an expected output, and the
actual output produced by a model. Decide whether the actual
output satisfies the expected output.
Rules:
- Equivalent wording may pass; word-for-word matching is not required.
- Missing or incorrect required substance must fail.
- Follow any format implied by the expected output.
- Do not reward extra content that was not requested.
- Be strict, consistent, and concise.
Return only JSON:
{"passed": true, "reason": "One short sentence."}Per-case message template
Send the same fields for every evaluated output. Use clear delimiters so content in the test case cannot be confused with judge instructions.
Test input:
{{input}}
Expected output:
{{expected_output}}
Actual output:
{{actual_output}}Optional workflow-specific rules
Add only rules that define a real pass condition. Overloading the rubric with broad qualities such as clarity, usefulness, correctness, tone, safety, and completeness makes disagreement difficult to diagnose.
- Classification: prose around a required single label is a format failure.
- RAG: claims outside the supplied context fail unless uncertainty is explicitly allowed.
- Support: required escalation or next-step language must be present.
- Extraction: missing required keys or invalid JSON fails.
- Content: blocked claims fail even when the rest of the answer is strong.
Calibration worksheet
Prepare at least ten human-reviewed outputs: clear passes, clear failures, and borderline cases. Record the human verdict, judge verdict, and disagreement reason. Repair the expectation or rule before blaming the model being evaluated.
{
"caseId": "support-outage-001",
"humanVerdict": false,
"judgeVerdict": true,
"disagreement": "Judge ignored the required one-label format.",
"action": "Make the format rule explicit and rerun calibration."
}