Evaluation guide
LLM-as-a-judge: a practical guide
Use a language model to score open-ended outputs with clear expectations, stable conditions, and reviewable reasons.
- Updated
- July 17, 2026
- Reading time
- 8 minutes
An LLM judge evaluates one model output against an expected answer or rubric. It is useful when exact string matching is too rigid but a human can still explain why an answer passes or fails.
The judge is not an oracle. It is another model call with its own variability and bias. A reliable setup narrows the question, fixes the judge configuration, requests a structured verdict, and keeps the reason visible for review.
Key takeaways
Use the cheapest reliable evaluator
Prefer exact checks for exact contracts and reserve a judge for semantic or qualitative decisions.
Ask for a verdict and a reason
A score without a concise explanation is difficult to debug, calibrate, or trust in a release review.
Calibrate against human decisions
Test the judge on examples your team already agrees about, including difficult borderline cases.
When should you use an LLM judge?
Use an LLM judge when several phrasings can satisfy the same expected behavior: whether an answer includes the required facts, responds helpfully to the supplied input, or follows a qualitative instruction. The judge can compare meaning without demanding word-for-word equality.
Do not use one just because it sounds sophisticated. Exact classification labels, valid JSON shape, required keywords, forbidden operations, and length bounds are usually cheaper and more predictable with deterministic checks.
- Good fit: groundedness against supplied facts, answer completeness, equivalent classifications, tone with concrete anchors.
- Poor fit: regex-valid formats, exact labels, numeric calculations, and rules ordinary code can enforce.
Define a narrow judge contract
Give the judge the original input, expected output, and actual output. Tell it what equivalence means, what format requirements are binding, and how to respond. A binary pass/fail verdict with one short reason is often more actionable than an unexplained score from zero to ten.
Hold the judge model, temperature, and instruction constant across the baseline and candidate. Otherwise the measurement changes at the same time as the thing being measured.
Return only JSON in this shape:
{
"passed": true,
"reason": "One short sentence explaining the verdict."
}
Judge the actual output against the expected output.
Equivalent wording may pass. Missing required substance or
violating an implied output format must fail.Calibrate the judge before using it as a gate
Create a calibration set with obvious passes, obvious failures, and borderline examples. Have at least one human reviewer label them independently. Compare the judge with those decisions and inspect disagreements rather than optimizing only for an average agreement percentage.
Look for systematic errors: verbosity bias, preference for the judge model's own style, failure to enforce format, or generosity toward partially correct answers. Tighten the expectation or replace the judge with a deterministic rule where possible.
Use judge results as evidence, not camouflage
In a regression report, show the expected output, actual output, verdict, and reason together. Review pass-to-fail flips first. If several failures share the same reason, that cluster is more useful than a single aggregate score because it points toward the prompt or model behavior that changed.
Keep a human review path for high-impact decisions and samples near the release threshold. The judge makes review scalable; it does not remove accountability for defining what good behavior means.
Frequently asked questions
Should the judge be a stronger model than the model being tested?
Often, but not automatically. Choose a judge that agrees reliably with your reviewed calibration set. A larger model is not a substitute for clear criteria.
What temperature should an LLM judge use?
Use a low, fixed temperature—commonly zero when the provider supports it—to reduce avoidable variability between runs.
Can the same judge compare two model versions?
Yes. Keep the judge configuration and test cases fixed, score both versions, and inspect where the candidate changes a previously passing case into a failure.