Understand results and coverage
Distinguish failed checks, execution errors, and incomplete review.
An evaluation records the version, inputs, outputs, and scoring decisions for a run. Read its status and case outcomes together: “completed” means the run finished, not that every output passed.
Interpret a case
| Outcome | Meaning | What to do |
|---|---|---|
| Passed | The applicable checks accepted the output | Inspect representative passes to confirm the rule is useful |
| Failed | The output was produced and a check rejected it | Read the output, reference or criteria, and reason |
| Unscored | A score has not been assigned | Complete review or inspect whether the scoring mode has enough information |
| Errored | Generation or scoring could not complete | Resolve the execution error before treating it as a quality result |
With configured evaluators, all applicable checks must pass for a case to pass. A judge's explanation helps you understand its decision, but you should still read the output yourself when the result is consequential or surprising.
Read aggregate scores carefully
Inspect total cases, scored cases, pending reviews, passed and failed cases, and pass rate together. A pass rate from partially scored work is not equivalent to a fully reviewed run. Execution errors also leave a coverage gap; they are not evidence that the prompt answered incorrectly.
Treat the denominator and case selection as part of the result. Comparing a Quick run on 10 cases with a Comprehensive run on 100 different cases does not isolate a prompt change.
Understand coverage
Quick uses the first 10 cases in dataset order, or all cases when there are fewer than 10. Comprehensive uses the entire dataset and requires Pro. Reorder the dataset so the Quick subset represents the most important behaviors.
Runs freeze their inputs. Editing the dataset later does not update an old run's coverage. Before publishing, check whether the evaluated dataset matches today's dataset and whether the scope covers the risks you care about. Publication warnings make those gaps visible but do not make the release decision for you.
Continue with human review or sharing after interpreting the results.