Compare a change with production

A comparison answers whether a saved candidate improves the behavior you care about relative to production. Save your draft first, then start a comparison from that version.

Choose coverage and baseline behavior

Select Quick for the first 10 dataset cases, or Comprehensive on Pro for the entire dataset. Use Quick for an initial check and broader coverage before releasing a change whose effects extend beyond those cases.

Choose whether to reuse a compatible completed production evaluation or rerun both sides. Reuse avoids repeating eligible baseline work. Rerunning both sides gives fresh outputs from each version and incurs model usage for both. A change to the dataset or other comparison requirements can make an older run unsuitable for reuse.

The comparison pins production when it starts. If another member publishes a version while it is running, your comparison continues against its original production snapshot. Inspect the version numbers before using the result to make a release decision.

Read the differences

For the ticket-router change, inspect the invoice/sign-in ticket first, then the original billing, technical, and account cases. Compare actual outputs and per-case outcomes, not just aggregate pass rate. An improvement in one category can hide a regression in another.

Look for differences in latency and cost as well as quality where results provide them. A slower or more expensive version may be justified, but that tradeoff should be deliberate.

Screenshot placeholder — Comparison review Show candidate and production identifiers, run coverage, aggregate results, and a case that changed outcome.

Make the release decision

A completed comparison does not publish the candidate. If the result is suitable, assign a label. If a side errors, retry the failed side after addressing the cause. If the output is a legitimate failure, change the draft and save another candidate rather than treating a rerun as a fix.