Support triage prompt regression: v16 vs v17
A small wording change removed an escalation rule. The candidate looked more specific, but its pass rate fell from 94% to 72% on the same 50 cases.
Baseline
94%
Prompt v16
Candidate
72%
Prompt v17
Pass → fail
11
of 50 cases
Decision
Block
Restore and rerun
The report
The aggregate drop and the evidence behind it
The result is useful because the conditions stayed fixed: GPT-5.4, temperature 0.2, the same 50-case dataset, and the same scoring contract. The prompt diff is the meaningful changed variable.
Support triage release check
Baseline v16 vs candidate v17 · 50 cases · same model, same dataset
Regression detected
11 cases passed the baseline but fail the candidate.
Prompt changes · v16 → v17
model gpt-5.4 · temperature 0.2 · unchanged
Cases that flipped · pass → fail
"The checkout page is down and we are losing orders every minute."
urgent ✓technical ✗
JudgeMissed the urgency signal on a production outage with revenue loss.
"I was charged twice and my card closes tomorrow."
urgent ✓billing ✗
JudgeDouble charge with deadline pressure should escalate as urgent.
"Can you send me the invoice for last month?"
billing ✓The message is about billing. ✗
JudgeReturned prose instead of the required single lowercase category.
Recommendation: Block the release. Restore the escalation rule in v17, then rerun the suite.
Methodology
What this study does—and does not—show
This is a worked, demo-safe benchmark of the release-review method. It is not a claim about general model quality or a production customer result.
Controlled comparison
Only the prompt version changes between the baseline and candidate.
Visible failure contract
Urgency handling and a single lowercase category are explicit pass conditions.
Reviewable release artifact
The prompt diff, failed cases, judge reasons, and decision remain together.
Not a universal model score
The pass rates do not generalize beyond this demonstration case set and prompt.
Representative pass-to-fail cases
These cases reveal two separate regressions: the candidate underweights explicit urgency signals and stops respecting the exact output format.
“The checkout page is down and we are losing orders every minute.”
The candidate ignored the revenue-impact escalation rule.
“I was charged twice and my card closes tomorrow.”
The narrower category displaced the explicit deadline escalation.
“Can you send me the invoice for last month?”
The candidate broke the required one-label output contract.
Run the same release check on your prompt
Keep the model and cases fixed, compare the candidate with a known-good version, and share every failure behind the decision.