Support triage regression report
A release check comparing known-good prompt v16 with candidate v17 across the same model, settings, and dataset.
Support triage release check
Baseline v16 vs candidate v17 · 50 cases · same model, same dataset
Regression detected
11 cases passed the baseline but fail the candidate.
Prompt changes · v16 → v17
model gpt-5.4 · temperature 0.2 · unchanged
Cases that flipped · pass → fail
"The checkout page is down and we are losing orders every minute."
urgent ✓technical ✗
JudgeMissed the urgency signal on a production outage with revenue loss.
"I was charged twice and my card closes tomorrow."
urgent ✓billing ✗
JudgeDouble charge with deadline pressure should escalate as urgent.
"Can you send me the invoice for last month?"
billing ✓The message is about billing. ✗
JudgeReturned prose instead of the required single lowercase category.
Recommendation: Block the release. Restore the escalation rule in v17, then rerun the suite.
Likely causes
What changed before quality dropped
Prompt v17 removed the escalation rule
The pass-to-fail cases all lost urgent handling after the explicit outage and revenue-loss rule was removed.
The candidate favors narrow categories over urgency
The new specificity instruction produces technical and billing labels where the baseline escalated time-sensitive cases.
Model, settings, and dataset changes
Both versions use GPT-5.4 at temperature 0.2 against the same 50-case dataset.
Release decision: block
Restore the escalation rule in prompt v17, then rerun the release suite before deploying.