# Catch LLM regressions before they reach production A practical guide to testing prompt, model, and configuration changes against a known-good baseline before release. An LLM regression happens when a change that looks reasonable overall makes a known workflow worse. A new prompt may improve average tone while dropping an escalation rule. A cheaper model may pass most examples but fail the few cases that create real customer risk. Regression testing makes that change reviewable. You run the same test cases against a known-good baseline and the candidate, score both with the same rules, and inspect the cases that move from pass to fail. The result is a release decision backed by examples rather than a general impression. ## Key points - **Compare changes, not isolated outputs:** A candidate only has meaning relative to the behavior your users already rely on. - **Build the suite from real failure modes:** The best cases cover expensive mistakes, edge conditions, and format contracts—not generic benchmark questions. - **Inspect pass-to-fail flips:** Aggregate scores show direction; flipped cases explain the release risk and what needs to change. ## A reproducible ten-case support triage example This worked example uses authored synthetic outputs to demonstrate the review calculation. It is not a measured benchmark of any model. Download the cases, output fixture, and Node script into one directory; no API key or dependency installation is needed. ### Run the illustrative calculation (Node.js) ``` node score-triage.mjs ``` - [Ten cases JSON](https://www.promptlens.io/examples/support-triage.json) - [Authored output fixture](https://www.promptlens.io/examples/triage-output-fixture.json) - [Scoring script](https://www.promptlens.io/examples/score-triage.mjs) ## Run the same workflow with real model outputs Create a support triage prompt in User message input mode. Use the system instruction below, import support-triage.json, and add an Exact match evaluator using the reference output with whitespace trimming enabled. Save the prompt version and select a model route you can access through a connected workspace key. Run an initial evaluation and inspect all ten outputs before assigning production. For a prompt-only experiment, save a candidate that removes the billing-priority instruction while keeping the model and settings fixed. Start a comparison against the pinned production baseline. Record the exact versions, route, settings, and evaluator configuration; execution errors must be retried or resolved before comparing scores. ### Baseline system instruction ``` Classify the support message. Return exactly one lowercase label: billing, technical, or account. Return no explanation. Payment, invoice, charge, and card issues are billing, even when account access is also mentioned. Software failures are technical, including login server errors. Identity, password, and account closure requests without a software failure are account. ``` ## Keep the complete prompt change reviewable The candidate below removes the rule that billing wins when account access is also mentioned. These are experiment instructions, not a claim that a particular model will return the authored fixture outputs. Run your selected route to discover its behavior. ### Candidate system instruction ``` Classify the support message. Return exactly one lowercase label: billing, technical, or account. Return no explanation. Payment, invoice, charge, and card issues are billing. Software failures are technical, including login server errors. Identity, password, and account closure requests without a software failure are account. ``` - [Baseline prompt text](https://www.promptlens.io/examples/triage-baseline-prompt.txt) - [Candidate prompt text](https://www.promptlens.io/examples/triage-candidate-prompt.txt) ## Explain the failure before deciding The authored fixture scores baseline 10/10 and candidate 8/10 under case-sensitive exact match with whitespace trimming. Cases 2 and 7 change from billing to account. The pass-to-fail count is two; the pass-rate delta is −20 percentage points. These numbers describe only the downloadable fixture. Inspect “I was charged twice and cannot access my account.” and “My card failed and I cannot sign in to pay.” Both fail the defined payment rule, so the fixture decision is block and restore billing priority. A real model run may produce different outputs and must be reviewed on its own evidence. After revising the candidate, rerun the fixed suite and preserve the results in a report. - [Inspect the illustrative report UI](https://www.promptlens.io/r/demo-support-triage) - [Migration example](https://www.promptlens.io/use-cases/model-migration-testing) ## What counts as an LLM regression? A regression is a behavior that met your release bar in the baseline and no longer meets it in the candidate. The candidate can be a prompt edit, a model upgrade, a provider-route change, a temperature change, or any other configuration that affects output. The definition should stay local to the workflow. A customer-support classifier regresses when it misses an urgent ticket or breaks the required output format. A RAG answer regresses when it omits a required fact or makes a claim outside the supplied context. One universal quality score cannot describe both jobs well. - Prompt regressions: instructions are removed, weakened, contradicted, or made ambiguous. - Model regressions: a candidate model handles a subset of your cases less reliably. - Configuration regressions: temperature, system messages, or routing changes alter behavior. - Format regressions: the answer is semantically right but violates a downstream contract. ## Build a regression suite around observable failures Start with 20 to 50 cases that represent the decisions your application must get right. Each row needs an input and a reviewable expectation. For deterministic tasks, that expectation may be an exact category, required keywords, a length range, or a regular expression. For open-ended answers, write an expected output that captures the required substance and format. Avoid a dataset made only of happy paths. Include ambiguous requests, missing context, adversarial wording, high-risk edge cases, and examples that previously failed in production. A small living suite connected to real failures is more useful than a large static benchmark that does not resemble your application. - [Use the dataset template](https://www.promptlens.io/templates/llm-evaluation-dataset) ## Keep the baseline and candidate comparable A valid regression comparison holds the case set and scoring method constant. If you change the dataset, judge, and model at the same time, a score difference does not tell you which change caused it. Freeze the evaluation conditions, change one meaningful variable, then rerun. Store the exact prompt version, model route, settings, case-set identifier, and scoring configuration with every run. That record turns a one-off experiment into evidence another reviewer can reproduce and challenge. - Run the same cases in both evaluations. - Use the same expected outputs and scoring method. - Record the prompt diff and model route explicitly. - Do not compare partially scored or unrelated runs. ## Turn the comparison into a release decision Review the pass-rate delta first, then move immediately to the cases that flipped from pass to fail. Group those failures by reason: missing instruction, wrong classification, unsupported claim, broken format, or weak handling of missing context. The cluster usually points to the corrective action. The decision does not have to be a simple winner. You can ship, block, revise the prompt, keep the current model, route a narrow workload differently, or rerun after fixing the dataset. The report should preserve the evidence behind that call so teammates do not have to repeat the entire evaluation. - [Inspect a sample report](https://www.promptlens.io/r/demo-support-triage) ## How many test cases do I need for LLM regression testing? Start with 20 to 50 representative cases. Coverage of important failure modes matters more than raw volume. Add new cases whenever users or reviewers discover a meaningful failure. ## Should I use exact match or an LLM judge? Use deterministic checks for exact formats, required phrases, length, and patterns. Use an LLM judge when equivalent wording can satisfy the same expectation. Many suites benefit from both. ## Can I regression-test a model migration? Yes. Keep the prompt, test cases, and scoring rules fixed, then compare the current and candidate model routes side by side. ## Related resources - [Test an LLM model migration before switching](https://www.promptlens.io/use-cases/model-migration-testing) - [LLM evaluation dataset template](https://www.promptlens.io/templates/llm-evaluation-dataset) - [Support triage regression study](https://www.promptlens.io/benchmarks/support-triage-prompt-regression-july-2026)