LLM evaluations.Ship with evidence.
Test prompt and model changes against production. See which cases regress, inspect the outputs, and decide what to publish with evidence in front of you.
No credit card required. Product access is subject to early-access approval.
Production · v1
3 / 3
cases passing
Candidate · v2
1 / 3
cases passing
Inspect a test case
Test input
The checkout page is down and we are losing orders every minute.
Exact-match expectation: urgent
Baseline · Pass
urgent
Candidate · Fail
technical
The candidate misses the urgency of an outage with revenue loss. The edited prompt no longer includes the escalation rule.
Sample data · Illustrative workflow, no live model calls.
PromptLens Evaluations tests saved prompt versions against a prompt's dataset. Compare a candidate with your production version, review scores and outputs, and keep the evidence behind each release decision.
A change is only better if it holds up.
Start with the cases your application needs to get right. Compare the same inputs and look closely at the failures.
Bring your cases
Build a dataset for your prompt with test inputs and the outputs you expect.
Compare versions
Evaluate a saved candidate against the production version pinned when the comparison starts.
Inspect and decide
Review changed outputs, assertion failures, and execution errors before moving a publishing label.
Quick feedback. A fuller check when you need it.
A quick evaluation runs the first 10 cases in your prompt dataset, or all cases if there are fewer. On Pro, a comprehensive evaluation runs the full dataset, with up to 500 cases per prompt.
Quick evaluations · Comprehensive evaluations on Pro
Keep the comparison anchored.
A comparison keeps the production version selected when it starts, even if production later moves. Reuse compatible completed results or choose to rerun both sides.
Pinned baseline · Compatible result reuse
Look beyond the overall score.
Inspect individual outputs and distinguish assertion failures from execution errors. Each run retains the prompt, model settings, and cases it tested, so you can review what produced the result.
Case-level results · Preserved run inputs
Keep the evidence close to the release.
Your evaluation belongs to a saved prompt version. Return to Prompts to review the change and deliberately move the production label when you're ready.
Explore PromptsQuestions about evaluations.
Straight answers about evaluations, how it works, and where to start.
Compare plans and limits- What is an LLM evaluation?
- An LLM evaluation tests model outputs against defined expectations. In PromptLens, you run a saved prompt version on its dataset, review case results, and compare a candidate with production to understand how a prompt or model change affects behavior.
- How do I test for prompt regressions?
- Save a candidate version and compare it with production on your prompt dataset. Inspect cases that passed in the baseline and fail in the candidate, then review the outputs before publishing. An improved overall score can still hide a regression in an important case.
- Can I compare model changes as well as prompt changes?
- Yes. Model settings are saved with each prompt version, so you can create a candidate with a different supported model and compare its results. Change one factor at a time when you want to understand what caused a difference.
- What is the difference between quick and comprehensive evaluations?
- Quick evaluations use the first 10 entries in a prompt's dataset, or all entries when there are fewer. Comprehensive evaluations require Pro and use every entry, including those first 10. Pro datasets support up to 500 entries per prompt.
- What happens if production changes during a comparison?
- The comparison keeps the production version that was selected when it started. Moving the production label later does not change that comparison's baseline.
- Do evaluations automatically block a release?
- Evaluations provide evidence and warnings for your publishing decision. They are not an automatic CI release gate. You review the results and explicitly choose which saved version receives a publishing label.
Know what changed before your users do.
Turn your test cases into evidence for the next prompt release.