Evaluation guide
How to build an LLM evaluation dataset
Create a small, representative test set that exposes real LLM failure modes and stays useful as your application changes.
- Updated
- July 17, 2026
- Reading time
- 7 minutes
An evaluation dataset is the set of inputs and expectations you rerun when a prompt, model, or application behavior changes. Its job is not to represent every possible request. Its job is to make important failures visible before users find them.
The most effective datasets start small, use real workflow language, and improve whenever the team learns something new. They behave more like a regression suite than a one-time benchmark.
Key takeaways
Start from decisions and failures
Choose cases based on what the system must decide correctly and what becomes expensive when it fails.
Write reviewable expectations
An expected output should name the required substance, format, or behavior without prescribing irrelevant wording.
Treat the dataset as a living asset
Add production failures, remove redundant cases, and version meaningful expectation changes.
1. Define the behavior contract
Write down what the AI feature is responsible for and what it must never do. A support triage system may need to select one category, escalate revenue-impacting outages, and avoid adding prose. A RAG answer may need to include two supplied facts, cite a source label, and state uncertainty when context is missing.
This contract becomes the basis for case selection and scoring. If the team cannot describe the expected behavior, the evaluation will collapse into subjective preference.
2. Collect representative and adversarial cases
Begin with real requests from development, support, user research, or sanitized production traces. Cover the normal distribution, then deliberately add boundary cases: unclear inputs, conflicting instructions, missing data, unusual formatting, and the high-impact failures that are rare but costly.
- Typical cases that represent the majority of traffic.
- Previously observed failures that must never return.
- High-risk cases involving money, access, safety, or irreversible actions.
- Format and integration cases that can break downstream code.
- Cases with insufficient context where the correct behavior is to ask or abstain.
3. Write expectations that can be scored
Match the expectation to the task. Use an exact label for classification, a required structure for extraction, required facts for grounded answers, and a concise reference answer for open-ended responses. Separate essential requirements from stylistic preferences.
When several answers could be correct, do not force a single sentence match. Capture the information and constraints a passing answer must satisfy, then use an LLM judge or deterministic checks appropriate to those requirements.
{
"input": "Checkout is down and we are losing orders.",
"expectedOutput": "urgent",
"tags": ["outage", "revenue-impact", "release-blocker"],
"source": "sanitized-support-example"
}4. Separate development cases from release cases
Use a working set while editing prompts and a stable holdout set for the final release comparison. Constantly tuning against every case can hide overfitting: the prompt becomes excellent at the visible examples without becoming more reliable for users.
Review the suite on a schedule. Merge duplicates, repair ambiguous expectations, and add every meaningful production failure. Version changes that alter the release bar so historical comparisons remain interpretable.
Frequently asked questions
Can I use synthetic test cases?
Yes, especially for expanding known failure categories, but review them before they enter the release suite. Keep a clear distinction between real, sanitized, and synthetic sources.
What fields should an LLM evaluation dataset contain?
At minimum: a stable identifier, input, expected output or behavior, and useful tags. Source, risk level, and notes make maintenance easier.
How often should the dataset change?
Update it whenever a meaningful failure is found and review it periodically for ambiguity or duplication. Preserve versions when expectations change materially.