Free template
LLM evaluation dataset template
A practical JSON and CSV-ready structure for collecting test inputs, expected behavior, risk tags, and failure history.
- Updated
- July 17, 2026
- Reading time
- 5 minutes
Use this template to turn real application behavior into a repeatable evaluation suite. It keeps the minimum scoring fields simple while preserving enough context to understand why a case belongs in the release gate.
Copy the compact version to start immediately. Add source and risk metadata when the dataset becomes a shared team asset.
Key takeaways
Required fields stay small
A stable ID, input, and expected output are enough to run a useful first comparison.
Metadata explains the risk
Tags, source, and risk level help reviewers understand coverage and choose the cases that should block a release.
The template works for many tasks
Use exact expectations for classification or structured extraction and behavioral expectations for open-ended answers.
Compact JSON template
This is the smallest useful format. Keep each ID stable so results remain traceable when you edit wording or add more cases.
[
{
"id": "case-001",
"input": "The checkout page is down and we are losing orders.",
"expectedOutput": "urgent"
},
{
"id": "case-002",
"input": "Please send last month's invoice.",
"expectedOutput": "billing"
}
]Extended team template
Use the extended fields to explain provenance and maintenance. Never put raw sensitive customer data into an evaluation file; sanitize the language while preserving the behavior that made the case difficult.
{
"id": "support-outage-001",
"input": "Checkout is down and orders are failing.",
"expectedOutput": "urgent",
"tags": ["support", "outage", "revenue-impact"],
"risk": "release-blocker",
"source": "sanitized-production-failure",
"addedAt": "2026-07-17",
"notes": "Must override the narrower technical category."
}CSV column reference
For spreadsheet workflows, use id, input, expectedOutput, tags, risk, source, addedAt, and notes as columns. Store tags as a pipe-separated list if your import process expects a single cell.
- id: stable, human-readable case identifier.
- input: the exact message or variable payload sent to the prompt.
- expectedOutput: the answer, facts, classification, or behavior required to pass.
- risk: informational, important, or release-blocker.
- source: synthetic, user research, support, or sanitized production failure.
- notes: why the case exists and what past change caused it to fail.
Dataset quality checklist
Before using the dataset as a release gate, review whether every expectation is understandable without private context and whether two reasonable reviewers would agree about a passing answer.
- Each case maps to a real behavior or known risk.
- Inputs do not expose secrets or identifiable customer data.
- Expected outputs describe requirements, not arbitrary stylistic preferences.
- The suite contains edge cases and previously observed failures.
- Duplicate cases have been merged and ambiguous cases repaired.