Build a dataset

Each prompt has one ordered dataset shared by its saved versions. Start with representative cases, then add examples that reveal mistakes. A useful dataset distinguishes competing prompt versions; a large collection of nearly identical happy paths often does not.

Add inputs and references

Each row needs an input. A reference output is optional unless the evaluator or scoring mode requires it. Exact matching needs the expected label; a JSON-schema evaluator needs a schema, not an identical reference document. A support-reply judge can score an answer against a policy supplied in the input.

Choose Text or JSON independently for the input and reference. JSON fields must contain valid JSON, including when the value is a string, number, or other scalar. Formatting a reference as JSON does not make the Exact match evaluator compare objects structurally.

For a ticket router, cover each team and ambiguous requests. For extraction, include missing optional information and unusual characters. For support replies, include requests that the policy allows and requests it denies. Keep synthetic examples recognizable and remove customer secrets before using real cases.

Screenshot placeholder — Dataset with formats Show input and reference columns, Text/JSON selectors, and several representative tutorial rows.

Understand what changes affect

Editing the dataset affects future evaluations across versions. A completed run preserves the inputs it used; it does not acquire newly added cases. Dataset revisions protect shared edits from overwriting one another, but they are not a browsable history of separate datasets.

Quick runs use the first 10 cases, so order those cases deliberately. Comprehensive runs use the whole dataset and require Pro. See coverage and limits.

Use imports for a prepared collection. After adding cases, inspect evaluator scope: a rule scoped to selected rows will not necessarily cover a new row. Before initial setup finishes, every case needs an applicable evaluator.