Evaluate structured output

A bug-report extractor is a useful schema example: the output must be valid JSON with specific fields and types, even when the wording varies. This lesson checks structure. Add separate checks or reviewed cases when you also need to measure whether extracted facts are correct.

Configure the extractor

Create Bug report extractor, choose an available model, and use User message input mode. Add this system message:

Extract a bug report from the user's message. Return only a JSON object.
Use exactly these fields: title (string), severity (low, medium, or high),
and steps (array of strings). If no steps are supplied, use an empty array.
Use high for data loss or a blocked core workflow, medium for a broken
feature with a workaround, and low for cosmetic issues.
Do not add Markdown fences or invent steps.

Use this output schema. You can also download bug-report.schema.json.

{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "type": "object",
  "required": ["title", "severity", "steps"],
  "additionalProperties": false,
  "properties": {
    "title": { "type": "string", "minLength": 1 },
    "severity": { "type": "string", "enum": ["low", "medium", "high"] },
    "steps": { "type": "array", "items": { "type": "string" } }
  }
}

Add cases and an evaluator

Import bug-reports.json, or add these inputs with no reference output:

Add a JSON schema evaluator named Bug report shape, paste the same schema, and apply it to all cases. The output-schema setting expresses the requested shape; the evaluator checks the actual returned text against that shape.

Screenshot placeholder — Schema configuration and evaluator Show the output schema and Bug report shape evaluator with all three cases in scope.

Inspect the results

Run all cases to establish the baseline. A valid output might be:

{"title":"Settings icon is blurry","severity":"low","steps":[]}

{"severity":"critical"} fails because required fields are missing and the value is outside the enum. JSON wrapped in Markdown fences also fails JSON parsing. A structurally valid report with the wrong severity can still pass this schema: the enum restricts allowed values, not the factual correctness of the chosen value.

Screenshot placeholder — JSON-schema result Show the raw generated JSON and the evaluator's validation explanation for a malformed or missing field.

If structure fails, strengthen the formatting instruction and compare a saved candidate. If facts fail, add targeted evaluation criteria rather than expanding the schema to represent a semantic judgment. See LLM judges for criteria requiring interpretation.