Use LLM judges

A support reply can be correct in many different words. Use an LLM judge to evaluate its meaning against explicit criteria. This lesson uses a fictional refund policy, so the judge has enough context to assess the answer.

Create the support prompt

Create Refund support assistant, select a generation model, and use Variables input mode. Add this system message:

Write a concise, helpful customer-support reply using only the supplied policy.
Explain eligibility and the next step. Do not claim to have processed a refund,
accessed an account, or made an exception. Do not invent policy details.

Add this user message:

Policy: {{policy}}
Customer: {{question}}

Import support-replies.json. It includes three JSON inputs using this fictional policy:

Unused purchases can be refunded within 30 days. Used purchases and purchases
older than 30 days are not eligible. To request a refund, contact
support@example.com with the order number. Support reviews requests;
no refund is completed in this chat.

The cases ask about an unused purchase after 12 days, a used purchase after 12 days, and an unused purchase after 45 days. They exercise an eligible request and two distinct reasons to decline.

Add a checklist judge

Create an LLM checklist evaluator named Policy-safe reply. Choose its judge model explicitly, apply it to all cases, and require All items to pass:

• The reply correctly determines eligibility from the policy and customer question in the input.
• The reply explains the relevant policy without adding an exception or unsupported condition.
• When eligible, the reply gives the support address and asks for the order number.
• The reply does not claim a refund was processed or an account was accessed.

The conditional third item matters: an ineligible customer should not fail merely because the reply does not encourage an eligible refund request. Write criteria so the judge can apply them to every scoped case.

Add a score judge

Create an LLM score evaluator named Helpful explanation, select its judge model, and set the criterion to “How clearly does the reply explain the policy decision and the appropriate next step for this customer?” Use these levels and a passing threshold of 2:

For this example, set each judge temperature to 0 and its maximum output tokens to 512. Each judge has its own model configuration; changing the generation model does not choose a judge model for you. Judge calls add model usage and may incur charges.

Screenshot placeholder — Support-reply judges Show the checklist's All setting, explicit judge model, score levels, threshold, and a case's explanation.

Calibrate before relying on the score

Run the cases and read the actual reply alongside the judge's explanation. A friendly answer that says “I've refunded you” should fail the checklist. A polite refusal for a 45-day purchase can pass even though the customer does not receive the refund they requested.

Compare judge decisions with your own review, especially near the threshold. Rewrite vague or contradictory criteria and rerun before treating scores as a release signal. With both evaluators in scope, a case must pass both; a high helpfulness score does not override a policy failure. See results and review for recording human decisions.