Cancel, retry, and review runs

Evaluations run in the background after PromptLens accepts them. Closing a tab, disconnecting an assistant, logging out, or interrupting a CLI wait does not cancel accepted work.

Cancel work explicitly

Open the run and use Cancel, or call the cancellation operation with its evaluation ID. In a comparison, inspect both candidate and production IDs so you know which work you are cancelling. Already completed provider calls may still incur charges; cancellation is not a refund.

With the CLI, stop waiting with Ctrl+C when you only want your terminal back. Use the explicit command when cancellation is intended:

promptlens evals cancel --evaluation-id EVALUATION_ID --yes

Retry a failed side

Inspect the run's error and resolve the underlying issue, such as missing provider access, exhausted credits, or a temporary provider failure. Retry the failed comparison side to use its frozen evaluation context. Retry does not substitute the latest draft or newly edited dataset into that original comparison.

promptlens evals retry --evaluation-id EVALUATION_ID --yes --wait --json

Retries can incur new model usage. If you want to test different instructions or cases, save the change and start a new evaluation instead.

Review unscored results

For a human-review run, open each pending result, read its input and output, and record a pass or fail decision with a useful explanation. Review the same criteria consistently across versions. “Failed because the reply promised a refund outside the policy” is more useful than “bad answer.”

Management interfaces expose expected, human_review, and llm_judge scoring modes. The routing walkthroughs use expected; select human review when a person should decide each outcome. Dashboard-configured LLM score and checklist evaluators have their own judge settings.

Finish pending reviews before presenting the aggregate as a complete result or creating a public report.