Prompt lifecycle from the CLI
Create, evaluate, edit, compare, and publish a ticket router.
This walkthrough uses the five ticket-routing references with expected scoring. Install and sign in to the CLI, select an organization, and ensure a model route is usable. It uses a POSIX shell, jq, and uuidgen.
Download ticket-router.prompt.json and ticket-routing.json into your working directory. The prompt file selects an OpenRouter route; change its model fields if your organization uses another route. Use a new tutorial prompt so the workflow does not overwrite existing work.
Create and import
promptlens organizations use --org YOUR_ORGANIZATION_ID
promptlens models list --all --json
CREATE_REQUEST_KEY=$(uuidgen)
promptlens prompts create --file ticket-router.prompt.json \
--idempotency-key "$CREATE_REQUEST_KEY" --json > created-prompt.json
PROMPT_ID=$(jq -er '.data.id' created-prompt.json)
promptlens dataset get --prompt-id "$PROMPT_ID" --json > dataset.json
jq --slurpfile cases ticket-routing.json \
'{revision: .data.revision, mode: "append", rows: $cases[0]}' \
dataset.json > import.json
promptlens dataset import --prompt-id "$PROMPT_ID" --file import.json --yes --jsonCLI JSON wraps success in data; raw API responses do not have that wrapper. Inspect every command's exit status and output before moving on. The imported references are the expected classification labels, not dashboard evaluator definitions.
Initialize the baseline
Running evaluations incurs model usage. --yes records your decision to proceed for the command; it does not bypass organization permissions.
promptlens draft get --prompt-id "$PROMPT_ID" --json > draft.json
DRAFT_REVISION=$(jq -er '.data.revision' draft.json)
promptlens prompts initialize --prompt-id "$PROMPT_ID" \
--draft-revision "$DRAFT_REVISION" --scope quick \
--scoring-method expected --yes --wait --json
promptlens versions list --prompt-id "$PROMPT_ID" --all --jsonWait for the initial run to complete and inspect its results before continuing. Use promptlens evals get --evaluation-id EVALUATION_ID --json and promptlens evals results --evaluation-id EVALUATION_ID --all --json with the returned run ID. A completed baseline establishes version 1 and its labels.
Save a focused change
Read the latest personal draft and add an ambiguity rule. This example edits the first system message from the supplied prompt file.
promptlens draft get --prompt-id "$PROMPT_ID" --json > draft.json
jq '.data | {revision, content} | .content.messages[0].content += "\nWhen multiple issues appear, route by the immediate blocker. If the customer cannot sign in, choose account even if they mention an invoice."' \
draft.json > updated-draft.json
promptlens draft update --prompt-id "$PROMPT_ID" \
--file updated-draft.json --json > saved-draft.json
DRAFT_REVISION=$(jq -er '.data.revision' saved-draft.json)
promptlens versions save --prompt-id "$PROMPT_ID" \
--draft-revision "$DRAFT_REVISION" --json > candidate.json
VERSION_ID=$(jq -er '.data.id' candidate.json)Add a case that exercises the new instruction. The dataset revision comes from a fresh read:
promptlens dataset get --prompt-id "$PROMPT_ID" --json > dataset.json
jq '{revision: .data.revision, mode: "append", rows: [{input: "I cannot sign in to download my invoice.", expectedOutput: "account"}]}' \
dataset.json > new-case.json
promptlens dataset import --prompt-id "$PROMPT_ID" --file new-case.json --yes --json
promptlens evals start --prompt-id "$PROMPT_ID" --version-id "$VERSION_ID" \
--scoring-method expected --scope quick --comparison rerun_both \
--yes --wait --jsonBoth sides now run against the six-case dataset. Inspect the candidate and production run IDs, actual outputs, and scores. Ctrl+C stops waiting without cancelling accepted runs. See run management for explicit cancellation and retry.
Publish after review
Read labels and prepare a production assignment to the candidate you inspected:
promptlens labels list --prompt-id "$PROMPT_ID" --all --json > labels.json
jq --arg version "$VERSION_ID" \
'.data.items[] | select(.name == "production") | {revision, versionId: $version}' \
labels.json > publish.json
promptlens labels assign --prompt-id "$PROMPT_ID" --label-name production \
--file publish.json --yes --jsonInspect data.published. If false, read data.warnings and the returned acknowledgment. Only after deciding to accept that coverage context, add acknowledgment to the request body and resubmit. If the label revision conflicts, reload and review the current assignment first.
To roll back, repeat the assignment using an earlier saved version ID and the current label revision. To use the published prompt, follow application retrieval.