start evaluation

Pin current production at request acceptance and freeze the latest ordered dataset, saved version/model/settings, and scoring policy. Quick uses first 10 (all available on Free); comprehensive uses all (Pro only). Reuse most recent compatible completed production evidence by default, otherwise evaluate both sides in parallel. rerun_both forces fresh evidence. No model/settings overrides for saved versions. Start requests are idempotent to prevent duplicate charges.

POST/api/v1/organizations/{organizationId}/prompts/{promptId}/evals

Authorization

ManagementOAuth management

AuthorizationBearer <token>

User-based Clerk OAuth. CLI is a public Authorization Code client using S256 PKCE and browser loopback redirect. Hosted MCP uses resource-bound OAuth. Verify issuer, token type, intended audience/resource, scopes, live membership and current role. Do not forward arbitrary OAuth tokens into Convex session JWT auth. OAuth URLs below are placeholders for deployment configuration.

In: header

Scope: management

Path Parameters

organizationId*string

Permanent organizationId; selected organization must be authorized for the current user.

promptId*string

Permanent promptId; selected organization must be authorized for the current user.

Header Parameters

Idempotency-Key*string

Management writes: scoped to OAuth user+client+organization+operation. Store request digest and response for at least 24 hours. Same key/body replays original response, including after completion; different body fails idempotency_conflict. Exception: create_retrieval_key stores a non-secret receipt referencing the created key, never its secret. Same-key/body retries return 409 secret_unavailable with existing key metadata and do not create another key. Evaluation creation must not execute twice.

Length8 <= length <= 128

Request Body

application/json

TypeScript Definitions

Use the request body type in TypeScript.

Response Body

`application/json`

curl -X POST "https://example.com/api/v1/organizations/string/prompts/string/evals" \  -H "Idempotency-Key: stringst" \  -H "Content-Type: application/json" \  -d '{    "versionId": "string",    "scoringMethod": "expected"  }'

{  "candidateRunId": "string",  "productionRunId": "string",  "productionReused": true}

POST/api/v1/organizations/{organizationId}/prompts/{promptId}/evals

Authorization

ManagementOAuth management

AuthorizationBearer <token>

In: header

Scope: management

Path Parameters

organizationId*string

Permanent organizationId; selected organization must be authorized for the current user.

promptId*string

Permanent promptId; selected organization must be authorized for the current user.

Header Parameters

Idempotency-Key*string

Length8 <= length <= 128

Request Body

application/json

TypeScript Definitions

Use the request body type in TypeScript.

versionId*string

scope?string

Default"quick"

Value in

"quick"
"comprehensive"

scoringMethod*string

Value in

"expected"
"human_review"
"llm_judge"

comparison?string

Default"reuse_production"

Value in

"reuse_production"
"rerun_both"

Response Body

`application/json`

curl -X POST "https://example.com/api/v1/organizations/string/prompts/string/evals" \  -H "Idempotency-Key: stringst" \  -H "Content-Type: application/json" \  -d '{    "versionId": "string",    "scoringMethod": "expected"  }'

{  "candidateRunId": "string",  "productionRunId": "string",  "productionReused": true}

Account and organizations

Members and invitations

Prompts and drafts

Versions and labels

Datasets

Evaluations

Model access

Retrieval keys

Billing

Warnings and observability

202application/json

400application/json

401application/json

403application/json

404application/json

409application/json

422application/json

429application/json

503application/json

Account and organizations

Members and invitations

Prompts and drafts

Versions and labels

Datasets

Evaluations

Model access

Retrieval keys

Billing

Warnings and observability

start evaluation

202application/json

400application/json

401application/json

403application/json

404application/json

409application/json

422application/json

429application/json

503application/json

`application/json`

`application/json`

`application/json`

`application/json`

`application/json`

`application/json`

`application/json`

`application/json`

`application/json`

`application/json`

`application/json`

`application/json`

`application/json`

`application/json`

`application/json`

`application/json`

`application/json`

`application/json`