Provider integration
Test OpenRouter models with PromptLens
Connect a workspace-owned OpenRouter key, select explicit model routes, and compare candidate behavior against a baseline.
- Updated
- July 17, 2026
- Reading time
- 6 minutes
OpenRouter provides one API surface for models from multiple providers. PromptLens adds the evaluation workflow around that access: a shared test set, explicit model routes, consistent scoring, measured usage when available, and a report that explains the release decision.
Your OpenRouter key belongs to the PromptLens workspace. Live runs use that key for the selected route; the demo remains available without connecting a provider.
Key takeaways
Use your workspace key
Create an OpenRouter key with the limit appropriate for evaluation and connect it at workspace scope.
Choose explicit model routes
A reproducible comparison records the exact route used for the baseline and candidate.
Evaluate behavior on your cases
Provider benchmarks are context; your release decision should use the inputs and failure modes of your application.
1. Create and connect an OpenRouter key
Create an API key in OpenRouter and set a credit limit that matches the evaluation workload. OpenRouter authenticates completion requests with a bearer token and supports limits on keys. Store the key in PromptLens workspace provider settings rather than putting it inside prompts, datasets, or source code.
PromptLens validates the provider connection before a live run. Workspace-level configuration keeps model access separate from individual prompt projects and makes the selected route visible to teammates.
2. Select the routes you want to compare
Choose a known-good baseline and one candidate change. Keep the prompt, dataset, and scoring configuration fixed when the purpose is model selection. If the purpose is prompt regression testing, keep the model route fixed and compare prompt versions instead.
OpenRouter can route across providers and offers provider preferences, fallbacks, data-policy filters, and performance thresholds. Treat routing configuration as part of the evaluated system: changing it can change latency, availability, price, or output behavior.
3. Run the same case set and scoring rules
Import representative inputs and expected outputs, run them through both routes, and score every result under the same conditions. Compare pass rate, failed cases, latency, and reported or estimated model cost when that metadata is available for the route.
Start with the failures rather than declaring a winner from one aggregate score. A cheaper model may be viable for a narrow workflow even when it is not the overall winner. A stronger model may still be unacceptable if it breaks a required output format.
4. Review OpenRouter routing and data policy controls
OpenRouter documents provider-specific data retention and offers routing controls such as data-collection filters and zero-data-retention endpoints. Those controls belong to the upstream route and account configuration; connecting OpenRouter to PromptLens does not replace reviewing them for your own requirements.
Record the route and policy assumptions used for the evaluation. If production will use a restricted provider set, the release comparison should use the same restriction rather than a more permissive route.
Frequently asked questions
Does PromptLens provide the OpenRouter credits?
No. Live evaluations use the OpenRouter key connected by your workspace, so provider billing and limits remain attached to that account.
Can I compare OpenRouter with a direct provider route?
Yes, when both routes are configured in the workspace. Record the route used for each evaluation and keep the case set and scoring method identical.
Does an OpenRouter model slug guarantee one upstream provider?
Not necessarily. OpenRouter supports provider routing and fallbacks. Use its routing controls when a comparison requires a specific provider or data policy.