Code Assistant Evaluation & Testing

AI code assistants generate millions of lines of code daily. Without systematic testing, bugs and security issues slip into production. 1. **Functional correctness** — Write test cases with specific inputs and expected outputs. Run the generated code in a sandbox and validate results. This catches logical errors that look correct in review. 2. **Security scanning** — Automatically scan generated code for OWASP Top 10 vulnerabilities. Pay special attention to SQL injection, XSS, and hardcoded credentials — patterns that LLMs frequently reproduce from training data. 3. **Style consistency** — Test that generated code follows your team's conventions: naming patterns, error handling approach, import organization. Inconsistent style increases maintenance burden. 4. **Regression across models** — When you switch models or update prompts, re-run your full test suite. A model upgrade that improves Python generation might degrade TypeScript output.

// Validate generated code quality
{
  prompt: "Write a function to validate email",
  tests: [
    { input: "user@example.com", expected: true },
    { input: "invalid-email", expected: false },
    { input: "a@b.c", expected: false },
  ],
  security_checks: [
    "no_eval_usage",
    "no_regex_dos",
    "input_sanitization"
  ]
}