{
  "markdown": "# Eval Skills for AI Coding Agents\n\n> [!IMPORTANT]\n> This repository is deprecated. The skills moved to [ai-evals-course/evals-skills](https://github.com/ai-evals-course/evals-skills), a plugin called `evals` that Shreya Shankar and Hamel Husain maintain together. Install it with:\n>\n> ```bash\n> npx skills add https://github.com/ai-evals-course/evals-skills\n> ```\n\nSkills that guide AI coding agents to help you build LLM evaluations.\n\nThese skills guard against common mistakes I've seen helping 50+ companies and teaching students in our [AI Evals course](https://maven.com/parlance-labs/evals?promoCode=evals-info-url). If you're new to evals, see [questions.md](questions.md) for free resources on the fundamentals.\n\n## New to Evals? Start Here\n\nIf you are new to evals, start with the `eval-audit` skill. Give your coding agent these instructions:\n\n> Install the eval skills plugin from https://github.com/hamelsmu/evals-skills, then run /evals-skills:eval-audit on my eval pipeline. Investigate each diagnostic area using a separate subagent in parallel, then synthesize the findings into a single report. Use other skills in the plugin as recommended by the audit.\n\nThe audit isn't a complete solution, but it will catch common problems we've seen in evals. It will also recommend other skills to use to fix the problems.\n\n## Installation\n\nIn Claude Code, run these two commands:\n\n```bash\n# Step 1: Register the plugin repository\n/plugin marketplace add hamelsmu/evals-skills\n\n# Step 2: Install the plugin\n/plugin install evals-skills@hamelsmu-evals-skills\n```\n\nTo upgrade:\n\n```bash\n/plugin update evals-skills@hamelsmu-evals-skills\n```\n\nAfter installation, restart Claude Code. The skills will appear as `/evals-skills:<skill-name>`.\n\n## Installation (npx skills)\n\nIf you use the open Skills CLI, install from this repo with:\n\n```bash\nnpx skills add https://github.com/hamelsmu/evals-skills\n```\n\nInstall one skill only:\n\n```bash\nnpx skills add https://github.com/hamelsmu/evals-skills --skill eval-audit\n```\n\nCheck for updates:\n\n```bash\nnpx skills check\nnpx skills update\n```\n\n## Available Skills\n\n| Skill | What it does |\n|-------|-------------|\n| eval-audit | Audit an eval pipeline and surface problems with prioritized severity |\n| error-analysis | Guide the user through reading traces and categorizing failures |\n| generate-synthetic-data | Create diverse synthetic test inputs using dimension-based tuple generation |\n| write-judge-prompt | Design LLM-as-Judge evaluators for subjective quality criteria |\n| validate-evaluator | Calibrate LLM judges against human labels using data splits, TPR/TNR, and bias correction |\n| evaluate-rag | Evaluate retrieval and generation quality in RAG pipelines |\n| build-review-interface | Build custom annotation interfaces for human trace review |\n\nInvoke a skill with `/evals-skills:skill-name`, e.g., `/evals-skills:error-analysis`.\n\n## Write Your Own Skills\n\nThese skills are a starting point and only encode common mistakes that generalize across projects. Skills grounded in your stack, your domain, and your data will outperform them. Start here, then write your own.\n\nThe [meta-skill](meta-skill.md) can help you ground custom skills. \n\n## Beyond These Skills\n\nThese skills handle the parts of eval work that generalize across projects. Much of the process doesn't: production monitoring, CI/CD integration, data analysis, and much more. The [course](https://maven.com/parlance-labs/evals?promoCode=evals-info-url) covers all of it.\n",
  "bytes": 3494,
  "sha": "859a62abd27ba9b0daa66455017dab9ad724f95e955481038fdaef981d6c394b",
  "repo_slug": "hamelsmu/evals-skills",
  "fonte": "repo",
  "truncated": false,
  "api": "https://agentalog.com/api/listings/skl_hamelsmu_evals_skills_eval_audit_076b089c/readme"
}