ml-superpowers
ML engineering skills: training, fine-tuning, evaluation, and catching silent model failures
Open source Open in the app JSON README (API)
About
ML engineering skills: training, fine-tuning, evaluation, and catching silent model failures
Details
- Kind
- Plugins
- Topic
- AI, RAG & memory
- Publisher
- umaraslam66
- Origin
- gemini
- Category
- ferramentas
- Version
- 0.2.0
- Stars
- 3
- Last push
- 2026-08-05T11:47:46Z
- Repository state
- ativo
- Language
- Shell
- License
- MIT
- Added
- 2026-08-30 14:13:39
- Updated
- 2026-08-30 14:13:39
- Origin id
umaraslam66/ml-superpowers
README
<div align="center">
# ML Superpowers
**Agent skills for machine learning, distilled from the people who wrote the
field's best practices.**
Karpathy's recipe · Google's Tuning Playbook · Andrew Ng · Zinkevich's Rules of
ML · applied LLM eval practice
[Install](#install) · [The skills](#the-skills) · [The auditor](#the-auditor) · [Attribution](#attribution)
</div>
---
Regular code fails loudly. You get a stack trace pointing at the line.
ML code doesn't. Overlap your train and test splits, permute your labels, invert
a padding mask, forget to load your adapter — **all of these run fine.** The loss
goes down. The number looks plausible. Nothing anywhere tells you the result is
garbage.
```python
n_train = 1800
X_train, y_train = X[:n_train], y[:n_train]
X_test, y_test = X[:200], y[:200] # should be X[n_train:]
```
That prints a small train/test gap — which reads as *textbook healthy
generalization* and is in fact caused by the leak.
The discipline that catches this isn't a trick. It's a known body of practice,
written down over fifteen years by people who learned it the hard way. This repo
packages it as skills your coding agent invokes at the moment it's relevant.
## Install
<details open>
<summary><b>Claude Code</b></summary>
```bash
/plugin marketplace add Umaraslam66/ml-superpowers
/plugin install ml-superpowers@ml-superpowers
```
</details>
<details>
<summary><b>Codex</b> · <b>Cursor</b> · <b>OpenCode</b></summary>
```bash
git clone https://github.com/Umaraslam66/ml-superpowers.git ~/.codex/plugins/ml-superpowers
git clone https://github.com/Umaraslam66/ml-superpowers.git ~/.cursor/plugins/ml-superpowers
git clone https://github.com/Umaraslam66/ml-superpowers.git ~/.config/opencode/plugins/ml-superpowers
```
Cursor without plugin support — generate a rules file instead:
```bash
./scripts/build-agents-md.sh --format cursor --out /path/to/project/.cursor/rules/ml-superpowers.mdc
```
</details>
<details>
<summary><b>Gemini CLI</b></summary>
```bash
gemini extensions install https://github.com/Umaraslam66/ml-superpowers
```
</details>
<details>
<summary><b>Any other agent</b> (Windsurf, Zed, Aider, Continue, Cline, …)</summary>
```bash
./scripts/build-agents-md.sh --out /path/to/project/AGENTS.md # full, ~11k words
./scripts/build-agents-md.sh --index-only --out /path/to/project/AGENTS.md # routing only, ~400
```
</details>
> Only the Claude Code path has been tried on a live install. The other manifests
> follow each harness's documented format — reports welcome.
## The skills
| Skill | Fires when |
|---|---|
| **training-neural-networks** | The recipe: six stages, six gates, in order |
| **becoming-one-with-the-data** | Before model code; suspicious results |
| **building-the-training-skeleton** | Writing a training loop; about to report a metric |
| **overfitting-first** | Choosing model size; tempted to reach for dropout |
| **regularizing-a-model** | Fits train, fails val |
| **tuning-hyperparameters** | Sweeps; final performance push |
| **debugging-silent-training-failures** | Runs clean, results are wrong |
| **designing-ml-experiments** | Is this difference real? |
| **choosing-what-to-fix** | What should I work on next? |
| **shipping-ml-systems** | Production; offline good, live bad |
| **evaluating-llm-systems** | Evals for an LLM feature, agent, or judge |
| **weakest-hypothesis** | Inducing a rule, lesson, or root cause from few examples |
The order of the first six is load-bearing:
```
look at data → dumb baselines → overfit → regularize → tune → squeeze
```
Regularizing before overfitting is the most expensive mistake in applied ML — it
makes underfitting and overfitting indistinguishable, and every later decision
becomes a guess. A model that can't overfit ten examples has a **bug**, and no
amount of regularization fixes a bug.
Each skill also maps its checks onto modern work, because the failure mode is
identical:
| Check | Fine-tuning | Eval harness | RAG |
|---|---|---|---|
| See the data | Read 200 prompts *post-template* | Check train/test contamination | Read retrieved chunks, not queries |
| Loss @ init | Match the base model's loss | Score a known-perfect answer | Answer with no context |
| Input-independent | Shuffle labels; must get worse | Grade a constant answer | Retrieve random chunks |
| Overfit one batch | 10 examples → proves the loss mask | Hand-grade 10; harness must agree | 5 cases end-to-end |
That catches loss applied to prompt tokens, chat-template mismatch between
training and inference, adapters never loaded at inference, and contaminated
eval sets.
## The auditor
`scripts/audit-run.py` is a static AST check for four leak patterns that all
produce a clean run and an optimistic number:
| Finding | Meaning |
|---|---|
| `LEAK` | Train and eval slice the same array with overlapping ranges |
| `GROUP` | Grouped data (patient, user, session, doc) split randomly — same entity on both sides |
| `TIME` | Data with a time column split randomly — training on the future, testing on the past |
| `PREP` | A scaler/encoder fit **before** the split — test statistics leaked into training |
It's one dependency-free file. You don't need a plugin, an agent, or this repo:
```bash
curl -O https://raw.githubusercontent.com/Umaraslam66/ml-superpowers/main/scripts/audit-run.py
python3 audit-run.py train.py # one file
python3 audit-run.py src/ pipelines/ # directories, recursively
python3 audit-run.py -q . # only print problems
```
Exit `0` clean, `1` findings, `2` unparseable. Files that neither train nor
evaluate a model are skipped silently, so pointing it at a whole repo is safe.
> **Read findings as advisory.** `LEAK` and `PREP` are structural. `GROUP` and
> `TIME` are name heuristics — they key off column names like `patient_id` or
> `timestamp`, so they can be wrong in both directions. Useful in a review, not
> strict enough to gate a merge.
Installed as a plugin, `hooks/post-tool-use` runs this automatically whenever a
training script executes and puts any findings in front of the agent. Silent on
clean scripts and unrelated commands.
**GitHub Actions:**
```yaml
- name: Audit ML scripts
run: |
curl -sO https://raw.githubusercontent.com/Umaraslam66/ml-superpowers/main/scripts/audit-run.py
python3 audit-run.py -q .
```
## Attribution
Everything here is someone else's insight, organized for an agent to reach for
at the right moment. Read the originals — they're better than any summary.
- **Andrej Karpathy**, [A Recipe for Training Neural Networks](https://karpathy.github.io/2019/04/25/recipe/) (2019) — stages 1–6.
- **Godbole, Dahl, Gilmer, Shallue, Nado**, [Deep Learning Tuning Playbook](https://github.com/google-research/tuning_playbook) — experiment design. Where it contradicts Karpathy (batch size as a regularizer), the skills document the tension rather than picking silently.
- **Andrew Ng**, *Machine Learning Yearning* — error analysis and prioritization.
- **Martin Zinkevich**, [Rules of Machine Learning](https://developers.google.com/machine-learning/guides/rules-of-ml) — production systems.
- **Hamel Husain**, **Shreya Shankar** — applied LLM evals.
- **Kapoor & Narayanan**, [Leakage and the Reproducibility Crisis in ML-based Science](https://arxiv.org/abs/2207.07048) — the leakage taxonomy behind the auditor.
- **Michael Timothy Bennett**, [The Optimal Choice of Hypothesis Is the Weakest, Not the Shortest](https://arxiv.org/abs/2301.12987) — hypothesis selection by weakness, not description length.
Repo structure follows [obra/superpowers](https://github.com/obra/superpowers).
## Contributing
New skills welcome — data pipelines, distributed training, inference
optimization, RLHF, quantization, serving. See [CONTRIBUTING.md](CONTRIBUTING.md).
## License
MIT — see [LICENSE](LICENSE).