{
"markdown": "\n\n# ML Superpowers\n\n**Agent skills for machine learning, distilled from the people who wrote the\nfield's best practices.**\n\nKarpathy's recipe · Google's Tuning Playbook · Andrew Ng · Zinkevich's Rules of\nML · applied LLM eval practice\n\n[Install](#install) · [The skills](#the-skills) · [The auditor](#the-auditor) · [Attribution](#attribution)\n\n
\n\n---\n\nRegular code fails loudly. You get a stack trace pointing at the line.\n\nML code doesn't. Overlap your train and test splits, permute your labels, invert\na padding mask, forget to load your adapter — **all of these run fine.** The loss\ngoes down. The number looks plausible. Nothing anywhere tells you the result is\ngarbage.\n\n```python\nn_train = 1800\nX_train, y_train = X[:n_train], y[:n_train]\nX_test, y_test = X[:200], y[:200] # should be X[n_train:]\n```\n\nThat prints a small train/test gap — which reads as *textbook healthy\ngeneralization* and is in fact caused by the leak.\n\nThe discipline that catches this isn't a trick. It's a known body of practice,\nwritten down over fifteen years by people who learned it the hard way. This repo\npackages it as skills your coding agent invokes at the moment it's relevant.\n\n## Install\n\n\nClaude Code
\n\n```bash\n/plugin marketplace add Umaraslam66/ml-superpowers\n/plugin install ml-superpowers@ml-superpowers\n```\n\n \n\n\nCodex · Cursor · OpenCode
\n\n```bash\ngit clone https://github.com/Umaraslam66/ml-superpowers.git ~/.codex/plugins/ml-superpowers\ngit clone https://github.com/Umaraslam66/ml-superpowers.git ~/.cursor/plugins/ml-superpowers\ngit clone https://github.com/Umaraslam66/ml-superpowers.git ~/.config/opencode/plugins/ml-superpowers\n```\n\nCursor without plugin support — generate a rules file instead:\n\n```bash\n./scripts/build-agents-md.sh --format cursor --out /path/to/project/.cursor/rules/ml-superpowers.mdc\n```\n\n \n\n\nGemini CLI
\n\n```bash\ngemini extensions install https://github.com/Umaraslam66/ml-superpowers\n```\n\n \n\n\nAny other agent (Windsurf, Zed, Aider, Continue, Cline, …)
\n\n```bash\n./scripts/build-agents-md.sh --out /path/to/project/AGENTS.md # full, ~11k words\n./scripts/build-agents-md.sh --index-only --out /path/to/project/AGENTS.md # routing only, ~400\n```\n\n \n\n> Only the Claude Code path has been tried on a live install. The other manifests\n> follow each harness's documented format — reports welcome.\n\n## The skills\n\n| Skill | Fires when |\n|---|---|\n| **training-neural-networks** | The recipe: six stages, six gates, in order |\n| **becoming-one-with-the-data** | Before model code; suspicious results |\n| **building-the-training-skeleton** | Writing a training loop; about to report a metric |\n| **overfitting-first** | Choosing model size; tempted to reach for dropout |\n| **regularizing-a-model** | Fits train, fails val |\n| **tuning-hyperparameters** | Sweeps; final performance push |\n| **debugging-silent-training-failures** | Runs clean, results are wrong |\n| **designing-ml-experiments** | Is this difference real? |\n| **choosing-what-to-fix** | What should I work on next? |\n| **shipping-ml-systems** | Production; offline good, live bad |\n| **evaluating-llm-systems** | Evals for an LLM feature, agent, or judge |\n| **weakest-hypothesis** | Inducing a rule, lesson, or root cause from few examples |\n\nThe order of the first six is load-bearing:\n\n```\nlook at data → dumb baselines → overfit → regularize → tune → squeeze\n```\n\nRegularizing before overfitting is the most expensive mistake in applied ML — it\nmakes underfitting and overfitting indistinguishable, and every later decision\nbecomes a guess. A model that can't overfit ten examples has a **bug**, and no\namount of regularization fixes a bug.\n\nEach skill also maps its checks onto modern work, because the failure mode is\nidentical:\n\n| Check | Fine-tuning | Eval harness | RAG |\n|---|---|---|---|\n| See the data | Read 200 prompts *post-template* | Check train/test contamination | Read retrieved chunks, not queries |\n| Loss @ init | Match the base model's loss | Score a known-perfect answer | Answer with no context |\n| Input-independent | Shuffle labels; must get worse | Grade a constant answer | Retrieve random chunks |\n| Overfit one batch | 10 examples → proves the loss mask | Hand-grade 10; harness must agree | 5 cases end-to-end |\n\nThat catches loss applied to prompt tokens, chat-template mismatch between\ntraining and inference, adapters never loaded at inference, and contaminated\neval sets.\n\n## The auditor\n\n`scripts/audit-run.py` is a static AST check for four leak patterns that all\nproduce a clean run and an optimistic number:\n\n| Finding | Meaning |\n|---|---|\n| `LEAK` | Train and eval slice the same array with overlapping ranges |\n| `GROUP` | Grouped data (patient, user, session, doc) split randomly — same entity on both sides |\n| `TIME` | Data with a time column split randomly — training on the future, testing on the past |\n| `PREP` | A scaler/encoder fit **before** the split — test statistics leaked into training |\n\nIt's one dependency-free file. You don't need a plugin, an agent, or this repo:\n\n```bash\ncurl -O https://raw.githubusercontent.com/Umaraslam66/ml-superpowers/main/scripts/audit-run.py\n\npython3 audit-run.py train.py # one file\npython3 audit-run.py src/ pipelines/ # directories, recursively\npython3 audit-run.py -q . # only print problems\n```\n\nExit `0` clean, `1` findings, `2` unparseable. Files that neither train nor\nevaluate a model are skipped silently, so pointing it at a whole repo is safe.\n\n> **Read findings as advisory.** `LEAK` and `PREP` are structural. `GROUP` and\n> `TIME` are name heuristics — they key off column names like `patient_id` or\n> `timestamp`, so they can be wrong in both directions. Useful in a review, not\n> strict enough to gate a merge.\n\nInstalled as a plugin, `hooks/post-tool-use` runs this automatically whenever a\ntraining script executes and puts any findings in front of the agent. Silent on\nclean scripts and unrelated commands.\n\n**GitHub Actions:**\n\n```yaml\n- name: Audit ML scripts\n run: |\n curl -sO https://raw.githubusercontent.com/Umaraslam66/ml-superpowers/main/scripts/audit-run.py\n python3 audit-run.py -q .\n```\n\n## Attribution\n\nEverything here is someone else's insight, organized for an agent to reach for\nat the right moment. Read the originals — they're better than any summary.\n\n- **Andrej Karpathy**, [A Recipe for Training Neural Networks](https://karpathy.github.io/2019/04/25/recipe/) (2019) — stages 1–6.\n- **Godbole, Dahl, Gilmer, Shallue, Nado**, [Deep Learning Tuning Playbook](https://github.com/google-research/tuning_playbook) — experiment design. Where it contradicts Karpathy (batch size as a regularizer), the skills document the tension rather than picking silently.\n- **Andrew Ng**, *Machine Learning Yearning* — error analysis and prioritization.\n- **Martin Zinkevich**, [Rules of Machine Learning](https://developers.google.com/machine-learning/guides/rules-of-ml) — production systems.\n- **Hamel Husain**, **Shreya Shankar** — applied LLM evals.\n- **Kapoor & Narayanan**, [Leakage and the Reproducibility Crisis in ML-based Science](https://arxiv.org/abs/2207.07048) — the leakage taxonomy behind the auditor.\n- **Michael Timothy Bennett**, [The Optimal Choice of Hypothesis Is the Weakest, Not the Shortest](https://arxiv.org/abs/2301.12987) — hypothesis selection by weakness, not description length.\n\nRepo structure follows [obra/superpowers](https://github.com/obra/superpowers).\n\n## Contributing\n\nNew skills welcome — data pipelines, distributed training, inference\noptimization, RLHF, quantization, serving. See [CONTRIBUTING.md](CONTRIBUTING.md).\n\n## License\n\nMIT — see [LICENSE](LICENSE).\n",
"bytes": 7867,
"sha": "4ade26f7cb2aafe0d7b8d3ad347390bacfe6cbb3c30b4b24249dc9c98e783fb5",
"repo_slug": "umaraslam66/ml-superpowers",
"fonte": "repo",
"truncated": false,
"api": "https://agentalog.com/api/listings/plg_umaraslam66_ml_superpowers_6369c2bb/readme"
}