{
  "markdown": "# Autoresearch — Claude Code Plugin\n\n93% of experiments fail. The value is in the 41 dead ends you eliminated, not only the 3 improvements you found.\n\nThis is a Claude Code plugin for running autonomous experiment loops on any codebase with a measurable metric. **The pattern:** one file, one metric, one loop. An agent edits a constrained file, runs an eval, keeps improvements, reverts failures, and repeats — unattended.\n\nBased on [Karpathy's autoresearch](https://github.com/karpathy/autoresearch), generalized beyond ML training to any code with a measurable outcome.\n\n## Skills\n\n### `/autoresearch-discover [path/to/directory]`\n\nDon't know where to start? This skill scans your codebase for autoresearch candidates — files with tunable parameters, magic numbers, scoring logic, or prompt templates that could be optimized against a metric. It outputs a ranked list with suggested metrics and eval difficulty, so you can pick a target and run `/autoresearch` on it.\n\n### `/autoresearch path/to/file.py`\n\nThe main skill. Once you know what to optimize:\n\n1. Reads the constrained file to identify tunable levers, then asks about your metric\n2. Generates a complete experiment harness: `instructions.md`, eval script, test data template, and launch prompt\n3. (Optional) Runs validation to confirm the eval produces a stable score\n4. Hands off to an autonomous Claude Code agent to run N iterations overnight\n\n## When to use it\n\n- **Discover:** You have a codebase and want to know what's optimizable\n- **Autoresearch:** You have a specific file with tunable code and a measurable metric\n\n## When NOT to use it\n\n- The problem requires a refactor, not tuning\n- There's no clear numerical metric to optimize\n- The eval is noisy or non-deterministic (network calls, random seeds, timing)\n- The fix is obvious and doesn't need iterative search\n\n## Installation\n\n### As a plugin (recommended)\n\n```\n/plugin marketplace add pjhoberman/autoresearch\n/plugin install autoresearch@autoresearch-marketplace\n```\n\nAfter installation, invoke with `/autoresearch:autoresearch path/to/file.py`.\n\n### Updating\n\n```\n/plugin marketplace update autoresearch-marketplace\n```\n\n### Manual\n\nCopy the `skills/autoresearch/` and `skills/autoresearch-discover/` directories into your project's `.claude/skills/` folder. This gives you `/autoresearch` and `/autoresearch-discover` directly.\n\n### Local development\n\n```bash\nclaude --plugin-dir /path/to/this/repo\n```\n\n## Repository structure\n\n```\n.claude-plugin/\n  plugin.json                   # Plugin manifest\n  marketplace.json              # Marketplace catalog for distribution\nskills/\n  autoresearch-discover/\n    SKILL.md                    # Codebase scanner — find optimization candidates\n  autoresearch/\n    SKILL.md                    # Main skill — generate experiment harness\n    templates/\n      instructions_template.md  # Template for the agent's instructions.md\n      eval_template.py          # Template for the eval script\n      launch_prompt.md          # Template for the Claude Code launch prompt\n    references/\n      lessons.md                # Real-world findings from production autoresearch runs\n```\n\n## Quick start\n\nIn Claude Code, with your codebase open:\n\n```\n# Step 1: Find optimization candidates\n/autoresearch-discover\n\n# Step 2: Pick a target and run autoresearch\n/autoresearch path/to/scoring.py\n```\n\nIf installed as a plugin, prefix with the plugin name: `/autoresearch:autoresearch-discover` and `/autoresearch:autoresearch path/to/scoring.py`.\n\nThe discover skill scans for tunable code and suggests metrics. Pick a candidate, then pass the file path to `/autoresearch` — it reads the file, identifies tunable levers, asks about your metric, generates the experiment harness, and hands off to an autonomous loop.\n\n## Templates\n\nThe `templates/` directory contains starter templates. **Do not use them as-is** — the skill adapts them heavily to your specific codebase. They define the structure and required sections.\n\n### `instructions_template.md`\n\nThe agent's operating manual. Covers:\n- What file it can edit (and what levers exist)\n- What it cannot touch (eval script, test data, frozen modules)\n- Exact eval command and metric definition\n- Strategy guidance: quick wins → main optimization → experimental\n- Commit discipline and log format\n\n### `eval_template.py`\n\nEval script with:\n- Metric functions: MRR, Precision@k, Hit Rate, Pass Rate\n- Caching pattern for expensive API calls that don't change between iterations\n- Standalone and Django management command entry points\n- Output format: must print `SCORE: X.XXXX` on its own line\n\n### `launch_prompt.md`\n\nShort prompt to paste into Claude Code. Points the agent at `instructions.md`, establishes baseline, starts the loop.\n\n## Lessons learned\n\n`references/lessons.md` documents findings from 60 iterations across two rounds on a production hybrid search system (Django/pgvector/Cohere/Claude Haiku). This is what separates this plugin from other autoresearch tools — real production data, not theory. Covers:\n\n- What worked and what didn't (with specific numbers)\n- The Redis caching trap that invalidated an entire round\n- Why temperature > 0 kills autoresearch signal\n- The co-optimization ceiling in sequential rounds\n- When to stop (diminishing returns arrive fast)\n\nRead this before designing your first experiment — it will save you an entire wasted round.\n\n## Key design principles\n\n**One file.** The constrained file discipline is what makes autonomous iteration safe. The agent can't break things it can't touch.\n\n**Fast evals.** Target < 60 seconds per iteration. Cache everything that doesn't change when the constrained file changes. At 30 iterations, a 6-minute eval is 3 hours; a 30-second eval is 15 minutes.\n\n**The failures are the output.** 93% of experiments fail. The value is as much in the definitively eliminated dead ends as in the improvements found. The \"what didn't work\" section of the final log is often more useful than the score improvement.\n\n**30 iterations is the default.** Most gains come in iterations 10-20. If the score plateaus for 5+ consecutive iterations, the ceiling is architectural, not parametric.\n\n**Guard what matters.** Optional guard metrics prevent the agent from improving one metric at the expense of another. Optimize precision while guarding MRR. Optimize speed while guarding accuracy.\n\n**Survive compaction.** The JSONL state file lets the agent recover after context window compaction — critical for 30+ iteration overnight runs.\n\n**Noise-aware.** Baseline stability checks and min-delta thresholds prevent the agent from chasing variance instead of signal.\n\n## Examples\n\n### Discovery output\n\nRunning `/autoresearch-discover` on a Django search backend might return something like this:\n\n```\nAutoresearch Discovery Report — 6 candidates found\nRanked by: eval feasibility × impact × isolation\n\n━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\nRANK 1 — search/geocoding.py                          ★★★ START HERE\n━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\nTunable parameters: 8\n  - NEARBY_RADIUS_KM (currently 25)\n  - CITY_POPULATION_TIERS = [10_000, 50_000, 250_000, 1_000_000]\n  - FALLBACK_RADIUS_MULTIPLIER (currently 2.5)\n  + 2 more distance/tier thresholds\n\nSuggested metric: address match accuracy on labeled dataset\nEval difficulty: LOW — validation tools already exist, ~200 labeled examples in fixtures/\nNotes: Best eval infrastructure already in place. Self-contained with no shared state.\n\n━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\nRANK 2 — search/scoring.py                            ★★ HIGH BUSINESS IMPACT\n━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\nTunable parameters: 5\n  - QUALITY_WEIGHTS = {completeness: 0.4, popularity: 0.3, recency: 0.2, verified: 0.1}\n  - MIN_SCORE_TO_INDEX (currently 0.45)\n\nSuggested metric: MRR@10 or NDCG@10 on search query log\nEval difficulty: MEDIUM — need to build eval harness, but labeled data exists\nNotes: Controls what surfaces to users. Most business-critical. Interacts with RANK 3.\n\n━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\nRANK 3 — search/promotion.py                          ★★ HIGH BUSINESS IMPACT\n━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\nTunable parameters: 3\n  - DISTANCE_DECAY_COEFFICIENT (currently 0.8)\n  - MIN_REVIEWS_TO_PROMOTE (currently 5)\n  - MIN_UNIQUE_USERS_TO_PROMOTE (currently 3)\n\nSuggested metric: precision@5 on promoted-result audit set\nEval difficulty: MEDIUM\nNotes: Gate for surfacing results to wider audiences. Optimize after RANK 2 — they interact.\n\n━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\nRANK 4 — matching/similarity.py                       ★ MODERATE\n━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\nTunable parameters: 4\n  - EMBEDDING_SIMILARITY_THRESHOLD (currently 0.82)\n  - EXACT_MATCH_BOOST (currently 1.5)\n  - FUZZY_MATCH_MIN_RATIO (currently 0.7)\n  - DEDUP_OVERLAP_THRESHOLD (currently 0.9)\n\nSuggested metric: duplicate detection F1 on labeled pairs\nEval difficulty: MEDIUM — labeled pairs dataset needs curation (~2h work)\n\n━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\nRANK 5 — display/snippet.py                           ★ MODERATE\n━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\nTunable parameters: 4\n  - MAX_SNIPPET_LENGTH (currently 160)\n  - CONTEXT_WINDOW_CHARS (currently 80)\n  - HIGHLIGHT_MAX_TERMS (currently 3)\n  - TRUNCATE_AT_SENTENCE_BOUNDARY (currently True)\n\nSuggested metric: human preference score or click-through rate proxy\nEval difficulty: HIGH — no ground truth; needs LLM-as-judge or user study\n\n━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\nRANK 6 — taxonomy/tag_linker.py                       ★ LOW PRIORITY\n━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\nTunable parameters: 3\n  - TAG_CONFIDENCE_THRESHOLD (currently 0.6)\n  - MAX_TAGS_PER_ITEM (currently 8)\n  - PARENT_TAG_BOOST (currently 1.2)\n\nSuggested metric: taxonomy coverage on labeled item set\nEval difficulty: HIGH — limited labeled data; tag ontology changes frequently\n\n━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\nRECOMMENDATION: Start with search/geocoding.py (RANK 1).\nIt has the best eval infrastructure already in place. Once you have a\nworking eval loop, RANK 2 + 3 are higher impact but require harness setup.\nAvoid RANK 5 and 6 until you have a reliable automated metric.\n━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\n```\n\nFrom here, run `/autoresearch search/geocoding.py` — it reads the file, confirms the metric with you, generates the eval harness, and hands off to the autonomous loop.\n\n## Prior art\n\n- [karpathy/autoresearch](https://github.com/karpathy/autoresearch) — The original. 630-line script, single GPU, ML training optimization. 42K+ stars.\n- [SkyPilot: Scaling Autoresearch](https://blog.skypilot.co/scaling-autoresearch/) — 16 GPUs, 910 experiments in 8 hours. Showed that parallelism changes what the agent can discover.\n- [Tobi Lütke / Shopify](https://x.com/tobi) — First non-ML application. 53% faster Liquid template rendering from ~120 automated experiments.\n- [uditgoenka/autoresearch](https://github.com/uditgoenka/autoresearch) — Domain-agnostic Claude skill with subcommands.\n- [pi-autoresearch](https://github.com/davebcn87/pi-autoresearch) — Autoresearch extension for the Pi editor.\n",
  "bytes": 11294,
  "sha": "db10fb5444969ddaee1d2dbe784aa1666d5f9f48569c9a450209a349af125300",
  "repo_slug": "pjhoberman/autoresearch",
  "fonte": "repo",
  "truncated": false,
  "api": "https://agentalog.com/api/listings/plg_pjhoberman_autoresearch_autoresearch_2cfd2462/readme"
}