{
  "markdown": "<p align=\"center\">\n  <img src=\"assets/banner.jpg\" alt=\"Harness Evolver\" width=\"100%\">\n</p>\n\n# Harness Evolver\n\n<p align=\"center\">\n  <a href=\"https://www.npmjs.com/package/harness-evolver\"><img src=\"https://img.shields.io/npm/v/harness-evolver?style=for-the-badge&color=blueviolet\" alt=\"npm\"></a>\n  <a href=\"https://github.com/raphaelchristi/harness-evolver/blob/main/LICENSE\"><img src=\"https://img.shields.io/badge/License-MIT-green?style=for-the-badge\" alt=\"License: MIT\"></a>\n  <a href=\"https://arxiv.org/abs/2603.28052\"><img src=\"https://img.shields.io/badge/Paper-Meta--Harness-FFD700?style=for-the-badge\" alt=\"Paper\"></a>\n  <a href=\"https://github.com/raphaelchristi/harness-evolver\"><img src=\"https://img.shields.io/badge/Built%20by-Raphael%20Valdetaro-ff69b4?style=for-the-badge\" alt=\"Built by Raphael Valdetaro\"></a>\n</p>\n\nPoint at any LLM agent codebase. Harness Evolver will autonomously improve it — prompts, routing, tools, architecture — using multi-agent evolution with LangSmith as the evaluation backend.\n\n---\n\n## Install\n\n### Claude Code Plugin (recommended)\n\n```\n/plugin marketplace add raphaelchristi/harness-evolver-marketplace\n/plugin install harness-evolver\n```\n\n### npx (first-time setup or non-Claude Code runtimes)\n\n```bash\nnpx harness-evolver@latest\n```\n\nWorks with Claude Code, Cursor, Codex, and Windsurf.\n\n---\n\n## Quick Start\n\n```bash\ncd my-llm-project\nexport LANGSMITH_API_KEY=\"lsv2_pt_...\"\nclaude\n\n/harness:setup      # explores project, configures LangSmith\n/harness:health     # check dataset quality (auto-corrects issues)\n/harness:evolve     # runs the optimization loop\n/harness:status     # check progress (rich ASCII chart)\n/harness:deploy     # tag, push, finalize\n```\n\n---\n\n## What It Looks Like\n\nTested on a RAG agent (Agno framework, Gemini 3.1 Flash Lite, light mode):\n\n```mermaid\nxychart-beta\n    title \"agno-deepknowledge: 0.575 → 1.000 (+74%)\"\n    x-axis [\"base\", \"v001\", \"v002\", \"v003\", \"v004\", \"v005\", \"v006\", \"v007\"]\n    y-axis \"Correctness\" 0 --> 1\n    line [0.575, 0.575, 0.950, 0.950, 0.950, 0.950, 0.950, 1.0]\n    bar [0.575, 0.333, 0.950, 0.720, 0.875, 0.680, 0.880, 1.0]\n```\n\n| Iter | Score | Merged? | What the proposer did |\n|---|---|---|---|\n| baseline | 0.575 | — | Original agent — hallucinations, broken tool calls, no retry logic |\n| v001 | 0.333 | Yes | Anti-hallucination prompt (100% correct when API responded, but 60% hit rate limits) |\n| v002 | 0.950 | Yes | **Breakthrough**: inlined 17-line KB into prompt, eliminated vector search entirely. 5.7x faster, zero rate limits |\n| v003 | 0.720 | **No** | Attempted hybrid retrieval — regressed, rejected by constraint gate |\n| v004 | 0.875 | **No** | Response completeness fix — improved one case but regressed others |\n| v005 | 0.680 | **No** | Reduced tool calls — broke edge cases, rejected |\n| v006 | 0.880 | Yes | Evolution memory insight: combined v001's anti-hallucination with one-shot example from archive |\n| v007 | 1.000 | Yes | One-shot example injection + rubric-aligned responses — perfect on held-out |\n\nThe line shows best score (only goes up — regressions aren't merged). The bars show each candidate's raw score. 4 merged, 3 rejected by gate checks. Not every iteration improves — that's the point.\n\n---\n\n## How It Works\n\n| | |\n|---|---|\n| **LangSmith-Native** | No custom scripts. Uses LangSmith Datasets, Experiments, and LLM-as-judge. Everything visible in the LangSmith UI. |\n| **Real Code Evolution** | Proposers modify actual code in isolated git worktrees. Winners merge automatically. |\n| **Self-Organizing Proposers** | Two-wave spawning, dynamic lenses from failure data, archive branching from losing candidates. Self-abstention when redundant. |\n| **Rubric-Based Evaluation** | LLM-as-judge with justification-before-score, rubrics, few-shot calibration, pairwise comparison. |\n| **Smart Gating** | Constraint gates, efficiency gate (cost/latency pre-merge), regression guards, Pareto selection, holdout enforcement, rate-limit early abort, stagnation detection. |\n\n[Full feature list](docs/FEATURES.md)\n\n---\n\n## Evolution Loop\n\n```\n/harness:evolve\n  |\n  +- 1. Preflight  (validate state + dataset health + baseline scoring)\n  +- 2. Analyze    (trace insights + failure clusters + strategy synthesis)\n  +- 3. Propose    (spawn N proposers in git worktrees, two-wave)\n  +- 4. Evaluate   (canary → run target → auto-spawn LLM-as-judge → rate-limit abort)\n  +- 5. Select     (held-out comparison → Pareto front → efficiency gate → constraint gate → merge)\n  +- 6. Learn      (archive candidates + regression guards + evolution memory)\n  +- 7. Gate       (plateau → target check → critic/architect → continue or stop)\n```\n\n[Detailed loop with all sub-steps](docs/ARCHITECTURE.md)\n\n---\n\n## Agents\n\n| Agent | Role |\n|---|---|\n| **Proposer** | Self-organizing — investigates a data-driven lens, decides own approach, may abstain |\n| **Evaluator** | LLM-as-judge — rubric-aware scoring via langsmith-cli, few-shot calibration |\n| **Architect** | ULTRAPLAN mode — deep topology analysis with Opus model |\n| **Critic** | Active — detects evaluator gaming, implements stricter evaluators |\n| **Consolidator** | Cross-iteration memory — anchored summarization, garbage collection |\n| **TestGen** | Generates test inputs with rubrics + adversarial injection |\n\n---\n\n## Requirements\n\n- **LangSmith account** + `LANGSMITH_API_KEY`\n- **Python 3.10+** · **Git** · **Claude Code** (or Cursor/Codex/Windsurf)\n\nDependencies installed automatically by the plugin hook or npx installer.\n\nLangSmith traces any AI framework: LangChain/LangGraph (auto), OpenAI/Anthropic SDK (`wrap_*`, 2 lines), CrewAI/AutoGen (OpenTelemetry), any Python (`@traceable`).\n\n---\n\n## Companion: LangSmith Tracing\n\nFor full observability into what each proposer does during evolution (every file read, edit, and commit), install the [LangSmith tracing plugin](https://github.com/langchain-ai/langsmith-claude-code-plugins):\n\n```\n/plugin marketplace add langchain-ai/langsmith-claude-code-plugins\n/plugin install langsmith-tracing@langsmith-claude-code-plugins\n```\n\nWith both plugins installed, the evolution loop traces to LangSmith as a hierarchy: iteration → proposers → tool calls.\n\n---\n\n## References\n\n- [Meta-Harness: End-to-End Optimization of Model Harnesses](https://arxiv.org/abs/2603.28052) — Lee et al., 2026\n- [Self-Organizing LLM Agents Outperform Designed Structures](https://arxiv.org/abs/2603.28990) — Dochkina, 2026\n- [Hermes Agent Self-Evolution](https://github.com/NousResearch/hermes-agent-self-evolution) — NousResearch\n- [Agent Skills for Context Engineering](https://github.com/muratcankoylan/Agent-Skills-for-Context-Engineering) — Koylan\n- [A-Evolve: Automated Agent Evolution](https://github.com/A-EVO-Lab/a-evolve) — Amazon (5-stage evolution loop, git-tagged mutations)\n- [Meta Context Engineering via Agentic Skill Evolution](https://arxiv.org/abs/2601.21557) — Ye et al., Peking University, 2026\n- [EvoAgentX: Evolving Agentic Workflows](https://github.com/EvoAgentX/EvoAgentX) — Wang et al., 2026\n- [Darwin Godel Machine](https://sakana.ai/dgm/) — Sakana AI\n- [AlphaEvolve](https://deepmind.google/blog/alphaevolve/) — DeepMind\n- [LangSmith Evaluation](https://docs.smith.langchain.com/evaluation) — LangChain\n- [Harnessing Claude's Intelligence](https://claude.com/blog/harnessing-claudes-intelligence) — Martin, Anthropic, 2026\n- [Traces Start the Agent Improvement Loop](https://www.langchain.com/conceptual-guides/traces-start-agent-improvement-loop) — LangChain\n\n---\n\n## License\n\nMIT\n",
  "bytes": 7519,
  "sha": "5d29e9565559d18899b0d4fa7ab222de6ae989461c1dab3bfdb089c69cc70488",
  "repo_slug": "raphaelchristi/harness-evolver",
  "fonte": "repo",
  "truncated": false,
  "api": "https://agentalog.com/api/listings/plg_raphaelchristi_harness_evolver_harness_e_fda0e6e7/readme"
}