{
"markdown": "\n
SlopCodeBench (SCBench)
\n\n [](https://www.python.org/downloads/)\n [](LICENSE)\n [](https://github.com/SprocketLab/slop-code-bench/stargazers)\n [](https://doi.org/10.5281/zenodo.19257129)\n\n [🌐 Website](https://www.scbench.ai) | [📄 Paper](https://arxiv.org/abs/2603.24755) | [📝 Blog Post](https://gabeorlanski.github.io/posts/slop-code-bench)\n\n\n\n---\n\n**SlopCodeBench** evaluates coding agents under iterative specification refinement: the agent implements a spec, then extends its own code as the spec changes. This exposes behaviors that single-shot benchmarks cannot measure, including path dependence, non-convergence, and trade-offs between explicit handling and structural stability. We release SCBench as an open, community-driven evaluation primitive rather than a finalized benchmark.\n\n\nProblem definitions now live in the separate [scb-problems repository](https://github.com/gabeorlanski/scb-problems) and are also available as a [Harbor dataset](https://registry.harborframework.com/datasets/gabeorlanski/slopcodebench/latest). We actively want more problems; follow [the creating a problem guide](/docs/contributing-problems/) and open a PR there.\n\n> [!NOTE]\n> This is an initial release. We're actively developing and welcome feedback via [GitHub Issues](https://github.com/SprocketLab/slop-code-bench/issues).\n\n## Prerequisites\n\nBefore installing, ensure you have:\n- **Python 3.12+** installed\n- **Docker** installed and running ([Get Docker](https://docs.docker.com/get-docker/))\n- An **API key** for your chosen agent (e.g., Anthropic, OpenAI, Google)\n- **8GB+ RAM** recommended for running evaluations\n- **10GB+ disk space** for Docker images and workspaces\n\n## 🚀 Install\n\n```bash\ncurl -LsSf https://astral.sh/uv/install.sh | sh\ngit clone https://github.com/SprocketLab/slop-code-bench.git && cd slop-code-bench && uv sync\nexport ANTHROPIC_API_KEY=\"your-key\"\n\n# Run!\nuv run slop-code run \\\n --agent claude_code \\\n --model anthropic/opus-4.5 \\\n --environment configs/environments/docker-python3.12-uv.yaml \\\n --prompt configs/prompts/just-solve.jinja \\\n --problem file_backup \\\n --problem execution_server \\\n thinking=low \\\n version=2.0.51\n```\n\n**Parameter Reference:**\n- `thinking=none|low|medium|high` - Controls extended thinking budget based on agent.\n- `version=X.Y.Z` - Agent version to use.\n\nResults are saved to:\n```\noutputs/opus-4.5/claude_code-just-solve_low_{timestamp}/\n```\n\n**First Run:** Docker images build automatically for that _VERSION_ of the agent (5-10 minutes). Subsequent runs are faster.\n\n### Troubleshooting\n\n**Docker not found:**\n```bash\n# Check Docker is running\ndocker ps\n# If not running, start Docker Desktop or daemon\n```\n\n**API key not found:**\n```bash\n# Verify your environment variable is set\necho $ANTHROPIC_API_KEY\n# Or pass it directly\nANTHROPIC_API_KEY=\"your-key\" uv run slop-code run ...\n```\n\n**Out of disk space:**\n```bash\n# Clean up old Docker images\ndocker system prune -a\n```\n\nFor more issues, see [GitHub Issues](https://github.com/SprocketLab/slop-code-bench/issues).\n\n## 📊 Evaluation\n\n**Evaluate a run:**\n```bash\nslop-code eval outputs/your-run-directory/\n```\n\n**Grade code quality with LLM judge:**\n```bash\nslop-code metrics judge \\\n --rubric configs/rubrics/llm_judge.jsonl \\\n --model \\\n --criteria-template configs/rubrics/templates/criteria_with_pn.j2 \\\n --prefix-template configs/rubrics/templates/no_expl.j2\n```\n\n## Contributing\n\nWe welcome contributions. Two ways to help:\n\n- **Add problems** — Expand the benchmark with new evaluation scenarios in the [scb-problems repository](https://github.com/gabeorlanski/scb-problems), also published as a [Harbor dataset](https://registry.harborframework.com/datasets/gabeorlanski/slopcodebench/latest). See the [Problem Tutorial](docs/problems/tutorial.md) and [Contributing Guide](CONTRIBUTING.md).\n- **Add agents** — Integrate new coding agents. See the [Agent Guide](docs/agents/README.md) and [Contributing Guide](CONTRIBUTING.md).\n\nThis is early-stage software. Your contributions will shape its direction.\n\n## Documentation\n\n| Guide | Description |\n|-------|-------------|\n| [❓ FAQ](docs/FAQ.md) | Frequently asked questions |\n| [📖 Problem Tutorial](docs/problems/tutorial.md) | Create your first problem (30 min hands-on) |\n| [📋 Quick Reference](docs/problems/quick-reference.md) | One-page cheat sheet for problem authoring |\n| [🤖 Agent Guide](docs/agents/README.md) | Configure agents, models, and credentials |\n| [🏗️ Architecture](docs/execution/README.md) | How sessions, workspaces, and runtimes work |\n| [✅ Evaluation System](docs/evaluation/README.md) | Test cases, adapters, loaders, and verifiers |\n| [💡 Problem Design](docs/contributing-problems/README.md) | What makes a good evaluation problem |\n| [⚠️ Known Issues](docs/KNOWN_ISSUES.md) | Current limitations and workarounds |\n| [📊 Commands](docs/commands/README.md) | CLI command reference (run, eval, metrics, viz, etc.) |\n\n## Citing Us\n\nIf you found this useful, please cite us as:\n```bibtex\n@article{Orlanski2025SlopCodeBench,\n author = {Orlanski, Gabriel and Roy, Devjeet and Yun, Alexander and Shin, Changho and Gu, Alex and Ge, Albert and Adila, Dyah and Albarghouthi, Aws and Sala, Frederic},\n title = {{SlopCodeBench: Measuring Code Erosion Under Iterative Specification Refinement}},\n journal = {arXiv preprint arXiv:2603.24755},\n year = {2025},\n url = {https://arxiv.org/abs/2603.24755}\n}\n```\n",
"bytes": 5760,
"sha": "ef50ba6c40896df260cd3011865d8c9bfb0427377c9d55acf2f3a7c096a14cb1",
"repo_slug": "sprocketlab/slop-code-bench",
"fonte": "repo",
"truncated": false,
"api": "https://agentalog.com/api/listings/skl_sprocketlab_slop_code_bench_codex_skills_691b560b/readme"
}