{ "markdown": "

\n \"DocPull\"\n

\n\n# DocPull\n\n\n\n**Active open-source project · MIT License**\n\nDocPull turns changing public web sources into cited, reproducible context for AI\nagents and retrieval pipelines. Use it when your application needs to know which\nsources it used, whether they changed, and how to rebuild the same context later.\n\n[![Python 3.10+](https://img.shields.io/badge/python-3.10+-blue.svg)](https://www.python.org/downloads/)\n[![PyPI version](https://img.shields.io/pypi/v/docpull.svg?label=package)](https://pypi.org/project/docpull/)\n[![License: MIT](https://img.shields.io/github/license/raintree-technology/docpull)](LICENSE)\n\n\n\n## Install and sync your first source\n\n```bash\npip install docpull\ndocpull init stripe-docs\ndocpull add https://docs.stripe.com\ndocpull sync\ndocpull diff\ndocpull export context-pack --target cursor\n```\n\nThe project stores declared sources in `docpull.yaml` and resolved inputs in\n`.docpull/context.lock.json`. Later syncs produce a hash-based diff while preserving\nsource URLs, content hashes, run IDs, citations, and export metadata.\n\n![DocPull project diff showing changed pages, local semantic categories, and zero failed URLs](docs/launch-assets/docpull-project-diff-demo.png)\n\nNo account or paid API is required for this path. Direct fetching, discovery,\nextraction, indexing, pack analysis, and diffs run locally.\n\n## Why use DocPull\n\n- **Reproduce agent context.** Stable IDs, hashes, manifests, and lockfiles show which\n source versions produced an answer or artifact.\n- **Detect source drift.** Sync and diff documentation, product pages, policies,\n feeds, repositories, packages, standards, and local documents.\n- **Keep evidence inspectable.** Markdown, NDJSON, SQLite, citations, and provenance\n sidecars remain readable without a hosted service.\n- **Choose the downstream surface.** Export context for agent clients, vector import,\n data workflows, or a versioned context-pack release.\n- **Keep expensive routes explicit.** Browser and cloud rendering require an explicit\n choice and can be blocked with a zero-dollar budget.\n\n## How it works\n\n```text\ndeclared sources → local acquisition → versioned evidence → diff and validation → export\n```\n\nDocPull's v3 pack contract separates raw extraction, agent-ready context, and\neval-grade evidence. Validate the level a downstream system requires:\n\n```bash\ndocpull pack prepare packs/docs --eval-grade\ndocpull pack validate packs/docs --level eval\ndocpull ci --prepare\n```\n\n`docpull ci` checks freshness, citation coverage, pack quality, rights metadata, and\nother configured gates. It writes `context-ci.report.json` and `CONTEXT_CI.md`, then\nexits non-zero when a hard gate fails.\n\n## Supported surfaces\n\n| Surface | Use it for | Start here |\n| --- | --- | --- |\n| CLI | Fetch, sync, diff, validate, and export | [`docs/cli-recipes.md`](docs/cli-recipes.md) |\n| Python SDK | Embed acquisition in Python applications | [`docs/surface-contract.md`](docs/surface-contract.md) |\n| MCP server | Give local agent clients source tools | [MCP server](#mcp-server) |\n| TypeScript SDK | Read local packs and invoke the CLI from Node or Bun | [`sdk/js/README.md`](sdk/js/README.md) |\n| Agent plugin | Install the supported MCP workflow in an agent client | [`plugin/README.md`](plugin/README.md) |\n\nCommon source shapes include static and server-rendered websites, OpenAPI documents,\nfeeds, papers, public GitHub repositories, npm and PyPI packages, standards, datasets,\ntranscripts, Wikimedia pages, product and policy pages, and local PDF or office files.\nSee [context-pack workflows](docs/context-packs.md) for the complete surface.\n\n## MCP server\n\n```bash\npip install 'docpull[mcp]'\ndocpull mcp\n```\n\nClaude Code can register the same local server:\n\n```bash\nclaude mcp add --transport stdio docpull -- docpull mcp\n```\n\nThe Python stdio server is the supported release path. The TypeScript code formerly\ndocumented under `mcp/` is an [internal semantic-search lab](docs/internal-mcp-lab.md),\nnot part of the package contract.\n\n## Limits and security boundary\n\nDocPull is an evidence-acquisition engine, not a hosted competitive-intelligence\nproduct. It owns fetching, explicit rendering adapters, versioning, citations,\nhashing, validation, replay, and export. Downstream products own scheduling, human\nreview, approved claims, legal conclusions, accounts, and notifications.\n\nThe default path does not handle complex interactive browser workflows, CAPTCHAs,\nstealth scraping, or private dashboards. JavaScript rendering is explicit. Authenticated\nsources require environment-variable references; DocPull does not persist credential\nvalues in project artifacts.\n\nSecurity defaults include HTTPS-only fetching, robots.txt compliance, SSRF and DNS\nrebinding protections, redirect guards, XXE protection, and path-traversal checks. Read\nthe [web-source boundary](docs/scraping-boundary.md), [security posture](docs/compliance.md),\nand [evidence-engine decision](docs/adr/0001-evidence-acquisition-engine.md) before\nextending acquisition behavior.\n\n## Documentation and evidence\n\n- [CLI recipes](docs/cli-recipes.md) — Common commands and advanced workflows.\n- [Context dependencies](docs/context-dependencies.md) — Project and lockfile model.\n- [Context Pack Contract v3](docs/context-pack-contract-v3.md) — Artifact levels and compatibility.\n- [Public contracts](docs/contracts.md) — Schemas and versioning rules.\n- [Alternatives](docs/alternatives.md) — Browser automation and hosted extraction tradeoffs.\n- [Evaluation lab](bench/docs/index.md) — Reproducible methods, results, and claim boundaries.\n- [Changelog](docs/CHANGELOG.md) — Release history.\n\n## Raintree open-source system\n\nDocPull owns evidence acquisition and reproducible agent context. It can be used\nindependently; the sibling projects do not imply a required integration or shared\nrelease cycle.\n\n| Project | Responsibility |\n| --- | --- |\n| [Raintree Standards](https://github.com/raintree-technology/raintree.standards) | Defines governed requirements and evidence. |\n| [Trellis](https://github.com/raintree-technology/trellis) | Enforces shared JavaScript and TypeScript code policy. |\n| [HIG Doctor](https://github.com/raintree-technology/hig-doctor) | Audits interface source and provides HIG guidance. |\n| [PolicyStrata](https://github.com/raintree-technology/policystrata) | Tests cross-layer policy behavior. |\n\nSee the [Raintree open-source portfolio](https://raintree.technology/portfolio#open-source)\nfor current lifecycle and distribution links.\n\n## Project policies\n\n[Contributing](CONTRIBUTING.md) · [Code of Conduct](CODE_OF_CONDUCT.md) · [Security](SECURITY.md) ·\n[Metrics and evidence limits](METRICS.md) · [Source repository](https://github.com/raintree-technology/docpull) ·\n[MIT License](LICENSE)\n", "bytes": 7025, "sha": "8246e7d88d2dd580a4d13bf25bcb5d5b903a2d3c68539d3efc71fdafa76986bf", "repo_slug": "raintree-technology/docpull", "fonte": "repo", "truncated": false, "api": "https://agentalog.com/api/listings/mcp_io_github_raintree_technology_docpull_376f75d3/readme" }