{
  "markdown": "# 🔥 PyreCrawl — Web Browsing Superpowers for Your AI Agent\n\n[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)\n[![MCP](https://img.shields.io/badge/MCP-1.0-blue.svg)](https://modelcontextprotocol.io/)\n[![Python 3.10+](https://img.shields.io/badge/python-3.10+-blue.svg)](https://www.python.org/)\n[![PyPI](https://img.shields.io/pypi/v/pyrecrawl.svg)](https://pypi.org/project/pyrecrawl/)\n\n**One command gives any AI agent the whole web.** Scrape, extract, crawl, map, and search —\nself-hosted, no API keys, no rate limits, no subscription.\n\nPyreCrawl speaks **MCP** (Model Context Protocol), the standard tool interface for Claude,\nCursor, VS Code, Codex, OpenCode, Hermes, and any MCP-compatible agent.\n\nA **smart auto-fallback ladder** always picks the cheapest method that succeeds:\n\n```\nfast HTTP\n    │  (403/503/Cloudflare challenge or empty body)\n    ▼\nstealth browser (real Chromium + Cloudflare solver)\n    │  (still blocked, or the page needs full JS rendering)\n    ▼\ndeep processing (LLM-ready markdown, citations, structured extraction)\n```\n\n## ⚡ Tools exposed\n\n| Tool | What it does |\n|---|---|\n| `scrape(url, prefer=\"auto\")` | Single URL → LLM-ready markdown |\n| `extract(url, schema)` | Scrape + structured extraction (JsonCss schema) |\n| `map_site(root, include_pattern=None, limit=200)` | Enumerate all internal URLs |\n| `crawl(root, max_pages=5, prefer=\"auto\")` | Multi-page crawl with auto-fallback per page |\n| `search(query, limit=10)` | Web search via DuckDuckGo HTML (no API key) |\n| `health()` | Versions + import sanity check |\n\n`prefer` options: `\"auto\"` (default ladder) · `\"fast\"` (HTTP only) · `\"stealth\"` (CF bypass) · `\"llm\"` (deep processing).\n\n---\n\n## 🚀 Install & Use (one-liner)\n\n### 1. Install\n\n```bash\n# Using uv (recommended — fast, isolated, no venv needed)\nuv tool install pyrecrawl\n\n# Or pipx (alternative)\npipx install pyrecrawl\n\n# Or pip into a venv\npip install pyrecrawl\n```\n\n### 2. One-time browser engines\n\n```bash\npyrecrawl setup\n```\n\nThis installs Chromium + stealth browser engines (~2 min, one-time).\n\n### 3. Register with your AI agent\n\n```bash\n# Auto-detect installed agents and write their MCP configs\npyrecrawl install\n\n# Or target specific agents\npyrecrawl install claude-desktop cursor\n\n# Dry-run to preview what would change\npyrecrawl install --dry-run\n```\n\nSupported agents: `claude-desktop`, `claude-code`, `cursor`, `vscode`, `codex`, `opencode`, `hermes`.\n\n### 4. Start chatting\n\nAfter installing + registering, **restart your agent** (or start a new session). Then ask:\n\n> *\"Scrape https://example.com and summarize it.\"*\n\nThe tools appear as `mcp_pyrecrawl_scrape`, `mcp_pyrecrawl_extract`, `mcp_pyrecrawl_map_site`, `mcp_pyrecrawl_crawl`, `mcp_pyrecrawl_search`, `mcp_pyrecrawl_health`.\n\n---\n\n## 📚 Manual config (if `pyrecrawl install` doesn't match your setup)\n\n### Claude Desktop\n\n**Config file**\n- Linux: `~/.config/Claude/claude_desktop_config.json`\n- macOS: `~/Library/Application Support/Claude/claude_desktop_config.json`\n- Windows: `%AppData%\\Claude\\claude_desktop_config.json`\n\n```json\n{\n  \"mcpServers\": {\n    \"pyrecrawl\": {\n      \"command\": \"uvx\",\n      \"args\": [\"--from\", \"pyrecrawl\", \"pyrecrawl\", \"serve\"]\n    }\n  }\n}\n```\n\n### Claude Code\n\n**Config file**: project-scoped `.mcp.json`\n\n```json\n{\n  \"mcpServers\": {\n    \"pyrecrawl\": {\n      \"command\": \"uvx\",\n      \"args\": [\"--from\", \"pyrecrawl\", \"pyrecrawl\", \"serve\"]\n    }\n  }\n}\n```\n\n### Cursor\n\n**Config file**: `~/.cursor/mcp.json`\n\n```json\n{\n  \"mcpServers\": {\n    \"pyrecrawl\": {\n      \"command\": \"uvx\",\n      \"args\": [\"--from\", \"pyrecrawl\", \"pyrecrawl\", \"serve\"]\n    }\n  }\n}\n```\n\n### VS Code / Copilot\n\n**Config file**: `.vscode/mcp.json` (project-scoped)\n\n```json\n{\n  \"servers\": {\n    \"pyrecrawl\": {\n      \"command\": \"uvx\",\n      \"args\": [\"--from\", \"pyrecrawl\", \"pyrecrawl\", \"serve\"],\n      \"type\": \"stdio\"\n    }\n  }\n}\n```\n\n### Codex CLI\n\n**Config file**: `~/.codex/config.toml`\n\n```toml\n[mcp_servers.pyrecrawl]\ncommand = \"uvx\"\nargs = [\"--from\", \"pyrecrawl\", \"pyrecrawl\", \"serve\"]\n```\n\n### OpenCode\n\n**Config file**: `~/.config/opencode/opencode.json`\n\n```json\n{\n  \"mcp\": {\n    \"pyrecrawl\": {\n      \"type\": \"local\",\n      \"command\": [\"uvx\", \"--from\", \"pyrecrawl\", \"pyrecrawl\", \"serve\"],\n      \"enabled\": true\n    }\n  }\n}\n```\n\n### Hermes\n\n**Config file**\n- Linux/macOS: `~/.hermes/config.yaml`\n- Windows: `%LocalAppData%\\hermes\\config.yaml`\n\n```yaml\nmcp_servers:\n  pyrecrawl:\n    command: uvx\n    args:\n      - --from\n      - pyrecrawl\n      - pyrecrawl\n      - serve\n    enabled: true\n```\n\n> **Windows note:** `uvx` must be on PATH. If not, use the full path to `uvx.exe` (e.g. `C:\\Users\\<you>\\AppData\\Local\\hermes\\bin\\uvx.exe`).\n\n---\n\n## 🧠 How the ladder chooses\n\nPyreCrawl runs each request through three tiers, stopping at the first one that returns\na complete, LLM-ready result:\n\n| Concern | Fast tier | Stealth tier | Deep tier |\n|---|---|---|---|\n| Static HTML page | ✅ ~200ms | — | — |\n| Cloudflare-protected | ❌ | ✅ Turnstile solver | — |\n| JS-heavy SPA | ❌ | ✅ real Chromium | — |\n| Live DOM data (input `.value`, JS state) | ❌ | ✅ `js` param | — |\n| LLM-ready markdown + citations | — | — | ✅ BM25, fit-markdown |\n| Structured extraction (CSS schema) | — | — | ✅ |\n| Deep crawl (BFS/DFS/BestFirst) | — | — | ✅ adaptive |\n\nThe agent never has to pick. `prefer=\"auto\"` does it every call.\n\n### Live DOM data with `js` and `wait_for`\n\nSome sites keep the data you want in a DOM *property* (e.g. an `<input>`'s `.value`)\nthat JS writes after an XHR — it never appears in the serialized HTML. The\n`scrape` tool accepts two stealth-tier params for exactly this:\n\n```json\n{\n  \"url\": \"https://temp-mail.org/id\",\n  \"prefer\": \"stealth\",\n  \"wait_for\": \"document.getElementById('mail').value.includes('@')\",\n  \"js\": \"document.getElementById('mail').value\"\n}\n```\n\n- `wait_for` — a JS **predicate expression** polled until truthy (bounded by `timeout`).\n  Use it instead of guessing a sleep for anything that arrives asynchronously.\n- `js` — a JS **expression** evaluated once the page settles; the value comes back\n  in `meta.js_result`. Errors are captured in `meta.js_error` (the page result is\n  still returned, never a crash).\n\n---\n\n## 📊 Compared to Firecrawl (hosted)\n\n| | Firecrawl | PyreCrawl |\n|---|---|---|\n| Cost | Free 1k/mo, then $16–333/mo | **Free, self-hosted** |\n| Local LLM support | ❌ | ✅ Ollama / any LLM |\n| Cloudflare bypass | ✅ (Fire-Engine, paid) | ✅ (free, built-in) |\n| Markdown + BM25 | ✅ | ✅ |\n| Self-host | ❌ | ✅ |\n| Hosted search API | ✅ /search | ⚠️ DuckDuckGo HTML (no key) |\n\n---\n\n## 🔧 Development\n\n```bash\ngit clone https://github.com/SanggonBoy/PyreCrawl.git\ncd PyreCrawl\nuv venv --python 3.12 .venv\nsource .venv/Scripts/activate  # Windows; or .venv/bin/activate on macOS/Linux\nuv pip install -e \".[dev]\"\npython -m playwright install chromium\nscrapling install\n```\n\n### Run tests\n\n```bash\npython scripts/selfcheck.py   # real-network smoke test\npython scripts/probe_stdio.py # stdio JSON-RPC probe\n```\n\n---\n\n## 📦 Publish\n\nMaintainers only:\n\n```bash\ngit tag v0.2.1\ngit push origin v0.2.1\n```\n\nGitHub Actions builds + uploads to PyPI via [trusted publishing](https://docs.pypi.org/trusted-publishers/).\n\n---\n\n## 📜 Uninstall\n\n```bash\n# Remove from all agent configs\npyrecrawl uninstall\n\n# Remove the package\nuv tool uninstall pyrecrawl\n```\n\n---\n\n## 🛡️ License\n\nMIT — see [LICENSE](LICENSE).\n\n## 🙏 Credits\n\nBuilt on the shoulders of [Scrapling](https://github.com/D4Vinci/Scrapling) and\n[Crawl4AI](https://github.com/unclecode/crawl4ai) — both MIT, both excellent.\n\n<!-- mcp-name: io.github.SanggonBoy/PyreCrawl -->\n",
  "bytes": 7638,
  "sha": "3da1f23efac90cbfd90f2e93feb2ac54b88bbe4ed4b31ee5509d1513f11c6f46",
  "repo_slug": "sanggonboy/pyrecrawl",
  "fonte": "repo",
  "truncated": false,
  "api": "https://agentalog.com/api/listings/mcp_io_github_sanggonboy_pyrecrawl_6dba1a25/readme"
}