{
  "markdown": "# CrawlEyes — Web Scraping & Search Toolkit for AI Agents\n\n<!-- MCP Registry metadata: associates this PyPI package with the MCP registry entry.\n     Name must match the mcpName in the registry listing. -->\n<!-- mcp-name: io.github.waiky-github/CrawlEyes -->\n\n[![GitHub stars](https://img.shields.io/github/stars/waiky-github/CrawlEyes)](https://github.com/waiky-github/CrawlEyes/stargazers)\n[![GitHub license](https://img.shields.io/github/license/waiky-github/CrawlEyes)](https://github.com/waiky-github/CrawlEyes/blob/main/LICENSE)\n[![Python](https://img.shields.io/badge/Python-3.10%2B-blue)](https://www.python.org/)\n[![MCP](https://img.shields.io/badge/MCP-server-7B68EE)](https://modelcontextprotocol.io)\n[![MIT](https://img.shields.io/badge/License-MIT-green)](LICENSE)\n\n**CrawlEyes** gives AI agents reliable **full-text extraction** (`web_extract`) and **robust search** (`web_search`) backends — the \"eyes\" that let agents see and read the web. Built and tested against [Hermes Agent](https://hermes-agent.nousresearch.com/docs).\n\nAlso ships as a **standard MCP server**, so any MCP client (Claude Desktop, Cursor, other agents) can reuse the same search + extraction capabilities.\n\n![CrawlEyes demo](assets/demo.png)\n\n## Why CrawlEyes?\n\nMost agent toolkits cover *one* slice of the pipeline. CrawlEyes is the rare **all-in-one** that you can actually run behind the Great Firewall without external accounts.\n\n| | Typical agent toolkit | **CrawlEyes** |\n|:--|:--|:--|\n| 🔍 **Search** | API key required, often blocked in CN | ✅ SearXNG (self-hosted) + **Tavily keyless fallback** — zero config, zero key |\n| 📄 **Extraction** | Separate scraper, or Firecrawl SaaS | ✅ Built-in Crawl4AI full-text extract, ~89% noise removal |\n| 🧠 **Semantic rerank** | Rarely included | ✅ Local fastembed rerank — no torch, ~50MB model |\n| 🔌 **MCP server** | Often missing | ✅ Standard MCP tools (`search` + `extract` + `deep_research` + `sitemap`), any client |\n| 🌐 **China-friendly** | Mostly English/GFW-blocked | ✅ Tested on a real mainland China server (baidu + yandex) |\n\n> **Zero API keys. Zero external accounts. One command.** CrawlEyes is the only toolkit in this space that combines search + extraction + semantic reranking + MCP in a single, China-friendly, self-hosted package.\n\n## Features\n\n| Capability | Where | Why it matters |\n|:--|:--|:--|\n| **Full-text extraction** | `scripts/crawl4ai_cli.py` | Headless-browser scraping → clean Markdown; handles ~80% of JS/dynamic/UA-blocked pages |\n| **Content denoising** (P1) | `crawl4ai_cli.py --noise-filter` | Prunes nav/ads/comments via Crawl4AI's `PruningContentFilter` — measured **24.6k→2.8k chars (~89% noise removed)** on a typical article |\n| **Retry with backoff** (P3) | `crawl4ai_cli.py --retry N` | Exponential backoff (1s/2s/4s) on transient failures |\n| **Browser session reuse** (P4) | `crawl4ai_cli.py --session NAME` | Reuses the browser context across scrapes in one process — no cold-start per URL |\n| **Keyword-focused extraction** | `crawl4ai_cli.py --bm25 KEYWORD` | Keeps only paragraphs relevant to a keyword (experimental — BM25 is English-centric; works best on English docs) |\n| **Search (primary)** | SearXNG (self-hosted meta-search) | Privacy-friendly search aggregator |\n| **Search (fallback)** | Tavily keyless API | Zero-config, no-key fallback when SearXNG is down/empty |\n| **Search orchestration** | `plugins/searxng-tavily/` | Hermes plugin provider: SearXNG first → auto-fallback to Tavily keyless; three-state circuit breaker (3 fails → 60s cooldown → half-open) + shared SQLite cache (TTL 3600s) |\n| **Semantic reranking** (P2) | `scripts/crawl_search_standalone.py` | Local embedding rerank of search results with `fastembed` + `BAAI/bge-small-zh-v1.5` (512-dim, **no torch dependency**, ~50MB, cached) — puts relevant results first. Measured: crawler-relevant items 0.817/0.732 float to top, irrelevant 0.302/0.139 sink |\n| **MCP server** (P5) | `scripts/mcp_crawl_server.py` | Exposes `search` + `extract` + `deep_research` + `sitemap` as standard MCP tools (**stdio default, or streamable-http** for remote clients). Works in *any* MCP client, no Hermes dependency. Extracted content is sanitized against prompt-injection (strips invisible chars + prompt-hijack lines). **Unified rate limiting + exponential backoff** guard every tool (sliding window, per-tool cost) so concurrent agent calls can't hammer downstream services |\n| **Sitemap discovery** (P1) | `crawleyes/sitemap.py` | `sitemap(origin)` → parses `sitemap.xml` (plain / gzip / index-recursion) with `robots.txt` fallback, returns a deduped URL map. Zero-key way to discover a site's URL surface for whole-site fetch or deep-research seeding |\n| **Multi-format extract** (P0) | `extract(..., format=)` | `markdown` (default) / `fit` (denoised) / `raw` (unfiltered) / `markdown_with_citations` — pick the level of cleanup you need |\n| **RAG-ready interfaces** | `crawleyes/rag.py` | One-liners `markdown(url)` / `search_markdown(query)` → clean, sanitized, LLM-ready Markdown for RAG corpora |\n| **Deep research** | `crawleyes/deep_research.py` | `deep_research(topic)` → decomposes topic into sub-questions → searches → extracts → synthesizes a **cited Markdown report**. Optional LLM (any OpenAI-compatible endpoint); degrades to evidence-aggregate mode without one |\n| **Verification** | `scripts/` | Clean subprocess scripts to verify each backend end-to-end per Hermes profile |\n\n## Project layout\n\n```\nplugins/searxng-tavily/   Hermes web-search provider plugin (SearXNG → Tavily keyless fallback)\n                          + three-state circuit breaker + shared SQLite cache\nscripts/\n  crawl4ai_cli.py          Universal scraping CLI (URL → Markdown), with denoise/retry/session/BM25\n  crawl_search_standalone.py  Standalone search (SearXNG → Tavily) + optional semantic rerank.\n                             No Hermes dependency — usable anywhere, powers the MCP server.\n  mcp_crawl_server.py      Standard MCP server exposing search + extract + deep_research + sitemap (stdio)\n  single_env_check.py      Verify crawl4ai provider registered+available+extracts (one profile)\n  verify_searxng_tavily.py Verify searxng-tavily provider: normal path + forced fallback\n  agent_link_check.py      Verify full agent tool chain: web_search_tool dispatch + logs\n```\n\n## Quick start\n\n### 1. Install Crawl4AI (China-friendly mirrors)\n\n```bash\npython3 -m venv .venv\n# Use Tsinghua PyPI mirror for speed (or any mirror you prefer)\n.venv/bin/pip install -i https://pypi.tuna.tsinghua.edu.cn/simple crawl4ai\n# Playwright browser kernel — use npmmirror binary mirror if cdn.playwright.dev is blocked\nPLAYWRIGHT_DOWNLOAD_HOST=https://registry.npmmirror.com/-/binary/playwright \\\n  .venv/bin/python -m playwright install chromium\n.venv/bin/crawl4ai-setup\n```\n\n### 2. Scrape a page\n\n```bash\n.venv/bin/python scripts/crawl4ai_cli.py https://example.com          # stdout Markdown\n.venv/bin/python scripts/crawl4ai_cli.py https://example.com -o out.md  # to file\n.venv/bin/python scripts/crawl4ai_cli.py URL --text --max-words 5000   # plain text, truncated\n\n# Multi-format extraction (markdown|fit|raw|markdown_with_citations)\n.venv/bin/python scripts/crawl4ai_cli.py URL --format raw              # unfiltered source markdown\n.venv/bin/python scripts/crawl4ai_cli.py URL --format markdown_with_citations  # + source URLs\n\n# Respect robots.txt (opt-in, default off)\n.venv/bin/python scripts/crawl4ai_cli.py URL --respect-robots\n\n# Denoise nav/ads + retry 3x + reuse session across scrapes\n.venv/bin/python scripts/crawl4ai_cli.py URL --noise-filter --retry 3 --session s1\n```\n\n### 3. Use the search + rerank (standalone, no Hermes)\n\n```bash\n# Optional: local semantic rerank of results (fastembed + bge-small-zh, auto-downloaded)\n.venv/bin/pip install -i https://pypi.tuna.tsinghua.edu.cn/simple fastembed\n\n# SearXNG first, Tavily keyless fallback, then rerank\nSEARXNG_URL=https://your-searxng .venv/bin/python -c \"\nimport sys; sys.path.insert(0, 'scripts')\nfrom crawl_search_standalone import CrawlSearch\nr = CrawlSearch(rerank=True).search('your query')\nprint(r['data']['web'])\"\n```\n\n> China-network note: the embedding model downloads from HuggingFace, which is blocked on mainland networks. Set `HF_ENDPOINT=https://hf-mirror.com` and `HF_HUB_DISABLE_XET=1` (hf-mirror doesn't support the xet protocol and returns 401 without this).\n\n### 4. Run as an MCP server (any client)\n\n```bash\n# Any MCP client can connect via stdio (default):\n.venv/bin/python scripts/mcp_crawl_server.py\n# Exposes tools:\n#   search(query, limit)                 - SearXNG → Tavily keyless, rerank, retry+rate-limit\n#   extract(url, max_words, format)      - markdown|fit|raw|markdown_with_citations\n#   deep_research(topic, num_questions)  - multi-round cited report\n#   sitemap(origin, max_urls)            - URL map from sitemap.xml / robots.txt\n\n# Or serve over HTTP (streamable-http) for remote clients:\n.venv/bin/python -m crawleyes.mcp_crawl_server --transport http --port 8765 --host 127.0.0.1\n#   → clients connect to http://127.0.0.1:8765/mcp\n#   (host/port configurable; default 127.0.0.1:8765)\n```\n\nFor Hermes specifically, add to `config.yaml`:\n\n```yaml\nmcp_servers:\n  crawl:\n    command: \"/path/to/crawl/.venv/bin/python\"\n    args: [\"/path/to/crawl/scripts/mcp_crawl_server.py\"]\n    timeout: 90\n    connect_timeout: 60\n```\n\n### 4b. Firecrawl-compatible `/scrape` endpoint\n\nAlready using Firecrawl's Python SDK? Point it at CrawlEyes and keep your code:\n\n```bash\n.venv/bin/python -m crawleyes.firecrawl_api --port 8899 --host 127.0.0.1\n#   POST /v2/scrape  →  { success, data: { markdown, metadata } }\n#   GET  /healthz    →  health check\n```\n\n```python\nfrom firecrawl import Firecrawl\nfc = Firecrawl(api_url=\"http://127.0.0.1:8899\", api_key=\"ignored\")\ndoc = fc.scrape(url=\"https://example.com\")   # → { markdown, metadata }\n```\n\nThis is a pragmatic subset of the Firecrawl API — the core `/scrape` contract\n(`success` + `data.markdown` + `data.metadata`), backed by CrawlEyes' own\nextraction engine. It does **not** implement Firecrawl's async `/crawl` queue,\n`/search`, or `/map` — see the design notes for the rationale.\n\n### 5. Install the search plugin (Hermes)\n\nCopy `plugins/searxng-tavily/` into a Hermes plugins dir, then:\n\n```bash\nhermes plugins enable web/searxng-tavily\nhermes config set web.search_backend searxng-tavily\n```\n\nSet `SEARXNG_URL` in your Hermes profile `.env` to point at your SearXNG instance. If unset or unreachable, the provider automatically falls back to the **Tavily keyless API** (no API key required).\n\n> Note: the plugin only takes effect for **newly started** agent sessions.\n\n### 6. Verify\n\n```bash\n# Requires the Hermes source tree + its venv\nvenv/bin/python scripts/verify_searxng_tavily.py $HERMES_HOME\nvenv/bin/python scripts/agent_link_check.py $HERMES_HOME\n```\n\n## Design notes\n\n- **Layered composition**: no single tool covers everything. Crawl4AI handles extraction; SearXNG + Tavily cover search; each layer has a tested fallback.\n- **Tavily keyless** works with zero configuration and no account — a cheap resilience net for the whole search path.\n- **Circuit breaker is SearXNG-only**: a Tavily fallback success does *not* reset the breaker (otherwise it would never trip). `record_success()` is only called when SearXNG itself succeeds.\n- **Shared SQLite cache** lives in the *real* user home (via `pwd.getpwuid`, not `$HOME` — which Hermes profiles override), so all profiles share one cache. WAL + 5s timeout + try/except degrade-to-no-cache under concurrency.\n- **Semantic rerank is cheap**: fastembed (ONNX) avoids the ~2GB torch dependency; model loads in ~0.6s once cached, embeddings in ~50ms.\n- **MCP server is standalone**: it does *not* import Hermes internals, so it runs on any Python 3.12 env and serves any MCP client.\n- **MCP transport is dual**: stdio (default, standard MCP clients) or `streamable-http` (`--transport http`), so a single codebase serves both local process and remote HTTP clients.\n- **Unified rate limiting is layered**: MCP tools *and* deep-research's internal search/extract all share one sliding-window limiter (per-tool cost), so concurrent agent fan-out can't hammer SearXNG/Tavily/Crawl4AI even through multi-round deep research.\n\n## Credits & inspiration\n\nThis project builds on a set of excellent open-source tools. All code here is an independent implementation (no copied code), but the *ideas* and *interfaces* are drawn from the following projects — full credit to their authors:\n\n| Feature in this repo | Inspired by | License |\n|:--|:--|:--|\n| Extraction engine (Crawl4AI wrapper) | [Crawl4AI](https://github.com/unclecode/crawl4ai) — direct dependency | Apache-2.0 |\n| Content denoising (P1) | [Readability](https://github.com/mozilla/readability), [GeneralNewsExtractor](https://github.com/kingname/GeneralNewsExtractor) (idea) | Apache-2.0 / MIT |\n| Semantic reranking (P2) | [Vane](https://github.com/reflex-dev/vane), [Perplexica](https://github.com/ItzCrazyKns/Perplexica) (idea) | MIT / MIT |\n| Retry with backoff (P3) | [Crawlee](https://github.com/apify/crawlee) (idea) | Apache-2.0 |\n| Browser session reuse (P4) | [camoufox](https://github.com/daijro/camoufox) (idea) | MIT |\n| MCP server (P5) | [playwright-mcp](https://github.com/microsoft/playwright-mcp), [exa-mcp-server](https://github.com/exa-labs/exa-mcp-server) (idea) | Apache-2.0 / MIT |\n| Search orchestration / fallback | [SearXNG](https://github.com/searxng/searxng) — self-hosted (official Docker image, no source modification), accessed via HTTP API only · [Tavily keyless](https://tavily.com) | AGPL-3.0 (server software, not linked/embedded) / proprietary API |\n\n> **Design independence**: the implementations here are written from scratch — we studied the above projects' *approaches* (denoising thresholds, rerank pipelines, backoff strategies, MCP tool patterns) but did not copy their source code. Dependencies are declared in [`requirements.txt`](requirements.txt). If you believe any attribution is missing or incorrect, please open an issue.\n\n## Compliance\n\nCrawlEyes is a general-purpose fetch toolkit for legitimate research and personal use. It deliberately does **not** include proxy pools, fingerprint rotation, or CAPTCHA-solving (anti-scraping evasion) — those are out of scope.\n\n**Robots.txt is opt-in** (default off): pass `respect_robots=True` to `extract` / `markdown` (or `--respect-robots` on the CLI) to check each target's `robots.txt` (RFC 9309) and refuse URLs it explicitly disallows. It's default-off so legitimate scraping isn't silently blocked by aggressive or broken robots rules — compliance is the caller's informed choice per use case. Always review each site's terms of service before scraping at scale.\n\n## License\n\nMIT — see [LICENSE](LICENSE).\n\n",
  "bytes": 14800,
  "sha": "14744fd3d4ab14fae81116a2dbd219e859b425bc4b7cac6ff3032d51a3f97ff7",
  "repo_slug": "waiky-github/crawleyes",
  "fonte": "repo",
  "truncated": false,
  "api": "https://agentalog.com/api/listings/mcp_io_github_waiky_github_crawleyes_8637d949/readme"
}