{
  "markdown": "# CrawlBit MCP\n\nFour free tools that answer one question: **can AI engines find, read and recognise a website?**\n\nNo account. No API key. No signup. Nothing here runs a language model, so nothing here costs you\nor us anything.\n\n```\n\"Can ChatGPT read stripe.com?\"\n\"Why does AI never mention my store?\"\n\"Where is my brand missing outside my own site?\"\n```\n\nAsk in plain language. Your AI client picks the right tool.\n\n---\n\n## Install\n\nRequires Node.js 18+.\n\n### Claude Code\n\n```bash\nclaude mcp add crawlbit -- npx -y crawlbit-mcp\n```\n\n### Claude Desktop\n\nAdd to `claude_desktop_config.json`:\n\n```json\n{\n  \"mcpServers\": {\n    \"crawlbit\": {\n      \"command\": \"npx\",\n      \"args\": [\"-y\", \"crawlbit-mcp\"]\n    }\n  }\n}\n```\n\n### Cursor\n\nAdd to `.cursor/mcp.json`:\n\n```json\n{\n  \"mcpServers\": {\n    \"crawlbit\": {\n      \"command\": \"npx\",\n      \"args\": [\"-y\", \"crawlbit-mcp\"]\n    }\n  }\n}\n```\n\nRestart your client. Four tools appear.\n\n---\n\n## The four tools\n\n| Tool | What it answers |\n|---|---|\n| `crawlbit_crawler_watch` | Can GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and Google-Extended read this site? |\n| `crawlbit_entity_check` | Does this brand read as a clear entity AI can recognise? |\n| `crawlbit_offpage_gaps` | Where is this brand missing **outside** its own website? |\n| `crawlbit_shopping` | Can AI shopping agents understand and recommend these products? |\n\n### Try these\n\n```\nCheck whether AI crawlers can read example.com\nDoes example.com read as a clear brand entity to AI?\nWhere is example.com missing off-page, and which gap should we close first?\nCompare example.com and competitor.com on AI crawler access\n```\n\nThe tools compose. A useful sequence is: crawler access first (a blocked crawler makes everything\nelse pointless), then the entity check, then the off-page gaps.\n\n---\n\n## What this does NOT do\n\nWritten plainly, because an SEO tool that overstates itself is worse than no tool.\n\n- **It does not test whether AI actually cites a brand.** That means running real buyer questions\n  against ChatGPT, Perplexity and Claude, reading the answers, and reporting who gets named\n  instead of you. It costs real money per run and is not part of this free surface. These four\n  tools measure the **conditions** that make a citation possible, which is a smaller claim.\n- **`crawler_watch` reads `robots.txt` only.** It cannot see blocking done at the CDN, firewall\n  or rate-limit layer. A site can pass here and still refuse the crawler in practice.\n- **`entity_check` reads what the site publishes about itself.** It says nothing about the\n  brand's reputation across the rest of the web.\n- **`shopping` inspects published structured data.** It cannot see a merchant feed submitted\n  privately to a platform.\n\nIf a site scores well on all four and still is not cited, the answer is almost always the same and\nit is not technical: **too few third-party pages mention it.** These tools will tell you that\nhonestly rather than sell you a fix that does not exist.\n\n---\n\n## Why only four tools\n\nCrawlBit also runs a full technical audit and generates `llms.txt`. Both use a language model,\nso both cost money per run, and neither is exposed here.\n\nThat is a deliberate line rather than a teaser. A free tool whose bill grows with its popularity\ngets rate-limited, degraded or withdrawn the moment it succeeds, and the people who installed it\nare the ones who pay for that. Everything in this server costs nothing to run, so nothing here\nhas to be taken back later. The audit and the `llms.txt` generator live at\n[crawlbit.app](https://www.crawlbit.app) if you want them.\n\n---\n\n## Rate limit\n\nThe API rate-limits per IP. Normal conversational use does not come close, and the server runs on\nyour machine, so the budget is yours rather than shared. If you do hit it, you get a plain\nsentence rather than a stack trace.\n\n---\n\n## Configuration\n\n| Variable | Default | Purpose |\n|---|---|---|\n| `CRAWLBIT_BASE_URL` | `https://www.crawlbit.app` | Point the server at another instance |\n\n---\n\n## Privacy\n\nThe server runs locally and stores nothing. The only data leaving your machine is the URL you\nask about, sent to the CrawlBit API to be analysed. There is no account, so there is nothing to\ntie a request to a person.\n\n---\n\n## About\n\nBuilt by [CrawlBit](https://www.crawlbit.app), which measures AI visibility and does the off-page\nwork that earns citations. The free tools here are the measurement half. The paid work is the\nother half, and you are not required to look at it to use this.\n\nMIT licensed. Issues and pull requests welcome.\n",
  "bytes": 4544,
  "sha": "f09f6a9275a51bfe97a188eb77861cf951db88df0af42cad4e63d32b5de2c52c",
  "repo_slug": "amati032-dev/crawlbit-mcp",
  "fonte": "repo",
  "truncated": false,
  "api": "https://agentalog.com/api/listings/mcp_io_github_amati032_dev_crawlbit_7691a906/readme"
}