{
  "markdown": "<div align=\"center\">\n\n<img src=\"https://capsule-render.vercel.app/api?type=waving&color=0:1a1a2e,50:16213e,100:0f3460&height=220&section=header&text=robotstxt.ai&fontSize=48&fontColor=22c55e&fontAlignY=35&desc=Visual%20robots.txt%20Manager%20for%20AI%20Crawlers&descSize=16&descColor=ffffff&descAlignY=55&animation=fadeIn\" width=\"100%\" />\n\n<br />\n\n![npm](https://img.shields.io/npm/v/robotstxt-ai-mcp?style=flat-square&color=22c55e)\n![GitHub stars](https://img.shields.io/github/stars/sharozdawa/robotstxt-ai?style=flat-square&color=22c55e)\n![License](https://img.shields.io/github/license/sharozdawa/robotstxt-ai?style=flat-square)\n[![robotstxt-ai MCP server](https://glama.ai/mcp/servers/sharozdawa/robotstxt-ai/badges/score.svg)](https://glama.ai/mcp/servers/sharozdawa/robotstxt-ai)\n\n**Manage which AI bots can crawl your website — visually.**\n\nToggle GPTBot, ClaudeBot, PerplexityBot, and 20+ AI crawlers on/off with a simple UI. Analyze any site's robots.txt instantly.\n\n</div>\n\n---\n\n## Features\n\n- **Visual Toggle UI** — Block or allow AI bots with simple on/off switches\n- **20+ AI Bots Database** — GPTBot, ClaudeBot, Google-Extended, CCBot, Bytespider, Diffbot, cohere-ai, Amazonbot, Meta-ExternalAgent, and more\n- **Analyze Existing robots.txt** — Paste or fetch any robots.txt to see which AI bots are blocked\n- **Generate robots.txt** — Create a complete robots.txt with your chosen rules\n- **MCP Server** — Use with Claude Desktop, Cursor, or any MCP-compatible AI assistant\n- **Check Bot Status** — Verify if a specific bot is blocked on any website\n\n## MCP Tools\n\n| Tool | Description |\n|------|-------------|\n| `fetch_robots` | Fetch and analyze a robots.txt from any URL |\n| `analyze_robots` | Analyze pasted robots.txt content for AI bot blocking status |\n| `generate_robots` | Generate a robots.txt with specified blocked bots and custom rules |\n| `list_ai_bots` | List all known AI bots with user-agents, companies, and descriptions |\n| `check_bot_status` | Check if a specific bot is blocked on a given website |\n\n## Installation\n\n### Web App\n\n```bash\ngit clone https://github.com/sharozdawa/robotstxt-ai.git\ncd robotstxt-ai\nnpm install\nnpm run dev\n```\n\nOpen [http://localhost:3000](http://localhost:3000) in your browser.\n\n### MCP Server — Claude Desktop\n\nAdd to your `claude_desktop_config.json`:\n\n```json\n{\n  \"mcpServers\": {\n    \"robotstxt-ai\": {\n      \"command\": \"npx\",\n      \"args\": [\"-y\", \"robotstxt-ai-mcp\"]\n    }\n  }\n}\n```\n\n### MCP Server — Cursor\n\nAdd to `.cursor/mcp.json`:\n\n```json\n{\n  \"mcpServers\": {\n    \"robotstxt-ai\": {\n      \"command\": \"npx\",\n      \"args\": [\"-y\", \"robotstxt-ai-mcp\"]\n    }\n  }\n}\n```\n\n## Tracked Bots\n\nThe server knows about 25+ bots including:\n\n- **AI Crawlers**: GPTBot, ClaudeBot, Google-Extended, CCBot, Bytespider, Diffbot, cohere-ai, Amazonbot, Meta-ExternalAgent\n- **AI Search**: ChatGPT-User, OAI-SearchBot, PerplexityBot, YouBot\n- **Search Engines**: Googlebot, Bingbot, YandexBot, Baiduspider, DuckDuckBot\n\n## Why robotstxt.ai vs Manual Editing\n\n| Feature | robotstxt.ai | Manual Editing |\n|---------|-------------|----------------|\n| Visual toggle UI | Yes | No |\n| 20+ AI bots database | Yes | Research yourself |\n| Analyze existing robots.txt | Yes | No |\n| MCP Server | Yes | No |\n| Price | Free | Free but tedious |\n\n## More Open Source SEO Tools\n\n| Tool | Description |\n|------|-------------|\n| [awesome-seo-mcp-servers](https://github.com/sharozdawa/awesome-seo-mcp-servers) | Curated list of SEO MCP servers and agent skills |\n| [indexnow-mcp](https://github.com/sharozdawa/indexnow-mcp) | Instant URL indexing via IndexNow |\n| [schema-gen](https://github.com/sharozdawa/schema-gen) | Schema.org JSON-LD markup generator |\n| [ai-visibility](https://github.com/sharozdawa/ai-visibility) | AI brand visibility tracker |\n\n## License\n\nMIT\n\n---\n\n<div align=\"center\">\n\nBuilt by **[Sharoz Dawa](https://sharozdawa.com)** — SEO Professional & Digital Marketing Expert\n\n[![GitHub](https://img.shields.io/badge/GitHub-@sharozdawa-181717?style=flat-square&logo=github)](https://github.com/sharozdawa)\n\n<img src=\"https://capsule-render.vercel.app/api?type=waving&color=0:1a1a2e,50:16213e,100:0f3460&height=100&section=footer\" width=\"100%\" />\n\n</div>\n",
  "bytes": 4210,
  "sha": "9af599ba53a16dbb7907d14ba9fcc21b4167ccf3f30d3bea5d153a76ad16ce28",
  "repo_slug": "sharozdawa/robotstxt-ai",
  "fonte": "repo",
  "truncated": false,
  "api": "https://agentalog.com/api/listings/mcp_io_github_sharozdawa_robotstxt_ai_2d8da5e9/readme"
}