io.github.jmshinhwa/robots-txt-ai-crawler-audit
Checks robots.txt line by line and says what each AI crawler token actually controls - training, AI
Open source Open in the app JSON
About
Checks robots.txt line by line and says what each AI crawler token actually controls - training, AI
Details
- Kind
- MCP servers
- Topic
- Web search, scraping & browser
- Publisher
- jmshinhwa
- Origin
- official
- Category
- ferramentas
- Transport
- local
- Version
- 0.1.1
- Added
- 2026-09-13 12:00:58
- Updated
- 2026-09-13 12:00:58
- Origin id
io.github.jmshinhwa/robots-txt-ai-crawler-audit