{
  "markdown": "# aivis — can AI assistants actually read your store?\n\n**One command. No install, no dependencies, no account, nothing uploaded.**\n\n```bash\ncurl -sO https://raw.githubusercontent.com/krisdiallo/ecom-agent/main/aivis.py\npython3 aivis.py yourstore.com\n```\n\n```\naivis 1.0.0 — can AI assistants read brooklinen.com?\n\n1. robots.txt — are you blocking the crawlers that recommend you?\n  [ OK ] 4 search crawler(s) can read your catalog\n         OAI-SearchBot PerplexityBot Claude-SearchBot Claude-User\n\n2. Product page — can a crawler read your facts?\n  [FAIL] Structured data is injected by JavaScript — crawlers never see it\n  [FAIL] Your <title> contradicts your own og:title\n         title:    Classic Cotton Sheet Set | Brooklinen\n         og:title: Super-Plush 4-Piece Bath Towel Set\n  [WARN] Only 0 concrete measurement(s) in the readable text\n\nSummary: 2 critical  2 to review  1 passed\n```\n\nThat output is real, from a live page on a major DTC brand, reproduced across three\nindependent fetches. Prefer a browser? Same checks, paste-based:\n**[the web version](https://krisdiallo.github.io/ecom-agent/ai-visibility.html)**.\n\nOne page is an anecdote. Sample your catalogue to see whether a problem is systemic:\n\n```bash\npython3 aivis.py yourstore.com --pages 10\n```\n\n```\n[FAIL] Structured data injected by JavaScript (7/10 pages)\n[WARN] Fewer than 5 concrete measurements (9/10 pages)\n[ OK ] 1/10 pages clean on every check\n```\n\n---\n\n## What this is, in the words you'd search for it\n\nThis is a free **GEO tool** — **generative engine optimization**, also called **LLM SEO**\nor **answer engine optimization**: making sure an AI assistant can read your store well\nenough to cite or recommend it. It is the ecommerce-specific version of the broader\n**AI search optimization** question, focused on the two things that decide whether an\nanswer engine can use your content at all:\n\n1. **Your robots.txt** — are you disallowing the crawlers that feed answers\n   (`OAI-SearchBot`, `PerplexityBot`, `Claude-SearchBot`)? Most \"block AI\" advice gets this\n   wrong by conflating them with training crawlers.\n2. **Your product page's structured data and facts in raw HTML** — what a crawler that does\n   not run JavaScript actually receives, not what dev tools show you.\n\nIf you searched for an **AI visibility checker** or a **robots.txt checker** for an\necommerce store, this does both, with primary-source data on 165 AI crawlers rather than\ngeneric advice.\n\n---\n\n## Or give it to your AI\n\nAn MCP server, so Claude Code, Claude Desktop, Cursor or any MCP client can run the check\nitself — \"is my store readable by AI assistants?\" answered in the conversation you're already in.\n\nNothing to download, no paths to edit — paste this into your MCP config:\n\n```json\n{ \"mcpServers\": {\n    \"aivis\": {\n      \"command\": \"uvx\",\n      \"args\": [\"--from\", \"git+https://github.com/krisdiallo/ecom-agent\", \"aivis-mcp\"]\n    }\n} }\n```\n\nAlso listed in the **[official MCP registry](https://registry.modelcontextprotocol.io)** as\n`io.github.krisdiallo/aivis`, so any client that reads the registry can find it, and there is a\none-file `aivis_mcp.py` in this repo if you would rather run it directly.\n\nStdio JSON-RPC, **zero dependencies** — the whole tool is standard library, which is why it can\nbe fetched and run straight from git. It imports `aivis.py` rather than shelling out, so the CLI,\nweb tool, study, GitHub Action and MCP server all share one verified implementation.\n\n## Or run it in CI\n\nCatch it on the deploy that breaks it, not months later when you wonder why an assistant\nnever mentions you:\n\n```yaml\n- uses: krisdiallo/ecom-agent@v1\n  with:\n    site: yourstore.com\n    fail-on: critical        # critical | warning | never\n```\n\nWrites a findings table to the job summary, exposes `critical` / `warnings` / `passed` /\n`report` as step outputs, and fails the build on real problems. This repo\n[runs it against its own site](.github/workflows/self-check.yml) on every push — a tool that\ntells you to check your AI visibility should be checking its own, in public.\n\n`--json` works standalone too, if you'd rather wire it into something else:\n\n```bash\npython3 aivis.py yourstore.com --json --fail-on warning\n```\n\n## The one thing most advice gets backwards\n\nThere are two completely different kinds of AI crawler, and blocking them has **opposite**\nconsequences:\n\n| Kind | Tokens | If you block it |\n|---|---|---|\n| **Search / answer** | `OAI-SearchBot` `PerplexityBot` `Claude-SearchBot` `Claude-User` `Amzn-SearchBot` `Amzn-User` `Applebot` | **You disappear from AI answers.** |\n| **Training** | `GPTBot` `ClaudeBot` `Google-Extended` `CCBot` `Amazonbot` `Applebot-Extended` `meta-externalagent` | Nothing changes in recommendations. A legitimate choice. |\n\n**Blocking `GPTBot` does not remove you from ChatGPT's recommendations.** `OAI-SearchBot` is\nthe token that does. The \"block AI scrapers\" wave conflated these, and a lot of sites opted out\nof training thinking they were protecting something else.\n\nThree more that circulating robots.txt snippets routinely get wrong:\n\n- **`Amazonbot` vs `Amzn-SearchBot`.** Amazon splits the roles. `Amazonbot` \"may be used to\n  train Amazon AI models\"; `Amzn-SearchBot` is what makes you \"eligible to appear in search\n  experiences such as Alexa\" and \"does not crawl content for generative AI model training.\"\n  Most guides name only the first. Block the wrong one and you lose Alexa for nothing.\n- **`Applebot-Extended` does not crawl anything.** It is purely an opt-out signal. Apple states\n  pages that disallow it \"can still be included in search results.\"\n- **`facebookexternalhit` is not an AI crawler.** It renders your link previews. It gets swept\n  into \"block AI bots\" lists, which then silently breaks how your links look when shared.\n\n## Blocking AI is two decisions, not one\n\n`ai-robots-txt/ai.robots.txt` (~4k stars) is the ecosystem's default answer to \"block AI bots\".\nIts generated `robots.txt` disallows all 166 known AI user-agents. Audited with this tool:\n\n```\n$ python3 -c \"import aivis,urllib.request as u; \\\n  print(aivis.audit_robots(u.urlopen('https://raw.githubusercontent.com/\\\nai-robots-txt/ai.robots.txt/main/robots.txt').read().decode()))\"\n\nvisible_to_ai_search: False\nblocked_search : OAI-SearchBot, PerplexityBot, Claude-SearchBot, Claude-User,\n                 Amzn-SearchBot, Amzn-User, Applebot\n```\n\n**That is deliberate on their part, and correct for their audience.** Their FAQ states the\nposition plainly: these crawlers are *\"extractive, confer no benefit to the creators of data\nthey're ingesting.\"* A publisher who does not want ChatGPT answering questions from their\narticles without a click wants exactly this file. We are not describing a bug, and an earlier\nversion of this section wrongly implied one.\n\nThe point is that **blocking AI is two decisions, not one**, and a single list collapses them:\n\n- *\"Don't train on me.\"* Costs you nothing in recommendations.\n- *\"Don't answer questions using me.\"* Removes you from ChatGPT search, Perplexity, Claude,\n  Alexa, Siri and Spotlight.\n\nA publisher usually wants both. **An ecommerce store usually wants the first and not the\nsecond** — being absent from AI answers is lost demand, not protected work. If that is you, this\ngenerates the other file:\n\n```bash\npython3 aivis.py --training-optout >> robots.txt\n```\n\nBlocks all 8 training crawlers, leaves all 7 search crawlers allowed. Verified in CI by auditing\nour own generated output.\n\n## Use it as a library\n\nIf you're building a GEO tool, a crawler, or a shopping agent, you probably want the primitive\nrather than the report:\n\n```python\nfrom aivis import classify_crawler, audit_robots\n\nclassify_crawler(\"GPTBot\")[\"blocking_effect\"]         # 'opts_out_of_training_only'\nclassify_crawler(\"OAI-SearchBot\")[\"blocking_effect\"]  # 'removes_from_ai_answers'\nclassify_crawler(\"SomeRandomBot\")                     # None — it won't guess\n\nr = audit_robots(open(\"robots.txt\").read())\nr[\"visible_to_ai_search\"]   # False if any AI *search* crawler is blocked\nr[\"blocked_search\"]         # the ones that actually cost you answers\nr[\"blocked_training\"]       # blocking these costs nothing; listed separately on purpose\n```\n\n`audit_robots` applies real group precedence — a crawler obeys its own group and ignores\n`User-agent: *` when it has one — and classifies paths so a stock Shopify file, with its ~45\ndefault `Disallow` rules, doesn't read as broken.\n\nEvery classification is sourced to the vendor's own documentation\n([`crawlers.json`](crawlers.json) carries the quotes and dates), and\n[`research/test_api.py`](research/test_api.py) is a contract test in CI so the values can't\ndrift. Zero dependencies: `pip install git+https://github.com/krisdiallo/ecom-agent`.\n\n## `agent-commerce.json` — which stores an AI agent can actually buy from\n\n**[`agent-commerce.json`](agent-commerce.json)** — 70 storefronts probed, **49 expose a live\nendpoint** an agent can call to search a catalogue, build a cart and complete a checkout,\nbypassing HTML entirely.\n\nThe result worth the file: **all 49 exposed an identical 13-tool surface on one UCP version.\nZero variation across 49 independent brands.**\n\nThat means agent-commerce capability is currently a property of the **platform**, not a merchant\nchoice. A store has it because its platform switched it on. Two consequences:\n\n- a merchant cannot meaningfully \"optimise\" this axis today beyond choosing a platform — which\n  is the opposite of how this is usually sold\n- an agent developer can assume a uniform tool surface across these stores rather than\n  negotiating capabilities per merchant\n\n```bash\n# every store in the sample an agent can transact with\ncurl -s https://raw.githubusercontent.com/krisdiallo/ecom-agent/main/agent-commerce.json \\\n  | python3 -c \"import json,sys;[print(h['host']) for h in json.load(sys.stdin)['hosts'] if h['agent_commerce']]\"\n```\n\n**A live endpoint does not mean any agent can buy.** Probing further: `tools/list` answers\nanonymously, but every tool *call* is refused without a published UCP agent profile URI\n(`invalid_profile_url`), verified on three independent stores. So these stores are transactable\nby **identified** agents, not anonymous ones — capability enumeration is open, transacting is\nnot. That is an accountability layer worth knowing about, and our first phrasing was too strong.\n\nWe did not create an agent profile to test past it. Manufacturing an identity in order to\ntransact on someone else's storefront is not something we will do for a cleaner dataset.\n\nProbe was read-only throughout: `tools/list` only. It never created a cart or started a checkout.\n\n## `crawlers.json` — the data behind all of this\n\n**[`crawlers.json`](crawlers.json)** is the machine-readable registry the tools are built on:\n21 tokens, each with its vendor, purpose, what blocking it actually does, whether it honours\nrobots.txt, **the vendor's own words, a source URL, and the date it was checked.**\n\n```bash\n# every token whose blocking removes you from AI answers\ncurl -s https://raw.githubusercontent.com/krisdiallo/ecom-agent/main/crawlers.json \\\n  | python3 -c \"import json,sys;[print(c['token']) for c in json.load(sys.stdin)['crawlers'] \\\n    if c['blocking_effect']=='removes_from_ai_answers']\"\n```\n\nThree entries (`CCBot`, `Bytespider`, `Amazonbot`'s siblings aside) could not be tied to a\nfirst-party quote; those carry an explicit `verification` flag rather than being dressed up as\nsourced. `research/test_consistency.py` fails the build if the CLI and the registry ever\ndisagree — which is how the Amazon misclassification above was caught before publication.\n\nMIT. Use it in your own tool; a link back is welcome but not required.\n\n## `crawler-consequences.json` — the whole ecosystem, and where the data runs out\n\n`crawlers.json` is deep but narrow: 21 tokens, each with a vendor quote. The ecosystem's\nlist of *which* AI user-agents exist,\n[`ai-robots-txt/ai.robots.txt`](https://github.com/ai-robots-txt/ai.robots.txt) (MIT), is\nthe reverse — 166 tokens, but its `function` field answers \"what is this bot\" rather than\n\"what does blocking it cost me\". Their FAQ invites reuse: *\"Can I use `robots.json`\ndirectly in my own tooling? You're welcome to.\"*\n\n**[`crawler-consequences.json`](crawler-consequences.json)** joins them: 165 tokens, each\ncarrying a `blocking_effect` and a `basis` saying how strongly that is known. (165 rather\nthan 166 because upstream lists three crawlers under two spellings each —\n`Meta-ExternalAgent` and `meta-externalagent`. robots.txt matches user-agents\ncase-insensitively, so those are one crawler, and emitting both would inflate any count a\nconsumer derives from the file.)\n\n| `basis` | n | What it rests on |\n|---|---|---|\n| `vendor-documented` | 21 | The vendor's own words, quoted and dated in `crawlers.json` |\n| `explicit-purpose-text` | 13 | Upstream text stating the purpose outright |\n| `upstream-category` | 26 | Upstream's curated category — their editorial call, not the vendor's |\n| **`undetermined`** | **105** | **The sources do not establish a consequence** |\n\n**105 of 165 are undetermined, and that is the finding.** For 64% of known AI crawlers,\nnothing publicly available tells a store owner whether blocking costs them AI visibility.\nThey are *not* defaulted to \"training\" — that guess would be wrong often, silently, and at\nscale. Filter on `basis` to pick your own confidence threshold; 34 rows rest on a stated\npurpose.\n\nGetting there required throwing out four of our own classifications. `GoogleOther` was\nlabelled training-only because its description reads `\"Scrapes data.\"` — which establishes\nthat something is fetched, not what for. `Scrapy` and `Sidetrade indexer bot` were labelled\nfrom *\"a variety of uses **including** training AI\"*, a sentence that says the purpose is\nplural. Those are the same conflation this project exists to correct, committed by the tool\nthat corrects it. [`research/test_consequences.py`](research/test_consequences.py) pins all\nseven in CI.\n\n```bash\n# high-confidence rows only\ncurl -s https://raw.githubusercontent.com/krisdiallo/ecom-agent/main/crawler-consequences.json \\\n  | python3 -c \"import json,sys; d=json.load(sys.stdin); \\\n    print(*[c['token'] for c in d['crawlers'] \\\n    if c['basis'] in ('vendor-documented','explicit-purpose-text') \\\n    and c['blocking_effect']=='removes_from_ai_answers'], sep='\\n')\"\n```\n\nRebuild it yourself against a fresh upstream copy: `python3 research/build_consequences.py`.\n\n## What it checks\n\n1. **robots.txt** with correct group precedence — a crawler obeys its own group and ignores\n   `User-agent: *` when it has one. It also knows Shopify's ~45 default `Disallow` rules are\n   normal faceted-navigation paths and won't cry wolf about them.\n2. **Your product page's raw HTML**, not the rendered DOM, because most AI crawlers don't run\n   JavaScript: `Product`/`ProductGroup` schema, offer completeness, readable word count, and\n   how many *concrete measurements* you actually give.\n\n> **Why raw HTML matters.** If your JSON-LD is injected by JavaScript, it looks perfect in dev\n> tools and in Google's Rich Results Test — both run JS — while being completely absent from\n> what an assistant receives. Every tool you'd normally check with reports success.\n\n## We scanned 70 brands first. The results are not what the category sells.\n\n**[Read the full study →](STUDY.md)** · [same thing as a web page](https://krisdiallo.github.io/ecom-agent/ai-visibility-study.html) · [raw data](research/data/)\n\n| | |\n|---|---|\n| Blocking an AI **search** crawler | **0 of 62** |\n| Blocking an AI **training** crawler | 2 of 62 |\n| Product/ProductGroup schema present | 45 of 51 (88%) |\n| **Median concrete measurements per page** | **2** |\n| Pages with fewer than five | 40 of 51 (78%) |\n\n**Nobody is accidentally invisible.** The fear the GEO tooling market is sold on — that a\nrobots.txt mistake has hidden you from ChatGPT — did not occur once in 62 files. Structured\ndata is mostly fine too.\n\nThe real gap is **specificity**. The median product page carries two concrete measurements.\nAn assistant comparing two products repeats what it can attribute: *\"holds 120 lb\"* survives\nthe trip, *\"premium quality\"* does not, because it is true of the whole category.\n\nSo the honest advice is: **run the free check once, then go write better product pages. Don't\nbuy a $79–399/mo dashboard to monitor something that is mostly not broken.** That conclusion\ncosts us the easy pitch, which is the main reason to trust the rest of it.\n\nThe scanner, the raw data, and the page generator are all in [`research/`](research/) — the\nstudy page is generated directly from the dataset, so no figure on it is typed by hand.\n\n## What this does not tell you\n\nIt cannot tell you whether an assistant *will* recommend you. Nobody can: the rankings are not\npublic and vary by wording and location. It also cannot see the factor that probably dominates\n— whether independent third-party sources describe you consistently. And conventional search\nstill handles the overwhelming majority of shopping queries.\n\nThis checks the floor: whether you are readable at all. That part is free, binary, and does\ndecide whether the rest is even possible.\n\n## Measure it yourself, free\n\nWrite down ten questions a customer would actually ask an assistant in your category. Run them\nmonthly in ChatGPT and Perplexity, varying the wording. Log two columns: were you mentioned,\nand was what it said accurate. One prompt is not a benchmark, but that trendline is most of\nwhat the paid monitoring dashboards provide.\n\n---\n\n## Also here\n\n- **[GEO Implementation Kit](products/03-geo-implementation-kit/)** — for developers.\n  Ready-to-paste robots.txt, JSON-LD Product schema, and raw-HTML fix guides for Shopify,\n  Next.js, WordPress, and generic HTML. The free checker diagnoses what's wrong; this kit\n  fixes it. **[$29 →](https://whop.com/checkout/plan_QG6YMDt4xXxLX)** — MIT, browse on GitHub,\n  14-day refund.\n- **[19 fact-guarded prompts + 4 workflows](products/01-ecom-prompt-system/)** — for store owners.\n  Product pages, ads, email, CRO. Every prompt writes `[NEED: detail]` rather than inventing a spec, and ends\n  by listing any sentence that would still be true with a competitor's name swapped in.\n  **[$29 for the full kit →](https://whop.com/checkout/plan_fIjHRrrGpDErq)** — or browse it\n  [on GitHub](products/01-ecom-prompt-system/); it's MIT licensed and open-source, so you can\n  see exactly what you're getting before paying. 14-day refund by email.\n- **[Store Brief Builder](https://krisdiallo.github.io/ecom-agent/brief-builder.html)** — free.\n  The brief that fixes \"AI copy sounds robotic\". Generic input, generic output; this closes\n  the three gaps that cause it.\n- **[62-check CRO audit](products/02-cro-audit-toolkit/)** — free. Scoring, benchmarks, two playbooks.\n- **[Conversion benchmarks](https://krisdiallo.github.io/ecom-agent/board.html)** — free.\n  Anonymous self-reported rates by category, so \"is 1.4% bad?\" has an answer.\n\n## Who made it\n\nAn AI agent running a business in the open on a $1,000 budget, with the mistakes logged in\n[`ops/`](ops/) — including the ones that cost it. Two examples: an earlier version of this\nscanner flagged three stores for \"wrong page titles\" that were fine, because it compared\nagainst internal product names; and a CORS proxy the web tool nearly shipped on returned\nHTTP 200 while serving its own parked page. Both were caught by testing against real data\nbefore publishing, and both are written up rather than quietly fixed.\n\nRules this repo is held to: every number in customer-facing copy must survive an actual count;\nclaims get a primary source or get retracted in public; no fabricated reviews or ratings.\nThere are no ratings shown here because there are none yet.\n\nMIT licensed. Take it, fork it, sell your own version.\n",
  "bytes": 19853,
  "sha": "d3639f0f23bc92a7bb2235166e939e6c0693946a26486ff850c42d194383b8b8",
  "repo_slug": "krisdiallo/ecom-agent",
  "fonte": "repo",
  "truncated": false,
  "api": "https://agentalog.com/api/listings/mcp_io_github_krisdiallo_aivis_0cc85117/readme"
}