{ "markdown": "# scrapewright\n\n[![PyPI](https://img.shields.io/pypi/v/scrapewright)](https://pypi.org/project/scrapewright/)\n[![Python](https://img.shields.io/pypi/pyversions/scrapewright)](https://pypi.org/project/scrapewright/)\n[![License: MIT](https://img.shields.io/badge/license-MIT-green.svg)](LICENSE)\n\n**Give it a URL. It writes the scraper.**\n\nMost e-commerce catalog scraping splits into two worlds: sites on a known\nplatform (Shopify, WooCommerce) that expose a clean JSON feed, and everything\nelse — bespoke HTML where you hand-write a parser per site and re-write it every\ntime the markup shifts. scrapewright collapses both into one call:\n\n1. **Detect** the platform behind a URL.\n2. For known platforms, **extract deterministically** from their public catalog\n API — free, stable, no LLM.\n3. For custom HTML, **synthesize a reusable extractor once** with an LLM, cache\n it, and **replay it deterministically forever after**.\n\nThe LLM is a *compiler*, not a runtime. It runs **once per site** to produce a\nrecipe of CSS selectors; every page after that is parsed by plain BeautifulSoup\nat zero marginal cost. That is the whole cost-control story — no per-page model\ncalls, no token bill that scales with your crawl.\n\n```\n ┌─────────────┐\n store URL ───▶ │ detect │\n └──────┬──────┘\n ┌──────────────────┼──────────────────┐\n ▼ ▼ ▼\n shopify woocommerce generic HTML\n products.json wc/store/products (page mode)\n │ │ │\n │ deterministic │ ▼\n │ (free) │ cached recipe? ──yes──▶ replay (free)\n └────────┬─────────┘ │ no\n ▼ ▼\n Product{} ◀───── selectors ── JSON-LD? ──yes──▶ Product{} (free)\n ▲ │ no\n │ ▼\n └──────── replay ◀── LLM synthesizes recipe ONCE ──▶ cache\n```\n\nEverything normalizes to one `Product` shape, so downstream code never knows or\ncares which path a record came from.\n\n## Install\n\n```bash\npip install scrapewright # deterministic paths (Shopify, Woo, JSON-LD)\npip install \"scrapewright[llm]\" # + LLM recipe synthesis for custom HTML\npip install \"scrapewright[llm,js,excel,mcp]\" # + JS rendering, XLSX, MCP server\nplaywright install chromium # only needed for --js\n```\n\n## Use it\n\n```python\nfrom scrapewright import Scrapewright\n\nsw = Scrapewright()\n\n# Catalog mode — a whole Shopify/WooCommerce store, deterministically\nfor product in sw.scrape_catalog(\"https://shop.example.com\", max_items=200):\n print(product.brand, product.title, product.price, product.currency)\n\n# Page mode — one custom-HTML product page.\n# First call: tries JSON-LD (free); if absent, the LLM writes a recipe once.\n# Every later call on that domain: replayed from the cached recipe, no LLM.\nitem = sw.scrape_page(\"https://boutique.example.com/products/wool-coat\")\nprint(item.model_dump(exclude={\"raw\"}))\n\n# Crawl mode — walk a WHOLE custom store from one listing/category URL.\n# The frontier discovers product pages (deterministic, free); the first page\n# pays the single synthesis cost, every other page replays the recipe.\nfor product in sw.crawl(\"https://boutique.example.com/collection\", max_items=100):\n print(product.title, product.price)\n```\n\n### CLI\n\n```bash\nscrapewright detect https://shop.example.com # platform + strategy\nscrapewright run https://shop.example.com --max 50 # scrape a catalog → JSONL\nscrapewright crawl https://boutique.example.com/collection -o products.xlsx\nscrapewright run https://shop.example.com -o products.csv # Excel-ready CSV\nscrapewright add https://boutique.example.com/products/coat # learn a site\nscrapewright run https://boutique.example.com/products/coat --no-llm\nscrapewright list # cached recipe domains\n```\n\n`-o` writes `.csv` (Excel-ready, UTF-8 BOM), `.xlsx` (`pip install scrapewright[excel]`),\nor `.jsonl`; without it, products stream to stdout as JSONL.\n\n### Know what you are dealing with\n\n`detect` answers the routing question before a job starts:\n\n```\n$ scrapewright detect https://some-store.com\nhttps://some-store.com\n platform: bigcommerce\n catalog: -\n strategy: crawl\n note: BigCommerce (Stencil) markup\n```\n\nTwelve platforms are recognized: **Shopify** and **WooCommerce** publish a free\nJSON catalog, so those route to `catalog` — deterministic, no LLM, no browser.\n**Magento, BigCommerce, Salesforce Commerce Cloud, Squarespace, Wix, Webflow,\nPrestaShop, Shopware, Ecwid** and **OpenCart** are recognized by fingerprint and\nroute to `crawl`, where the recipe path handles them like any custom site — the\npoint of naming them is knowing what you face, not writing twelve parsers.\nWix and Ecwid render client-side, so detection says `crawl+js` up front.\n\nA site behind an anti-bot wall reports `strategy: blocked` with the HTTP status,\nrather than pretending it found nothing.\n\n### Bring your own schema\n\nProducts are just the built-in default. Declare the fields you want and the same\ncompile-once/replay-free loop works on any structured page — job posts, listings,\nregistry records:\n\n```bash\nscrapewright run https://jobs.example.com/p/123 -f title -f company -f salary:number -f tags:list --schema-name job\n```\n\n```python\nfrom scrapewright import Scrapewright, Schema\n\njob = Schema.from_names([\"title\", \"company\", \"salary:number\", \"tags:list\"], name=\"job\")\nrecord = Scrapewright().extract(\"https://jobs.example.com/p/123\", job)\nprint(record.data) # {'title': ..., 'company': ..., 'salary': ..., 'tags': [...]}\n```\n\nField kinds are `text` (default), `number`, `url`, and `list`. Recipes are cached\nper site *and* per schema, so one domain can be compiled against several field\nsets without them overwriting each other.\n\n### Use it from an AI agent (MCP)\n\nscrapewright ships an [MCP](https://modelcontextprotocol.io) server, so an agent can\ncall it as a tool instead of reading raw HTML itself:\n\n```bash\npip install \"scrapewright[mcp,llm]\"\nscrapewright mcp\n```\n\nPoint any MCP client at that command and the agent gains five tools: `detect_site`,\n`scrape_catalog`, `extract_page`, `crawl_site`, and `list_learned_sites`.\n\nDrop this into your client's config — Claude Desktop, Cursor, or anything else that\nspeaks MCP:\n\n```json\n{\n \"mcpServers\": {\n \"scrapewright\": {\n \"command\": \"uvx\",\n \"args\": [\"--from\", \"scrapewright[mcp,llm]\", \"scrapewright\", \"mcp\"],\n \"env\": { \"ANTHROPIC_API_KEY\": \"sk-ant-...\" }\n }\n }\n}\n```\n\nThe key is only needed for sites on no known platform, where a recipe has to be\nwritten once. Shopify and WooCommerce stores work without it.\n\n\n\nThe economics are the point. An agent that reads pages itself pays model tokens per\npage, forever. These tools pay **once per site** — an agent crawling 500 pages spends\none synthesis, not five hundred, and platform stores (Shopify, WooCommerce) cost\nnothing at all.\n\n### Run it as a service\n\nThe same core behind an HTTP API, with keys, quotas, metering and background\njobs:\n\n```bash\npip install \"scrapewright[service,llm]\"\nscrapewright keys create --label alice --plan free\nscrapewright serve --port 8000\n```\n\n```bash\ncurl -X POST localhost:8000/v1/extract -H \"X-API-Key: sw_...\" -H \"Content-Type: application/json\" -d '{\"url\": \"https://shop.example.com/products/coat\"}'\n```\n\n| Endpoint | Purpose |\n|---|---|\n| `POST /v1/detect` | platform + strategy (cheap) |\n| `POST /v1/extract` | one page -> structured record |\n| `POST /v1/crawl` | a whole site -> job id (crawls outlive a request) |\n| `GET /v1/jobs/{id}` | poll a crawl |\n| `GET /v1/usage` | what this key has consumed, against its plan |\n\n#### Prepaid credits, no subscription\n\nOne action costs real money: **compiling a new site**, a single LLM pass over a\npage, measured at $0.02 on a small product page and $0.15 on a heavy rendered\none. Everything after that is BeautifulSoup — the ten-thousandth record from a\ncompiled site is free to serve. So credits are priced off that one action, and\neverything else is denominated relative to it:\n\n| Action | Credits |\n|---|---|\n| 1 record delivered | 1 |\n| 1 browser render | 5 |\n| 1 new site compiled | 300 |\n| page fetches, `detect` | free |\n\n```\n$ scrapewright plans\npack credits price $/credit margin\nstarter 10,000 $10 0.00100 80.0%\ngrowth 50,000 $40 0.00080 75.0%\nscale 250,000 $150 0.00060 66.7%\n\nFree: 1,000 credits a month, resetting.\n```\n\nMargin is measured on compiling a site, because that is the only step that\ncosts anything; a test fails if a price edit drops any pack below 60%. A free\naccount can cost us at most $0.20 a month, even if every free credit goes to\nthe most expensive action there is.\n\nCredits are a **ledger, not a counter** — every grant and every charge is a row,\nso a disputed bill can be reconstructed line by line, and a replayed payment\nwebhook cannot double-credit (grants take an idempotency key). Running out\nreturns `402` with the balance and what to do about it; a crawl is capped by the\ncredits on hand, so a job stops at what the caller can pay for instead of\noverdrawing.\n\n```bash\nscrapewright credits grant --pack starter --idempotency \nscrapewright credits balance \n```\n\n#### Taking payment\n\nStripe is wired in and turned on by environment, not by a code change:\n\n```bash\npip install \"scrapewright[service,stripe]\"\nexport STRIPE_SECRET_KEY=sk_test_... # absent -> nothing is for sale\nexport STRIPE_WEBHOOK_SECRET=whsec_... # absent -> webhooks are refused\nscrapewright serve\n```\n\n| Endpoint | Purpose |\n|---|---|\n| `GET /v1/credits/packs` | the price list — public, no key needed |\n| `POST /v1/credits/checkout` | start a purchase, returns a Stripe Checkout URL |\n| `POST /v1/webhooks/stripe` | payment notifications from Stripe |\n\nThe webhook endpoint takes **no API key** — Stripe is the caller, so the\nsignature *is* the credential, and an unverified endpoint would be a free credit\nprinter for anyone who guessed the URL. Three rules hold the integration up:\n\n* **Verify every signature.** No signing secret configured means webhooks are\n refused outright, rather than accepted unverified.\n* **Never trust an amount off the wire.** The event names a pack; how many\n credits that pack is worth is looked up from our own price list, so a tampered\n payload buys exactly what it paid for or nothing at all.\n* **Grant idempotently, keyed on the Checkout session.** Stripe retries\n deliveries, and one payment can produce several event types — the session id\n is what identifies the money that actually moved.\n\n`examples/stripe_smoke_test.py` runs the whole path against Stripe's test mode\nwith the 4242 card. Any other provider plugs into the same two-method\n`BillingProvider` protocol in `scrapewright.service.billing`; without one, the\nservice simply runs free, which is the right default for a demo or a self-hosted\ninstance.\n\nDocker:\n\n```bash\ndocker build -t scrapewright . # static paths\ndocker build -t scrapewright --build-arg WITH_JS=1 . # + headless Chromium\ndocker run -p 8000:8000 -v sw-data:/data scrapewright\n```\n\n### Client-side-rendered stores\n\nAdd `--js` (or `Scrapewright(js=True)`) and pages that render their catalog in the\nbrowser become extractable:\n\n```bash\nscrapewright run https://spa-store.example.com/products/x --page --js\nscrapewright crawl https://spa-store.example.com/shop --js -o products.xlsx\n```\n\nRendering stays **rare by construction**: the static fetch runs first, and Chromium is\nonly started when the static HTML is an empty client-side shell or extraction on it\nfails. A recipe learned from rendered HTML is tagged `needs_js`, so later runs on that\nsite skip the wasted static hop. The browser starts at most once per run and is reused\nfor every page.\n\n## The `Product` shape\n\n```python\nurl: str # canonical product URL\ntitle: str\nbrand: str | None\nprice: Decimal | None # parsed from \"1,250.00\" / \"1.250,00\" / \"€1290\" alike\ncurrency: str | None\navailable: bool | None\nimages: list[str] # absolute URLs\nsizes: list[str]\ndescription: str | None\nsku: str | None\nsource_platform: str # shopify | woocommerce | json-ld | selector\n```\n\nA record is **usable** when it carries a title, a price, and a URL. The\nvalidator (`scrapewright.coverage`) reports the usable ratio across a batch —\nthe number a recipe is trusted on before it's cached.\n\n## How the pieces fit\n\n| Module | Role |\n|---|---|\n| `detect` | Platform registry: free-catalog probes, then fingerprints for 12 platforms; returns the strategy to use |\n| `extract/shopify`, `extract/woocommerce` | Deterministic catalog extractors |\n| `extract/jsonld` | schema.org/Product from `