\n\n---\n\n> Not using exclusively Claude Code? See [Zyte Coding Agent Add-Ons](https://docs.zyte.com/ai-code.html) for alternatives.\n\n## Install\n\n```bash\nclaude plugin marketplace add zytedata/claude-skills\nclaude plugin install zyte-web-data@zyte-ai\n```\n\nIf Claude Code is already running, reload plugins in the active session:\n\n```bash\n/reload-plugins\n```\n\nIf `/reload-plugins` isn't available (e.g. in the VS Code extension), restart Claude Code.\n\nSee also: [Discovering and installing plugins](https://code.claude.com/docs/en/discover-plugins.md)\n\n---\n\n## What it does\n\nThis is Zyte's official [Claude Code](https://code.claude.com) plugin that generates production-ready [Scrapy](https://scrapy.org) spiders with [web-poet](https://web-poet.readthedocs.io) page objects from a plain-English prompt. Give it a URL and describe what you want to extract. It handles site exploration, schema discovery, code generation, and smoke testing: no boilerplate, no manual selector hunting.\n\nThe plugin explores the target site, discovers available fields, and presents a schema for your approval before generating a single line of code. After you confirm the schema, it creates a Scrapy project with all dependencies configured, generates web-poet page objects and test fixtures, wires up the spider, and runs a smoke test to verify that extraction is working before handing the project back to you.\n\nOptionally, use `/scrape-scrapy-cloud` to deploy directly to [Scrapy Cloud](https://www.zyte.com/scrapy-cloud/) for scheduled runs, job history, and monitoring. A [free tier is available](https://docs.zyte.com/scrapy-cloud/pricing.md).\n\n---\n\n## Use cases\n\nThe `/scrape` skill works on any website with repeating structured content: detail pages linked from a listing or category page. Examples from the skill:\n\n- Product catalogs\n- Job listings\n- Recipes\n\n---\n\n## How does it work?\n\nThe `/scrape` skill orchestrates five stages automatically:\n\n```\n1. Decide which fields to extract → /scrape-define\n2. Analyze the website → /scrape-spec\n3. Create the Scrapy project → /scrape-ensure-project\n4. Generate the extraction code → /scrape-codegen\n5. Generate the spider → /scrape-create-spider\n```\n\nEach stage feeds directly into the next. When the pipeline completes, you have a runnable spider and a passing test suite:\n\n```bash\nuv run scrapy crawl \nuv run pytest fixtures/\n```\n\n---\n\n## Skills\n\n### Orchestration\n\n| Skill | Description |\n|---|---|\n| `scrape` | End-to-end web scraping workflow — from URL to working spider with web-poet page objects |\n\n### Pipeline stages (called automatically by `/scrape`)\n\n| Skill | Description |\n|---|---|\n| `scrape-define` | Quick schema definition: explore one detail page, discover fields, fast approval loop |\n| `scrape-spec` | Explore diverse pages and validate the extraction spec: downloads pages, compares variants, optional browser review |\n| `scrape-explore-site` | Explore a website to find and save diverse pages (start, list, detail) with classified links |\n| `scrape-analyze-page` | Extract all available fields with values from a detail page |\n| `scrape-ensure-project` | Ensure a Scrapy project exists with scrapy-poet and Zyte API support |\n| `scrape-codegen` | Generate web-poet page object code from an extraction spec |\n| `scrape-codegen-analyze` | Analyze an HTML page to produce field extraction instructions for code generation |\n| `scrape-codegen-generate` | Generate web-poet page object code from per-page extraction analyses |\n| `scrape-create-spider` | Generate a Scrapy spider that wires page objects together |\n\n### Utilities\n\n| Skill | Description |\n|---|---|\n| `scrape-add-page-object` | Add an empty web-poet page object to a Scrapy project |\n| `scrape-review-schema` | Generate an HTML review page for schema and extracted data verification |\n\n### Deployment\n\n| Skill | Description |\n|---|---|\n| `scrape-scrapy-cloud` | Deploy projects, schedule spiders, list/stop jobs, and view items or logs on [Scrapy Cloud](https://www.zyte.com/scrapy-cloud/) |\n| `scrape-zyte-login` | Set up your Zyte account and credentials |\n\n---\n\n## Prerequisites\n\n- [Claude Code](https://code.claude.com) (CLI or desktop app)\n- [`uv`](https://docs.astral.sh/uv/) — used to create and manage the Scrapy project\n\nProject dependencies (scrapy, scrapy-poet, scrapy-zyte-api, web-poet, extruct, price-parser, pytest) are installed automatically by the skills.\n\n---\n\n## Quickstart\n\nAny scraping prompt triggers the skill automatically. For example:\n\n```\nScrape books.toscrape.com\n```\n\nThe plugin walks you through schema approval interactively, then generates a complete, tested Scrapy project.\n\n---\n\n## Update\n\nWe recommend enabling automatic updates:\n\n1. Enter `/plugin` in a Claude Code session\n2. Select **Marketplaces** → **zyte-ai** → **Enable auto-update**\n\nTo update manually:\n\n```bash\nclaude plugin marketplace update zytedata/claude-skills\n```\n\nThen, in a Claude Code session:\n\n```bash\n/reload-plugins\n```\n\nIf `/reload-plugins` isn't available (e.g. in the VS Code extension), restart Claude Code.\n\n---\n\n## Evaluation\n\nWe automatically evaluate skills and track both wall time and cost. We measure and aim to improve these metrics over time.\n\n---\n\n## Feedback\n\nIf you find any issue — such as prompts that did not work as expected, or that caused excessive wall time or cost — please [open a GitHub issue](https://github.com/zytedata/claude-skills/issues).\n\nProvide as much detail as possible to help us reproduce the issue. You are welcome to anonymize target websites or other data.\n\n---\n\n## Frequently asked questions\n\n### Is a Zyte account required?\n\nNo. The generated spider is a standard Scrapy project that runs locally with `uv`. A Zyte account is required only if you want to deploy to [Scrapy Cloud](https://www.zyte.com/scrapy-cloud/) or use [Zyte API](https://www.zyte.com/zyte-api/) to access sites that block standard scrapers. If you want to use Zyte API, you'll need an account to generate an API key.\n\n### Does it handle JavaScript-rendered pages?\n\nThe generated project includes `scrapy-zyte-api` as a dependency. Enabling headless browser rendering requires a [Zyte API](https://www.zyte.com/zyte-api/) key. The `/scrape-zyte-login` skill guides you through setting up your credentials.\n\n### What Python libraries does the generated project use?\n\nThe project template includes `scrapy`, `scrapy-poet`, `scrapy-zyte-api`, `web-poet`, `extruct`, `price-parser`, and `pytest`. All dependencies are installed automatically via `uv sync`.\n\n### Can the generated spider run without Claude Code?\n\nYes. The plugin generates a standard Scrapy project. Run it directly with:\n\n```bash\nuv run scrapy crawl \n```\n\nYou can extend, modify, and deploy it independently of Claude Code.\n\n---\n\n## License\n\nSee [LICENSE.md](LICENSE.md) for the Zyte End User License Agreement.\n\n---\n\n## Demo\n\n
![\"Zyte\"](\"assets/zyte-logo.png\"){kind=logo}
\n![\"Version](\"https://img.shields.io/badge/version-0.2.3-blue\"){kind=icon}
![\"Zyte](\"https://img.shields.io/badge/license-Zyte%20EULA-b02cce\"){kind=icon}
![\"GitHub](\"https://img.shields.io/github/stars/zytedata/claude-skills?style=social\")
![\"Demo:](\"https://img.youtube.com/vi/KU8DJISQYeM/maxresdefault.jpg\"\n\t\t\t\t)
\n\t\n