tavily-crawl
Crawl websites and extract content from multiple pages via the Tavily CLI. Use this skill when the user wants to crawl a site, download docu
Open source Repository Open in the app JSON README (API)
About
Crawl websites and extract content from multiple pages via the Tavily CLI. Use this skill when the user wants to crawl a site, download documentation, extract an entire docs section, bulk-extract pages, save a site as local markdown files, or says "crawl", "get all the pages", "download the docs", "extract everything under /docs", "bulk extract", or needs content from many pages on the same domain. Supports depth/breadth control, path filtering, semantic instructions, and saving each page as a local markdown file.
Details
- Kind
- Agent skills
- Topic
- Web search, scraping & browser
- Publisher
- tavily-ai
- Origin
- skillssh
- Category
- ferramentas
- Stars
- 473
- Forks
- 37
- Open pull requests
- 1
- Last push
- 2026-09-04T15:16:07Z
- Repository state
- ativo
- License
- MIT
- Added
- 2026-08-30 15:21:15
- Updated
- 2026-09-10 18:04:52
- Origin id
tavily-ai/skills/tavily-crawl
README
# Tavily Agent Skills Web search, content extraction, site crawling, URL discovery, and deep research — powered by the Tavily CLI. ## Installation ### Recommended: guided CLI setup Install the [Tavily CLI](https://github.com/tavily-ai/tavily-cli): ```bash curl -fsSL https://cli.tavily.com/install.sh | bash ``` On a fresh interactive desktop install, the installer starts `tvly init` automatically. Otherwise run it after installation: ```bash tvly init # Prefer to open the sign-in link yourself tvly init --no-browser ``` `tvly init` authenticates, detects Claude Code, Codex, and Cursor, installs these skills, and verifies a live search. It reuses an existing credential and is safe to rerun. Skills installed this way are pinned to the CLI release, so run `tvly update` before `tvly init` when refreshing them. `tvly search` and `tvly extract` also work without authentication, subject to a keyless rate-limit cap. Use `tvly init --skip-auth` to keep keyless mode while installing the skills. `map`, `crawl`, and `research` require authentication. Browser-based OAuth is the preferred interactive sign-in method. `--no-browser` simply prints the sign-in link instead of opening it automatically; the flow still returns to a localhost callback on the machine running `tvly`. In remote sessions, make sure that callback is reachable (SSH may require port forwarding). For unattended agents or CI, authenticate beforehand or provide `TAVILY_API_KEY` securely so the process does not wait for human input. ### Manual skill installation For other agents, or when you only want to install the skills: ```bash npx skills add https://github.com/tavily-ai/skills ``` Install the CLI separately if needed: ```bash uv tool install tavily-cli # or: pip install tavily-cli ``` Then use `tvly init --skip-skills` for guided authentication and verification, or `tvly login` when you only need to authenticate. ### Keep the CLI current ```bash tvly update --check tvly update ``` ## Available Skills | Skill | Description | |-------|-------------| | **[tavily-search](skills/tavily-search/SKILL.md)** | Search the web with LLM-optimized results. Supports domain filtering, time ranges, and multiple search depths. | | **[tavily-extract](skills/tavily-extract/SKILL.md)** | Extract clean markdown/text content from specific URLs. Handles JS-rendered pages. | | **[tavily-crawl](skills/tavily-crawl/SKILL.md)** | Crawl websites and extract content from multiple pages. Save as local markdown files. | | **[tavily-map](skills/tavily-map/SKILL.md)** | Discover all URLs on a website without extracting content. Faster than crawling. | | **[tavily-research](skills/tavily-research/SKILL.md)** | Comprehensive AI-powered research with citations. Multi-source synthesis in 30-120s. | | **[tavily-dynamic-search](skills/tavily-dynamic-search/SKILL.md)** | Filter and deduplicate large search or extraction results without flooding agent context. | | **[tavily-cli](skills/tavily-cli/SKILL.md)** | Overview skill with workflow guide, install/auth instructions. | | **[tavily-best-practices](skills/tavily-best-practices/SKILL.md)** | Reference docs for building production-ready Tavily integrations. | ## Workflow Start simple, escalate when needed: 1. **Search** — Find pages on a topic (`tvly search "query" --json`) 2. **Extract** — Get content from a specific URL (`tvly extract "https://..." --json`) 3. **Map** — Discover URLs on a site (`tvly map "https://..." --json`) 4. **Crawl** — Bulk extract from a site section (`tvly crawl "https://..." --output-dir ./docs/`) 5. **Research** — Deep multi-source analysis (`tvly research "topic" --model pro`)