web-scraping
yfe404/web-scraper · skills.sh
Open source Repository Open in the app JSON README (API)
About
Skill publicada por yfe404/web-scraper no skills.sh. Instale com: npx skills add yfe404/web-scraper@web-scraping
Details
- Kind
- Agent skills
- Topic
- Web search, scraping & browser
- Publisher
- yfe404
- Origin
- skillssh
- Category
- ferramentas
- Stars
- 99
- Forks
- 20
- Last push
- 2026-03-18T12:13:46Z
- Repository state
- ativo
- Language
- TypeScript
- License
- MIT
- Added
- 2026-10-06 04:49:19
- Updated
- 2026-10-06 04:49:19
- Origin id
yfe404/web-scraper/web-scraping
README
# Web Scraping Skill
Intelligent web scraping with automatic strategy selection and TypeScript-first Apify Actor development.
## Overview
This skill provides:
- **Adaptive reconnaissance** - Phases 0-5 with quality gates that skip unnecessary work (curl first, browser only if needed)
- **Framework-aware detection** - Identifies site framework before searching, skips irrelevant patterns
- **Validated findings** - Every claimed selector/path/API is tested before reporting
- **Self-critiquing reports** - Intelligence reports include gap analysis and staleness warnings
- **Iterative implementation** - Starts simple, adds complexity only if needed
- **Production-ready guidance** - TypeScript-first Apify Actor development
## Installation
Add this skill to Claude Code by placing this directory in the skills folder.
## Quick Start
### Scenario 1: Scrape a Website
```
User: "Scrape https://example.com"
Claude will automatically:
1. Phase 0: curl raw HTML — detect framework, search for data points, check sitemaps
2. QUALITY GATE: All data in HTML? → Skip browser, go to validation
3. Phase 1: Launch stealth browser (only if needed) — capture traffic, rendered DOM
4. Phase 2: Deep scan (only for missing data) — test interactions, sniff APIs
5. Phase 3: Validate every finding — test selectors, replay APIs, confirm paths
6. Phase 4: Protection testing (only if signals detected or user requested)
7. Phase 5: Generate intelligence report with self-critique
8. Implement recommended approach iteratively
9. Test with small batch, then scale
```
### Scenario 2: Create Apify Actor
```
User: "Make this an Apify Actor"
Claude will:
1. Recommend TypeScript (strongly)
2. Guide through `apify create` command
3. Help choose appropriate template (Cheerio vs Playwright)
4. Port scraping logic to Actor format
5. Configure input schema
6. Test and deploy
```
## Directory Structure
```
web-scraping/
├── SKILL.md # Main entry point (proactive workflow)
├── workflows/ # Implementation patterns
│ ├── reconnaissance.md # Phase 1 interactive reconnaissance (CRITICAL)
│ ├── implementation.md # Phase 4 iterative implementation
│ └── productionization.md # Phase 5 Actor creation
├── strategies/ # Deep-dive guides
│ ├── framework-signatures.md # Framework detection lookup tables
│ ├── cheerio-vs-browser-test.md # Cheerio vs Browser decision + early exit
│ ├── proxy-escalation.md # Protection testing skip/run conditions
│ ├── traffic-interception.md # MITM proxy traffic capture
│ ├── sitemap-discovery.md # 60x faster URL discovery
│ ├── api-discovery.md # 10-100x faster than scraping
│ ├── dom-scraping.md # DevTools bridge + humanizer
│ ├── cheerio-scraping.md # HTTP-only (5x faster)
│ ├── hybrid-approaches.md # Combining strategies
│ ├── anti-blocking.md # Multi-layer anti-detection
│ └── session-workflows.md # Session recording, HAR, replay
├── examples/ # Runnable code
│ ├── traffic-interception-basic.js
│ ├── sitemap-basic.js
│ ├── api-scraper.js
│ ├── hybrid-sitemap-api.js
│ └── iterative-fallback.js
├── reference/ # Quick lookup
│ ├── report-schema.md # Intelligence report format (Sections 1-7)
│ ├── proxy-tool-reference.md # Proxy-MCP tools (80+)
│ ├── regex-patterns.md
│ ├── fingerprint-patterns.md
│ └── anti-patterns.md
├── apify/ # Production deployment
│ ├── typescript-first.md # Why TypeScript
│ ├── cli-workflow.md # apify create (CRITICAL)
│ ├── templates/ # TypeScript boilerplate
│ └── examples/ # Working actors
└── README.md # This file
```
## Best Practices Applied
This skill follows Anthropic's official best practices for skill development:
### 1. Progressive Disclosure Architecture ✓
**Pattern**: Three-level loading system to manage context efficiently
- **Level 1**: YAML frontmatter (~85 tokens) - Always loaded
- **Level 2**: Main SKILL.md (~356 lines) - Loaded when skill invoked
- **Level 3**: Subdirectories - Loaded on-demand as needed
**Result**: 70-80% token reduction vs monolithic documentation
**Source**: [skill-creator/SKILL.md](https://github.com/anthropics/skills/blob/main/skill-creator/SKILL.md#progressive-disclosure-design-principle)
### 2. Imperative/Infinitive Form Writing Style ✓
**Pattern**: Write instructions using verb-first commands, not second-person language
**Examples**:
- ✅ "Load this workflow when user requests"
- ✅ "Check for sitemaps automatically"
- ❌ "You should load this workflow"
- ❌ "You need to check for sitemaps"
**Exception**: Second-person is acceptable in user-facing prompts, code comments, and tutorial examples
**Source**: [skill-creator/SKILL.md](https://github.com/anthropics/skills/blob/main/skill-creator/SKILL.md#update-skillmd)
### 3. Clear YAML Frontmatter ✓
**Pattern**: Concise, specific name and description that determine when Claude invokes the skill
**Applied**:
- `name: web-scraping` - Clear, hyphen-case identifier
- `description:` - Specific about activation triggers and capabilities (189 chars, optimized from 244)
**Source**: [agent_skills_spec.md](https://github.com/anthropics/skills/blob/main/agent_skills_spec.md#yaml-frontmatter)
### 4. Lean SKILL.md with Reference Files ✓
**Pattern**: Keep only essential procedural instructions in SKILL.md; move detailed information to subdirectories
**Applied**:
- SKILL.md: Core 4-phase workflow (~356 lines)
- `workflows/`: Detailed implementation patterns
- `strategies/`: Deep-dive guides
- `examples/`: Runnable code
- `reference/`: Quick lookup patterns
- `apify/`: Production deployment guides
**Source**: [skill-creator/SKILL.md](https://github.com/anthropics/skills/blob/main/skill-creator/SKILL.md#references-references)
### 5. Scripts, References, and Assets Organization ✓
**Pattern**: Separate executable code, documentation, and output resources
**Applied**:
- `examples/` - Executable JavaScript learning examples (like scripts/)
- `workflows/`, `strategies/`, `reference/`, `apify/` - Documentation loaded as needed (like references/)
- `apify/templates/`, `apify/examples/` - Boilerplate code and templates (like assets/)
**Source**: [skill-creator/SKILL.md](https://github.com/anthropics/skills/blob/main/skill-creator/SKILL.md#bundled-resources-optional)
### 6. Purpose-Driven Skill Scope ✓
**Pattern**: Create focused skills for specific purposes rather than one skill that does everything
**Applied**: This skill focuses specifically on web scraping and Apify Actor development, not general web development
**Source**: [Anthropic Skills Best Practices](https://www.anthropic.com/news/skills)
### 7. Objective, Instructional Language ✓
**Pattern**: Use clear, technical language focused on "what" and "how" rather than persuasive or promotional tone
**Applied**: Direct technical guidance throughout ("Check for sitemaps", "Implement iteratively") vs. marketing language
**Source**: [skill-creator/SKILL.md](https://github.com/anthropics/skills/blob/main/skill-creator/SKILL.md#update-skillmd)
## Key Features
### 1. Adaptive Reconnaissance (Phases 0-5)
Quality-gated workflow that skips unnecessary phases:
- **Phase 0**: curl-based assessment — detect framework, search for data, check protections
- **Phase 1**: Browser only if needed — stealth Chrome, traffic capture, rendered DOM
- **Phase 2**: Deep scan only for missing data — targeted interactions, framework-aware API sniffing
- **Phase 3**: Validate every finding — test selectors, replay APIs, confirm JSON paths
- **Phase 4**: Protection testing only if signals warrant — conditional escalation
- **Phase 5**: Self-critiquing report — gaps, assumptions, staleness warnings
### 2. Framework-Aware Detection
Uses `strategies/framework-signatures.md` lookup tables:
- Response headers → framework identification
- HTML signatures → data location mapping
- Known major sites → direct strategy (e.g., Amazon: custom SSR, no JSON-LD)
- Detect first, then search only relevant patterns
### 3. Validated Intelligence Reports
Reports follow `reference/report-schema.md` with:
- `Validated?` column for every extraction strategy (YES / PARTIAL / NO)
- Self-Critique section: gaps, skipped steps, assumptions, staleness risk
- Targeted re-investigation for fixable gaps
### 4. Iterative Implementation (Phase 4)
- Start with simplest approach
- Test small batch (5-10 items)
- Scale or fallback based on results
- Add robustness last
### 5. TypeScript-First Apify (Phase 5)
For production actors:
- **Strongly recommend** TypeScript
- **Always use** `apify create` command
- **Choose template** based on site type (Cheerio for static, Playwright for JS-heavy)
- Type-safe input/output
## Example Workflows
### Workflow 1: Unknown Site
```
1. User: "Scrape example.com"
2. Phase 0: curl raw HTML → detect Next.js (__NEXT_DATA__), find product data in JSON
3. GATE A: All data in __NEXT_DATA__? → YES → Skip browser
4. Phase 3: Validate JSON paths resolve to expected values
5. Phase 5: Generate report with self-critique
6. Result: No browser needed, Cheerio + JSON parsing sufficient
```
### Workflow 1b: Site Needing Browser
```
1. User: "Scrape protected-shop.com"
2. Phase 0: curl returns 403 → protection detected, no data in HTML
3. GATE A: NO → Continue to Phase 1
4. Phase 1: Stealth browser loads page, traffic reveals API endpoint
5. GATE B: All data covered via API → Skip Phase 2
6. Phase 3: Replay API request, validate response structure
7. Phase 4: Protection testing (403 was detected) → stealth browser + proxy needed
8. Phase 5: Report + self-critique
9. Implements with discovered API + upstream proxies
10. Tests with 10 items, scales to full dataset
```
### Workflow 2: Make it an Actor
```
1. User: "Make this an Apify Actor"
2. Claude loads apify/ module
3. Recommends TypeScript? (Yes)
4. Guides through: apify create
5. Analyzes site: Static HTML → Selects Cheerio template
6. Ports scraping logic to TypeScript
7. Adds input schema
8. Tests: apify run
9. Deploys: apify push
10. Result: Production-ready actor
```
## Performance Benefits
| Approach | Time (1000 pages) | vs Crawling |
|----------|-------------------|-------------|
| Sitemap + API | 5 minutes | 60x faster |
| Sitemap + Playwright | 20 minutes | 15x faster |
| API only | 8 minutes | 40x faster |
| Playwright crawl | 45 minutes | Baseline |
## Best Practices Summary
### Reconnaissance (Phases 0-5)
- Start with curl (Phase 0) before launching browser
- Detect framework first, then search relevant patterns only
- Quality gates skip phases when data is sufficient
- Validate every selector/path/API before reporting
- Self-critique: check for gaps, assumptions, staleness
- Protection testing only when signals warrant it
### Implementation Phase (Phase 4)
- Start simple (traffic interception → sitemap → API → DOM scraping)
- Test small batch first
- Handle errors gracefully
- Respect rate limits
### Production Phase (Phase 5)
- Use TypeScript for Apify Actors
- Always use `apify create` command
- Choose template based on Phase 1 findings (Cheerio vs Playwright)
- Test locally with `apify run`
- Deploy with `apify push`
## Troubleshooting
### "No URLs found in sitemap"
→ See `strategies/sitemap-discovery.md` troubleshooting section
### "API requires authentication"
→ See `strategies/api-discovery.md` authentication section
### "DOM scraping too slow"
→ See `strategies/dom-scraping.md` and consider API discovered via traffic capture
### "Actor deployment fails"
→ See `apify/cli-workflow.md` common issues section
## Resources
- **Main skill**: Read `SKILL.md` for complete workflow
- **Workflows**: Implementation patterns in `workflows/`
- **Strategies**: Browse `strategies/` for detailed guides
- **Examples**: Run code in `examples/` directory
- **Reference**: Quick lookups in `reference/`
- **Apify**: Production deployment in `apify/`
## Philosophy
**Intelligence first, implementation second!**
This skill prioritizes:
1. **Reconnaissance** - Understand before coding (APIs > Sitemaps > Scraping)
2. **Speed** - Fastest approach that works (API 10-100x faster than HTML)
3. **Reliability** - Structured data > HTML parsing
4. **Maintainability** - TypeScript, proper tooling
5. **Best practices** - Industry standards
## Version
**5.0.0** - Traffic-interception-first scraping:
- **NEW**: Proxy-MCP integration (MITM traffic interception + stealth browser + humanizer)
- **NEW**: Automatic API discovery via traffic capture
- **NEW**: Multi-layer anti-detection (stealth mode, humanizer, upstream proxies, TLS spoofing)
- **NEW**: Session recording and HAR export/replay
- Progressive disclosure architecture
- Proactive strategy discovery
- TypeScript-first Apify guidance
- Comprehensive examples
- Modular organization
## References
All best practices sourced from official Anthropic documentation:
- [Anthropic Skills Repository](https://github.com/anthropics/skills)
- [Agent Skills Specification](https://github.com/anthropics/skills/blob/main/agent_skills_spec.md)
- [skill-creator/SKILL.md](https://github.com/anthropics/skills/blob/main/skill-creator/SKILL.md)
- [Anthropic Skills Announcement](https://www.anthropic.com/news/skills)
---
**Start here**: Read `SKILL.md` for the complete proactive workflow.