io.github.carrasquelalex1/webscrape-mcp
Fetch any URL and get clean Markdown. Web scraping for AI agents.
Open source Repository Open in the app JSON README (API)
About
Fetch any URL and get clean Markdown. Web scraping for AI agents.
Details
- Kind
- MCP servers
- Topic
- Files & documents
- Publisher
- carrasquelalex1
- Origin
- official
- Category
- ferramentas
- Transport
- http
- Version
- 1.0.0
- Stars
- 1
- Open pull requests
- 1
- Last push
- 2026-06-28T17:58:24Z
- Repository state
- ativo
- Language
- Python
- Added
- 2026-08-29 03:02:33
- Updated
- 2026-08-29 03:02:33
- Origin id
io.github.carrasquelalex1/webscrape-mcp
README
# WebScrape MCP Server
English · [Español](#español)
---
## English
MCP server that lets AI agents search the web and extract clean Markdown content — no ads, no clutter, just the text your LLM needs.
### What's New in v2.0.0
**New features:**
- **JS rendering** — Render JavaScript-heavy sites with Playwright (Chromium headless). Auto-fallback when httpx gets 403 or empty content
- **Structured data extraction** — Extract JSON-LD, Open Graph, Twitter Cards, meta tags, canonical URLs, and hreflang links with `extract_schema=True`
- **Screenshots** — New `webscrape_screenshot` tool captures page screenshots with configurable viewport, full-page mode, and PNG/JPEG format
- **Multi-engine search** — DuckDuckGo primary, automatic fallback to Google and Bing if DDGS is unavailable
- **Smart truncation** — Content is truncated at paragraph/sentence boundaries instead of mid-word
**Improvements:**
- Enhanced cache: 500 entries with 15-minute TTL (was 200, no expiry)
- Better error handling with specific messages for 403, 404, 429, timeouts
- Updated Dockerfile with Chromium dependencies for Playwright
### Tools
| Tool | Description |
|------|-------------|
| `webscrape_fetch_url` | Fetch a single URL and return clean Markdown. Supports `use_readability`, `js_render`, `extract_schema`, and auto-detects PDFs |
| `webscrape_batch_fetch` | Fetch up to 5 URLs in parallel. Supports PDF auto-detection, JS rendering, and structured data |
| `webscrape_search` | Search the web (DuckDuckGo → Google → Bing fallback) and scrape results into Markdown |
| `webscrape_screenshot` | Capture a screenshot of any web page with headless Chromium. Supports PNG/JPEG, viewport sizing, and full-page capture |
### Features
- **PDF support**: URLs ending in `.pdf` or with `application/pdf` content-type are auto-detected and text is extracted page by page
- **Readability mode**: Pass `use_readability=True` to `webscrape_fetch_url` for cleaner article extraction using Mozilla Readability (removes nav, sidebars, ads, comments)
- **JS rendering**: Pass `js_render=True` to render JavaScript-heavy sites with Playwright (headless Chromium). Auto-fallback when httpx gets 403 or empty content
- **Structured data extraction**: Pass `extract_schema=True` to extract JSON-LD, Open Graph, Twitter Cards, meta tags, canonical URLs, and hreflang links
- **Multi-engine search**: DuckDuckGo primary, automatic fallback to Google and Bing if DDGS is unavailable
- **Screenshots**: Capture page screenshots with configurable viewport, full-page mode, and PNG/JPEG format
- **Built-in cache**: 500-entry cache with TTL-based eviction (15 min) for repeated URLs
- **Batch fetching**: Up to 5 URLs in parallel
- **Smart truncation**: Content is truncated at paragraph/sentence boundaries instead of mid-word
### How to use
#### Option 1 — MCPize (recommended)
1. Go to https://mcpize.com/marketplace
2. Search **Web Scrape** and click **Start Free**
3. You'll get an API key
4. Configure in your AI client:
```json
{
"mcpServers": {
"webscrape": {
"url": "https://webscrape.mcpize.run",
"headers": {
"Authorization": "Bearer your-api-key"
}
}
}
}
```
#### Option 2 — Render (dev)
```json
{
"mcpServers": {
"webscrape": {
"url": "https://webscrape-mcp.onrender.com"
}
}
}
```
#### Option 3 — Local
```bash
git clone https://github.com/carrasquelalex1/webscrape-mcp.git
cd webscrape-mcp
pip install -r requirements.txt
playwright install chromium
python webscrape_mcp.py
```
### Official Registry
`io.github.carrasquelalex1/webscrape-mcp`
### Dependencies
`mcp`, `httpx`, `beautifulsoup4`, `markdownify`, `pydantic`, `ddgs`, `readability-lxml`, `PyMuPDF`, `playwright`
### License
MIT
---
## Español
Servidor MCP que permite a agentes de IA buscar en la web y extraer contenido limpio en Markdown — sin anuncios, sin navegación, solo el texto que tu LLM necesita.
### Novedades en v2.0.0
**Nuevas características:**
- **Rendering JS** — Renderiza sitios con JavaScript usando Playwright (Chromium headless). Fallback automático cuando httpx recibe 403 o contenido vacío
- **Extracción de datos estructurados** — Extrae JSON-LD, Open Graph, Twitter Cards, meta tags, URLs canónicas y links hreflang con `extract_schema=True`
- **Capturas de pantalla** — Nueva herramienta `webscrape_screenshot` que captura screenshots con viewport configurable, modo full-page y formato PNG/JPEG
- **Búsqueda multi-motor** — DuckDuckGo primario, fallback automático a Google y Bing si DDGS no está disponible
- **Truncado inteligente** — El contenido se trunca en límites de párrafo/oración en vez de cortar palabras a la mitad
**Mejoras:**
- Caché mejorada: 500 entradas con TTL de 15 minutos (antes 200, sin expiración)
- Mejor manejo de errores con mensajes específicos para 403, 404, 429, timeouts
- Dockerfile actualizado con dependencias de Chromium para Playwright
### Tools
| Tool | Descripción |
|------|-------------|
| `webscrape_fetch_url` | Obtiene una URL y la convierte a Markdown limpio. Soporta `use_readability`, `js_render`, `extract_schema`, y detecta PDFs automáticamente |
| `webscrape_batch_fetch` | Obtiene hasta 5 URLs en paralelo. Soporta detección de PDFs, rendering JS, y datos estructurados |
| `webscrape_search` | Busca en la web (DuckDuckGo → Google → Bing como fallback) y extrae los resultados a Markdown |
| `webscrape_screenshot` | Captura una captura de pantalla de cualquier página web con Chromium headless. Soporta PNG/JPEG, tamaño de viewport, y captura completa |
### Características
- **Soporte PDF**: URLs que terminan en `.pdf` o con content-type `application/pdf` se detectan automáticamente y se extrae el texto página por página
- **Modo Readability**: Usá `use_readability=True` en `webscrape_fetch_url` para extraer artículos de forma más limpia (elimina navegación, barras laterales, anuncios, comentarios)
- **Rendering JS**: Usá `js_render=True` para renderizar sitios con JavaScript usando Playwright (Chromium headless). Fallback automático cuando httpx recibe 403 o contenido vacío
- **Extracción de datos estructurados**: Usá `extract_schema=True` para extraer JSON-LD, Open Graph, Twitter Cards, meta tags, URLs canónicas, y links hreflang
- **Búsqueda multi-motor**: DuckDuckGo primario, fallback automático a Google y Bing si DDGS no está disponible
- **Capturas de pantalla**: Captura de páginas con viewport configurable, modo full-page, y formato PNG/JPEG
- **Caché integrada**: 500 entradas con evicción por TTL (15 min) para URLs repetidas
- **Batch fetching**: Hasta 5 URLs en paralelo
- **Truncado inteligente**: El contenido se trunca en límites de párrafo/oración en vez de cortar palabras a la mitad
### Cómo usarlo
#### Opción 1 — MCPize (recomendada)
1. Ve a https://mcpize.com/marketplace
2. Busca **Web Scrape** y haz clic en **Start Free**
3. Obtendrás una API key
4. Configura en tu cliente de IA:
```json
{
"mcpServers": {
"webscrape": {
"url": "https://webscrape.mcpize.run",
"headers": {
"Authorization": "Bearer tu-api-key"
}
}
}
}
```
#### Opción 2 — Render (desarrollo)
```json
{
"mcpServers": {
"webscrape": {
"url": "https://webscrape-mcp.onrender.com"
}
}
}
```
#### Opción 3 — Local
```bash
git clone https://github.com/carrasquelalex1/webscrape-mcp.git
cd webscrape-mcp
pip install -r requirements.txt
playwright install chromium
python webscrape_mcp.py
```
### Registro Oficial
`io.github.carrasquelalex1/webscrape-mcp`
### Dependencias
`mcp`, `httpx`, `beautifulsoup4`, `markdownify`, `pydantic`, `ddgs`, `readability-lxml`, `PyMuPDF`, `playwright`
### Licencia
MIT