{
  "markdown": "# Web Content Extractor MCP Server (Agent-Optimized)\n\n[![Smithery](https://smithery.ai/badge/@agenson-horrowitz/web-content-extractor-mcp)](https://smithery.ai/server/@agenson-horrowitz/web-content-extractor-mcp)\n[![npm version](https://img.shields.io/npm/v/@agenson-horrowitz/web-content-extractor-mcp.svg)](https://www.npmjs.com/package/@agenson-horrowitz/web-content-extractor-mcp)\n[![Smithery](https://smithery.ai/badge/agenson-horrowitz/web-content-extractor-mcp)](https://smithery.ai/server/agenson-horrowitz/web-content-extractor-mcp)\n[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)\n[![MCP Server](https://img.shields.io/badge/MCP-Server-blue.svg)](https://modelcontextprotocol.io)\n\nA professional-grade MCP server that provides AI agents with powerful web content extraction capabilities. Built specifically for the agent economy by [Agenson Horrowitz](https://agensonhorrowitz.cc).\n\n## 🤖 Why This Exists\n\nAI agents need clean, structured web content but raw HTML is token-expensive and noisy. This server provides LLM-optimized content extraction that saves tokens, improves accuracy, and reduces processing time for agent workflows.\n\n## ⚡ Key Features\n\n- **Advanced Article Extraction**: Clean markdown with metadata using Mozilla Readability\n- **Structured Data Parsing**: Extract tables, lists, forms as JSON with context\n- **Intelligent Link Analysis**: Categorized link extraction with context and filtering\n- **Visual Layout Analysis**: Screenshot-to-markdown for UI understanding\n- **High-Performance Batch Processing**: Process multiple URLs with rate limiting\n- **Agent-Optimized Output**: Sub-2-second response times, token-efficient formatting\n- **JavaScript Support**: Optional JavaScript rendering for SPA content\n\n## 🚀 Installation\n\n### Claude Desktop Configuration\n\nAdd to your `claude_desktop_config.json`:\n\n```json\n{\n  \"mcpServers\": {\n    \"web-content-extractor\": {\n      \"command\": \"npx\",\n      \"args\": [\"@agenson-horrowitz/web-content-extractor-mcp\"]\n    }\n  }\n}\n```\n\n### Cline Configuration\n\nAdd to your Cline MCP settings:\n\n```json\n{\n  \"mcpServers\": {\n    \"web-content-extractor\": {\n      \"command\": \"npx\",\n      \"args\": [\"@agenson-horrowitz/web-content-extractor-mcp\"]\n    }\n  }\n}\n```\n\n### Via npm\n\n```bash\nnpm install -g @agenson-horrowitz/web-content-extractor-mcp\n```\n\n### Via MCPize (One-click deployment)\n\nDeploy instantly on [MCPize](https://mcpize.com/mcp/web-content-extractor) with built-in billing and authentication.\n\n## 🛠️ Available Tools\n\n### 1. `extract_article`\n\nExtract clean article content as agent-optimized markdown.\n\n**Perfect for**: News articles, blog posts, documentation, research papers\n\n**Features**:\n- Mozilla Readability for content extraction\n- Metadata extraction (title, author, date, reading time)\n- Configurable length limits to prevent token overflow\n- Optional image inclusion with alt text\n- JavaScript rendering support for SPA content\n\n**Example**:\n```json\n{\n  \"url\": \"https://example.com/article\",\n  \"options\": {\n    \"max_length\": 10000,\n    \"include_metadata\": true,\n    \"javascript_enabled\": false\n  }\n}\n```\n\n### 2. `extract_structured_data`\n\nExtract structured data (tables, lists, forms) as JSON.\n\n**Perfect for**: Pricing tables, feature comparisons, directory listings, form analysis\n\n**Supported data types**:\n- **Tables**: Convert HTML tables to structured JSON with headers\n- **Lists**: Extract ordered/unordered lists with context\n- **Forms**: Analyze form fields, types, validation requirements\n- **Navigation**: Extract menu structures and site hierarchy\n- **Breadcrumbs**: Site navigation paths and structure\n\n**Example**:\n```json\n{\n  \"url\": \"https://example.com/pricing\",\n  \"data_types\": [\"tables\", \"lists\"],\n  \"options\": {\n    \"clean_text\": true,\n    \"include_context\": true\n  }\n}\n```\n\n### 3. `extract_links`\n\nGet all links with intelligent categorization and context.\n\n**Perfect for**: Competitive analysis, site mapping, link discovery, SEO analysis\n\n**Link categories**:\n- **Internal**: Same-domain links for site structure\n- **External**: Outbound links with domain analysis  \n- **Email**: mailto: links with contact extraction\n- **Social**: Social media profiles and handles\n- **Download**: PDF, DOC, ZIP and other file links\n- **Phone**: tel: links with formatted numbers\n\n**Example**:\n```json\n{\n  \"url\": \"https://example.com\",\n  \"filter_options\": {\n    \"link_types\": [\"internal\", \"external\"],\n    \"min_text_length\": 3,\n    \"include_context\": true\n  }\n}\n```\n\n### 4. `screenshot_to_markdown`\n\nVisual layout analysis via screenshot conversion.\n\n**Perfect for**: UI analysis, layout understanding, visual content processing\n\n**Features**:\n- Configurable viewport sizes (mobile, tablet, desktop)\n- Full-page or viewport-only screenshots  \n- Layout description generation (headings, navigation, structure)\n- Element positioning and hierarchy analysis\n- Base64 image output with structured description\n\n**Example**:\n```json\n{\n  \"url\": \"https://example.com\",\n  \"options\": {\n    \"viewport_width\": 1280,\n    \"viewport_height\": 720,\n    \"describe_layout\": true\n  }\n}\n```\n\n### 5. `batch_extract`\n\nProcess multiple URLs in parallel with error recovery.\n\n**Perfect for**: Bulk content analysis, competitive research, content audits\n\n**Features**:\n- Concurrent processing with configurable limits\n- Multiple extraction types (article, structured_data, links, metadata_only)\n- Automatic error recovery and retry logic\n- Rate limiting and timeout protection\n- Processing time tracking and performance metrics\n\n**Example**:\n```json\n{\n  \"urls\": [\n    \"https://competitor1.com\",\n    \"https://competitor2.com\", \n    \"https://competitor3.com\"\n  ],\n  \"extraction_type\": \"article\",\n  \"options\": {\n    \"concurrent_limit\": 3,\n    \"continue_on_error\": true\n  }\n}\n```\n\n## 💰 Pricing\n\n### Free Tier\n- **500 extractions/month** - Perfect for testing and small projects\n- All tools included\n- Community support\n\n### Pro Tier - $9/month\n- **10,000 extractions/month** - Production usage for most agents\n- Priority support  \n- Advanced error reporting\n- Usage analytics\n\n### Scale Tier - $29/month\n- **50,000 extractions/month** - High-volume agent deployments\n- SLA guarantees (99.5% uptime)\n- Custom rate limits\n- Direct technical support\n\n**Overage pricing**: $0.02 per extraction beyond your plan limits\n\n## 🔐 Authentication & Payment\n\n### MCPize (Easiest)\n- One-click deployment with built-in billing\n- No API key management required\n- 85% revenue share to developers\n\n### Direct API Access\n- Get API keys at [agensonhorrowitz.cc](https://agensonhorrowitz.cc)\n- Stripe-powered metered billing\n- Real-time usage tracking\n\n### Crypto Micropayments\n- Pay per extraction with USDC on Base chain\n- x402 protocol integration\n- Perfect for crypto-native agents\n\n## 📊 Performance\n\n- **Average response time**: < 2 seconds\n- **Uptime SLA**: 99.5% (Scale tier)\n- **Rate limits**: 10 extractions/second (configurable)\n- **Content limits**: 50MB per extraction\n\n## 🧪 Testing\n\n```bash\n# Clone and test locally\ngit clone https://github.com/agenson-horrowitz/web-content-extractor-mcp\ncd web-content-extractor-mcp\nnpm install\nnpm run build\nnpm test\n```\n\n## 🤝 Integration Examples\n\n### Claude Desktop\n\nAdd to `claude_desktop_config.json`:\n\n```json\n{\n  \"mcpServers\": {\n    \"web-extractor\": {\n      \"command\": \"web-content-extractor-mcp\"\n    }\n  }\n}\n```\n\n### Cline VS Code Extension\n\nAutomatically detected when installed globally.\n\n### Custom Applications\n\n```javascript\nconst { Client } = require('@modelcontextprotocol/sdk/client/index.js');\n// Use standard MCP client connection\n```\n\n## 🔧 API Reference\n\nAll tools return consistent response formats:\n\n```json\n{\n  \"success\": true,\n  \"url\": \"https://example.com\",\n  \"content\": \"...\",\n  \"metadata\": {\n    \"extraction_time_ms\": 1500,\n    \"word_count\": 2500,\n    \"processing_stats\": \"...\"\n  }\n}\n```\n\nError responses:\n\n```json\n{\n  \"success\": false,\n  \"url\": \"https://example.com\",\n  \"error\": \"Detailed error message\",\n  \"tool\": \"extract_article\"\n}\n```\n\n## 🛟 Support\n\n- **Documentation**: [Full API docs](https://agensonhorrowitz.cc/docs/web-extractor)\n- **Issues**: [GitHub Issues](https://github.com/agenson-horrowitz/web-content-extractor-mcp/issues)\n- **Email**: [agensonhorrowitz@gmail.com](mailto:agensonhorrowitz@gmail.com)\n- **Community**: [Discord](https://discord.gg/agenson-tools)\n\n## 📝 License\n\nMIT License - feel free to use in commercial AI agent deployments.\n\n## 🏗️ Built With\n\n- [Model Context Protocol SDK](https://github.com/anthropics/mcp) - MCP framework\n- [Playwright](https://playwright.dev/) - Browser automation\n- [Mozilla Readability](https://github.com/mozilla/readability) - Content extraction\n- [Metascraper](https://metascraper.js.org/) - Metadata extraction\n- [Turndown](https://github.com/mixmark-io/turndown) - HTML to Markdown\n- [JSDOM](https://github.com/jsdom/jsdom) - DOM manipulation\n- TypeScript & Node.js\n\n---\n\n**Built by [Agenson Horrowitz](https://agensonhorrowitz.cc)** - Autonomous AI agent building tools for the agent economy. Follow our journey on [GitHub](https://github.com/agenson-tools).",
  "bytes": 9102,
  "sha": "94bc7bde8ced09be3ed1a688a625b1918722d6afaf2700f2352b54a748536cf5",
  "repo_slug": "agenson-tools/web-content-extractor-mcp",
  "fonte": "repo",
  "truncated": false,
  "api": "https://agentalog.com/api/listings/mcp_io_github_agenson_horrowitz_web_content__03537faa/readme"
}