{
  "markdown": "# mcp-server-web-fetcher\n\n[![CI](https://github.com/vojtisprime11/mcp-server-web-fetcher/actions/workflows/ci.yml/badge.svg)](https://github.com/vojtisprime11/mcp-server-web-fetcher/actions/workflows/ci.yml)\n[![npm version](https://img.shields.io/npm/v/mcp-server-web-fetcher.svg)](https://www.npmjs.com/package/mcp-server-web-fetcher)\n[![License: MIT](https://img.shields.io/badge/license-MIT-yellow.svg)](LICENSE)\n[![TypeScript](https://img.shields.io/badge/TypeScript-strict-3178c6.svg)](tsconfig.json)\n[![Node.js](https://img.shields.io/badge/node-%3E%3D20.18.1-5fa04e.svg)](https://nodejs.org)\n[![MCP](https://img.shields.io/badge/MCP-1.30-6b4fbb.svg)](https://modelcontextprotocol.io)\n[![MCP Registry](https://img.shields.io/badge/MCP%20Registry-listed-6b4fbb.svg)](https://registry.modelcontextprotocol.io/)\n\nA fast, dependency-light [Model Context Protocol](https://modelcontextprotocol.io) server that turns\nany web page into something a language model can actually read: clean Markdown, structured metadata,\nand a classified list of links.\n\nWeb pages are 90% chrome. Scripts, cookie banners, navigation, sidebars and tracking pixels burn\ncontext and derail reasoning. This server strips all of that on the way in, so the model sees the\narticle and nothing else.\n\n```\n┌────────────┐   stdio/JSON-RPC   ┌──────────────────────┐   HTTPS   ┌──────────┐\n│ MCP client │ ─────────────────► │ mcp-server-web-      │ ────────► │ web page │\n│ (Claude,   │ ◄───────────────── │ fetcher              │ ◄──────── │          │\n│  Kiro, …)  │  Markdown + JSON   │ fetch → clean → md   │           └──────────┘\n└────────────┘                    └──────────────────────┘\n```\n\n## Features\n\n- **Clean Markdown output.** Readability-style main-content detection, GFM tables, language-tagged\n  code fences, absolute links. No leftover HTML, even for gnarly nested tables.\n- **Context-window aware.** Long pages are paginated with `startIndex` / `nextStartIndex` instead of\n  being silently cut in half.\n- **Structured metadata.** Title, description, canonical, `lang`, author, publish/modify dates,\n  Open Graph, Twitter cards, JSON-LD, `hreflang` alternates, RSS/Atom feeds, h1–h6 outline and raw\n  HTTP headers.\n- **Link intelligence.** Absolute URLs, anchor text, `rel`, `nofollow`, internal vs external\n  classification, scope filters, de-duplication.\n- **SSRF hardening.** Loopback, private, link-local and cloud-metadata addresses are blocked on\n  _every_ redirect hop, not just the first request. Credentials in URLs are stripped.\n- **Predictable failures.** Every error carries a stable machine-readable `code`\n  (`TIMEOUT`, `HTTP_ERROR`, `BLOCKED_HOST`, `RESPONSE_TOO_LARGE`, …), a retryability flag and a\n  recovery hint, so the model can self-correct instead of guessing.\n- **Typed end to end.** Strict Zod input schemas plus published `outputSchema`, so clients get\n  validated `structuredContent`, not prose they have to re-parse.\n- **Well behaved.** Byte caps, timeouts, redirect limits, charset sniffing (including legacy\n  encodings), a small TTL cache, transient-failure retries and optional `robots.txt` enforcement.\n- **115 tests, no network required.** Vitest unit tests plus in-memory MCP protocol tests.\n\n## Quickstart\n\nRequires Node.js 20.18.1 or newer (inherited from cheerio, which needs undici 7).\n\n```bash\n# run it without installing\nnpx -y mcp-server-web-fetcher\n\n# or install globally\nnpm install -g mcp-server-web-fetcher\nmcp-server-web-fetcher\n```\n\nThe server speaks MCP over stdio, so on its own it just waits for a client. Point a client at it:\n\n### Claude Desktop\n\n`~/Library/Application Support/Claude/claude_desktop_config.json` on macOS,\n`%APPDATA%\\Claude\\claude_desktop_config.json` on Windows:\n\n```json\n{\n  \"mcpServers\": {\n    \"web-fetcher\": {\n      \"command\": \"npx\",\n      \"args\": [\"-y\", \"mcp-server-web-fetcher\"]\n    }\n  }\n}\n```\n\nWith configuration, using a global install:\n\n```json\n{\n  \"mcpServers\": {\n    \"web-fetcher\": {\n      \"command\": \"mcp-server-web-fetcher\",\n      \"env\": {\n        \"WEB_FETCHER_TIMEOUT_MS\": \"20000\",\n        \"WEB_FETCHER_RESPECT_ROBOTS\": \"true\"\n      }\n    }\n  }\n}\n```\n\nRestart Claude Desktop, then ask it to summarise a URL.\n\n### Claude Code\n\n```bash\nclaude mcp add web-fetcher -- npx -y mcp-server-web-fetcher\n```\n\n### Kiro, Cursor, Windsurf and other MCP clients\n\nSame shape, in `.kiro/settings/mcp.json` / the client's MCP config file:\n\n```json\n{\n  \"mcpServers\": {\n    \"web-fetcher\": {\n      \"command\": \"npx\",\n      \"args\": [\"-y\", \"mcp-server-web-fetcher\"],\n      \"disabled\": false,\n      \"autoApprove\": [\"fetch_page_markdown\", \"extract_metadata\", \"extract_links\"]\n    }\n  }\n}\n```\n\n### MCP Registry\n\nThe server is listed in the [official MCP Registry](https://registry.modelcontextprotocol.io/)\nas `io.github.vojtisprime11/web-fetcher`, so clients that read the registry can discover it\ndirectly:\n\n```bash\ncurl \"https://registry.modelcontextprotocol.io/v0.1/servers?search=io.github.vojtisprime11/web-fetcher\"\n```\n\n### Docker\n\n```bash\ndocker build -t mcp-server-web-fetcher .\n```\n\n```json\n{\n  \"mcpServers\": {\n    \"web-fetcher\": {\n      \"command\": \"docker\",\n      \"args\": [\"run\", \"--rm\", \"-i\", \"mcp-server-web-fetcher\"]\n    }\n  }\n}\n```\n\nThe image runs as the unprivileged `node` user and carries production dependencies only. CI builds\nit and drives it with a real MCP client, so it is verified to start and answer introspection.\n\n### From source\n\n```bash\ngit clone https://github.com/vojtisprime11/mcp-server-web-fetcher.git\ncd mcp-server-web-fetcher\nnpm install\nnpm run build\nnode dist/index.js          # or: npm run inspect\n```\n\n```json\n{\n  \"mcpServers\": {\n    \"web-fetcher\": {\n      \"command\": \"node\",\n      \"args\": [\"/absolute/path/to/mcp-server-web-fetcher/dist/index.js\"]\n    }\n  }\n}\n```\n\n## Tools\n\nAll three tools are read-only, idempotent and open-world (annotated as such in the protocol), and\nall take an absolute `http(s)` URL.\n\n### `fetch_page_markdown`\n\nDownloads a page and returns clean, LLM-friendly Markdown.\n\n| Parameter         | Type                | Default       | Description                                                    |\n| ----------------- | ------------------- | ------------- | -------------------------------------------------------------- |\n| `url`             | string              | required      | Absolute http(s) URL.                                          |\n| `maxLength`       | integer 500–1000000 | `25000`       | Markdown characters to return per call.                        |\n| `startIndex`      | integer ≥ 0         | `0`           | Character offset; use `nextStartIndex` from the previous call. |\n| `mainContentOnly` | boolean             | `true`        | Drop nav/header/footer/sidebar, keep the densest block.        |\n| `includeLinks`    | boolean             | `true`        | Keep Markdown links (`false` inlines the text only).           |\n| `includeImages`   | boolean             | `false`       | Keep images as `![alt](src)`.                                  |\n| `includeMetadata` | boolean             | `true`        | Attach a short metadata summary.                               |\n| `timeoutMs`       | integer 1000–120000 | env / `15000` | Per-request timeout.                                           |\n\nRequest:\n\n```json\n{\n  \"name\": \"fetch_page_markdown\",\n  \"arguments\": {\n    \"url\": \"https://example.com/blog/caching\",\n    \"maxLength\": 8000,\n    \"mainContentOnly\": true,\n    \"includeImages\": false\n  }\n}\n```\n\n`structuredContent`:\n\n```json\n{\n  \"url\": \"https://example.com/blog/caching\",\n  \"requestedUrl\": \"https://example.com/blog/caching\",\n  \"status\": 200,\n  \"contentType\": \"text/html; charset=utf-8\",\n  \"title\": \"How caching works\",\n  \"markdown\": \"# How caching works\\n\\nCaching is the art of **not** doing work twice...\",\n  \"markdownLength\": 7984,\n  \"totalLength\": 21874,\n  \"startIndex\": 0,\n  \"endIndex\": 7984,\n  \"nextStartIndex\": 7984,\n  \"truncated\": true,\n  \"wordCount\": 1203,\n  \"bytesDownloaded\": 148213,\n  \"elapsedMs\": 412,\n  \"fromCache\": false,\n  \"redirects\": [],\n  \"metadata\": {\n    \"title\": \"How caching works\",\n    \"description\": \"A deep dive into HTTP caching.\",\n    \"canonical\": \"https://example.com/blog/caching\",\n    \"language\": \"en\",\n    \"author\": \"Ada Lovelace\",\n    \"publishedTime\": \"2026-01-15T09:00:00Z\"\n  }\n}\n```\n\nTo read the rest, call again with `\"startIndex\": 7984`. Keep going while `nextStartIndex` is not\n`null`.\n\n### `extract_metadata`\n\nEverything a model needs to classify a page, without spending context on its body.\n\n| Parameter            | Type                | Default       | Description                                                 |\n| -------------------- | ------------------- | ------------- | ----------------------------------------------------------- |\n| `url`                | string              | required      | Absolute http(s) URL.                                       |\n| `includeJsonLd`      | boolean             | `true`        | Include parsed JSON-LD blocks (malformed ones are skipped). |\n| `includeHeadings`    | boolean             | `true`        | Include the h1–h6 outline.                                  |\n| `includeHttpHeaders` | boolean             | `true`        | Include response headers, lower-cased.                      |\n| `timeoutMs`          | integer 1000–120000 | env / `15000` | Per-request timeout.                                        |\n\n```json\n{\n  \"name\": \"extract_metadata\",\n  \"arguments\": { \"url\": \"https://example.com/blog/caching\", \"includeJsonLd\": true }\n}\n```\n\n`structuredContent` (abridged):\n\n```json\n{\n  \"url\": \"https://example.com/blog/caching\",\n  \"status\": 200,\n  \"charset\": \"utf-8\",\n  \"title\": \"How caching works\",\n  \"description\": \"A deep dive into HTTP caching.\",\n  \"canonical\": \"https://example.com/blog/caching\",\n  \"language\": \"en\",\n  \"author\": \"Ada Lovelace\",\n  \"publishedTime\": \"2026-01-15T09:00:00Z\",\n  \"modifiedTime\": null,\n  \"robots\": \"index, follow\",\n  \"favicon\": \"https://example.com/favicon.ico\",\n  \"openGraph\": {\n    \"title\": \"How caching works\",\n    \"type\": \"article\",\n    \"image\": \"https://example.com/img/cover.png\"\n  },\n  \"twitter\": { \"card\": \"summary_large_image\", \"site\": \"@example\" },\n  \"alternates\": [{ \"hreflang\": \"de\", \"href\": \"https://example.com/de/blog/caching\" }],\n  \"feeds\": [\n    { \"title\": \"Feed\", \"href\": \"https://example.com/feed.xml\", \"type\": \"application/rss+xml\" }\n  ],\n  \"jsonLd\": [{ \"@type\": \"Article\", \"headline\": \"How caching works\" }],\n  \"headings\": [\n    { \"level\": 1, \"text\": \"How caching works\", \"id\": null },\n    { \"level\": 2, \"text\": \"Directives\", \"id\": \"directives\" }\n  ],\n  \"httpHeaders\": { \"content-type\": \"text/html; charset=utf-8\", \"x-cache\": \"HIT\" },\n  \"wordCount\": 1203,\n  \"redirects\": [],\n  \"fromCache\": false\n}\n```\n\n### `extract_links`\n\n| Parameter        | Type                              | Default       | Description                        |\n| ---------------- | --------------------------------- | ------------- | ---------------------------------- |\n| `url`            | string                            | required      | Absolute http(s) URL.              |\n| `scope`          | `all` \\| `internal` \\| `external` | `all`         | Same-site, off-site, or both.      |\n| `includeAnchors` | boolean                           | `false`       | Include in-page `#fragment` links. |\n| `deduplicate`    | boolean                           | `true`        | Collapse repeated URLs.            |\n| `limit`          | integer 1–2000                    | `200`         | Maximum links returned.            |\n| `timeoutMs`      | integer 1000–120000               | env / `15000` | Per-request timeout.               |\n\n```json\n{\n  \"name\": \"extract_links\",\n  \"arguments\": { \"url\": \"https://example.com/blog/caching\", \"scope\": \"external\", \"limit\": 50 }\n}\n```\n\n`structuredContent`:\n\n```json\n{\n  \"url\": \"https://example.com/blog/caching\",\n  \"status\": 200,\n  \"totalFound\": 2,\n  \"returned\": 2,\n  \"internalCount\": 0,\n  \"externalCount\": 2,\n  \"truncated\": false,\n  \"links\": [\n    {\n      \"url\": \"https://developer.mozilla.org/en-US/docs/Web/HTTP/Caching\",\n      \"text\": \"MDN article\",\n      \"title\": null,\n      \"rel\": \"noopener nofollow\",\n      \"internal\": false,\n      \"nofollow\": true\n    }\n  ],\n  \"fromCache\": false\n}\n```\n\n`www.example.com` and `example.com` count as the same site; `blog.example.com` does not.\n\n## Error handling\n\nFailures come back as MCP tool errors (`isError: true`), never as silent empty results:\n\n```\nHTTP_ERROR: 404 Client Error for https://example.com/missing\nRetryable: no\nHint: Check the URL, or retry later if the status is 429/5xx.\n```\n\n`structuredContent.error` carries the same information as JSON: `{ code, message, retryable, url, status }`.\n\n| Code                       | Meaning                                                         |\n| -------------------------- | --------------------------------------------------------------- |\n| `INVALID_URL`              | Not an absolute URL.                                            |\n| `BLOCKED_SCHEME`           | Scheme other than `http`/`https`.                               |\n| `BLOCKED_HOST`             | Loopback, private, link-local or metadata address (SSRF guard). |\n| `DNS_FAILURE`              | Host does not resolve.                                          |\n| `TIMEOUT`                  | Request exceeded `timeoutMs`.                                   |\n| `HTTP_ERROR`               | Response status ≥ 400, or a redirect without `Location`.        |\n| `TOO_MANY_REDIRECTS`       | Redirect chain longer than the limit.                           |\n| `RESPONSE_TOO_LARGE`       | Declared body larger than the byte cap.                         |\n| `UNSUPPORTED_CONTENT_TYPE` | Not a text/HTML/XML/JSON document.                              |\n| `ROBOTS_DISALLOWED`        | Blocked by `robots.txt` (only when enforcement is on).          |\n| `NETWORK_ERROR`            | Connection reset, TLS failure, and similar.                     |\n| `PARSE_ERROR`              | Document could not be parsed.                                   |\n\nTimeouts, 5xx, 408 and 429 responses are retried once automatically before the error surfaces.\n\n## Configuration\n\nEverything is optional; the defaults are safe.\n\n| Variable                          | Default                                        | Description                                                |\n| --------------------------------- | ---------------------------------------------- | ---------------------------------------------------------- |\n| `WEB_FETCHER_USER_AGENT`          | `mcp-server-web-fetcher/<version> (+repo url)` | Outgoing `User-Agent`.                                     |\n| `WEB_FETCHER_TIMEOUT_MS`          | `15000`                                        | Default timeout (1000–120000).                             |\n| `WEB_FETCHER_MAX_BYTES`           | `5000000`                                      | Per-response byte cap (10000–50000000).                    |\n| `WEB_FETCHER_MAX_REDIRECTS`       | `5`                                            | Redirect hops allowed (0–20).                              |\n| `WEB_FETCHER_ALLOW_PRIVATE_HOSTS` | `false`                                        | Set to `true` only to fetch localhost/LAN URLs on purpose. |\n| `WEB_FETCHER_RESPECT_ROBOTS`      | `false`                                        | Enforce `robots.txt` (RFC 9309 subset) before fetching.    |\n| `WEB_FETCHER_CACHE_TTL_MS`        | `60000`                                        | Response cache TTL; `0` disables caching.                  |\n| `WEB_FETCHER_CACHE_MAX_ENTRIES`   | `50`                                           | Maximum cached responses.                                  |\n\n## Security\n\n- **SSRF guard on by default.** Hostnames are resolved and checked against loopback, RFC 1918,\n  CGNAT, link-local (including `169.254.169.254`), multicast and IPv4-mapped IPv6 ranges — for the\n  initial URL _and_ every redirect target. `localhost`, `*.localhost` and `*.internal` are refused\n  outright. Turning the guard off is an explicit opt-in.\n- **Byte and time caps.** Responses are streamed and cut at the byte cap; every request is aborted\n  at the timeout. Content-Length larger than the cap is rejected before download.\n- **No code execution.** JavaScript is never run; `<script>`, `<style>`, `<iframe>` and friends are\n  removed before conversion. Fetched content is data, not instructions — treat page text reaching a\n  model as untrusted input.\n- **Credentials stripped.** `user:pass@` is removed from URLs before the request is made.\n- **Read-only.** The server has no write, filesystem or shell surface, and no authentication state.\n\nFound a vulnerability? See [SECURITY.md](SECURITY.md).\n\n## Architecture\n\n```\nsrc/\n├── index.ts              # stdio entry point (logs to stderr only)\n├── server.ts             # transport-agnostic server factory + public API\n├── types.ts              # Zod input/output schemas for every tool\n├── tools/\n│   ├── fetchPageMarkdown.ts\n│   ├── extractMetadata.ts\n│   ├── extractLinks.ts\n│   ├── shared.ts         # result shaping, dependency injection\n│   └── index.ts\n└── lib/\n    ├── config.ts         # env-driven configuration\n    ├── errors.ts         # typed error codes + recovery hints\n    ├── net.ts            # URL parsing, SSRF guard, link resolution\n    ├── http.ts           # fetch pipeline: timeouts, caps, redirects, charset, cache\n    ├── robots.ts         # optional robots.txt support\n    ├── html.ts           # cheerio cleanup, metadata + link extraction\n    ├── markdown.ts       # turndown configuration, tables, pagination\n    └── cache.ts          # TTL + LRU cache\ntests/                    # unit tests + in-memory MCP protocol tests\n```\n\nTool handlers take an injectable `fetchImpl`, so every test runs offline against a scripted HTTP\nstub. The library is also importable directly:\n\n```ts\nimport { runFetchPageMarkdown } from 'mcp-server-web-fetcher';\n\nconst page = await runFetchPageMarkdown({ url: 'https://example.com' });\nconsole.log(page.markdown);\n```\n\n## Development\n\nThe minimum supported Node.js version is 20.18.1, which comes from cheerio (it depends on undici 7,\nand undici 7 needs the `File` global that landed in Node 20). CI runs the suite on Node 20/22/24 and\nhas a separate job that completes an MCP handshake against the built server on exactly 20.18.1, so\nthe declared floor is verified rather than assumed.\n\n```bash\nnpm install\nnpm run dev            # tsx watch\nnpm run typecheck\nnpm run lint\nnpm test               # vitest run\nnpm run test:coverage  # thresholds enforced\nnpm run build\nnpm run inspect        # MCP Inspector against dist/index.js\nnode scripts/smoke.mjs https://example.com   # live end-to-end check\n```\n\nContributions welcome — see [CONTRIBUTING.md](CONTRIBUTING.md) and\n[CODE_OF_CONDUCT.md](CODE_OF_CONDUCT.md).\n\n## Limitations\n\n- Static HTML only. Client-rendered SPAs return whatever the server sends; no headless browser.\n- No PDF or Office document extraction.\n- Sites behind bot protection (Cloudflare challenges, login walls) will return 403.\n- Main-content detection is a text-density heuristic. Use `mainContentOnly: false` when a page has\n  an unusual structure.\n\n## Roadmap\n\n- `search_page` tool: return only the sections matching a query.\n- Optional headless rendering behind a flag, for JS-only pages.\n- Streamable HTTP transport in addition to stdio.\n- Conditional requests (`ETag` / `If-Modified-Since`) in the cache layer.\n\n## Acknowledgements\n\nBuilt on the [MCP TypeScript SDK](https://github.com/modelcontextprotocol/typescript-sdk),\n[cheerio](https://cheerio.js.org/), [turndown](https://github.com/mixmark-io/turndown) and\n[zod](https://zod.dev/).\n\n## License\n\n[MIT](LICENSE) © Vojta Holes and contributors\n",
  "bytes": 19718,
  "sha": "b1e488d4e2e50e6a9dcf3179375f79004c948050f1a47df21afbed8aaa10e976",
  "repo_slug": "vojtisprime11/mcp-server-web-fetcher",
  "fonte": "repo",
  "truncated": false,
  "api": "https://agentalog.com/api/listings/mcp_io_github_vojtisprime11_web_fetcher_7cabc013/readme"
}