{
  "markdown": "# NetLens\n\n[![npm](https://img.shields.io/npm/v/netlens-mcp.svg)](https://www.npmjs.com/package/netlens-mcp)\n[![PyPI](https://img.shields.io/pypi/v/netlens-mcp.svg)](https://pypi.org/project/netlens-mcp/)\n[![CI](https://github.com/pzalutski-pixel/netlens-mcp/actions/workflows/ci.yml/badge.svg)](https://github.com/pzalutski-pixel/netlens-mcp/actions/workflows/ci.yml)\n[![License: Apache 2.0](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](LICENSE)\n[![Python 3.10+](https://img.shields.io/badge/python-3.10+-blue.svg)](https://www.python.org/downloads/)\n\nAn MCP server for **unobstructed web reading**. It fetches any URL directly with\nbrowser-like headers — past `robots.txt` and naive bot blocks — and returns the\n**full page** as clean, ad-stripped Markdown, not a summary. Plus web search that\nreturns real links. Zero dependencies: pure Python standard library.\n\n## Built for AI Agents\n\nAI agents constantly hit pages their built-in tools can't read. NetLens fixes the\nthree usual reasons a fetch comes back empty or useless:\n\n| Native web tools | NetLens |\n|---|---|\n| Honor `robots.txt`, so crawler-disallowed pages return nothing | Reads like the browser you'd open yourself — doesn't consult `robots.txt` |\n| Blocked by header/User-Agent bot filters (`403`/`202` to non-browser clients) | Sends real browser headers via the system `curl`; commonly turns `403 → 200` |\n| Return a **summary** of the page | Returns the **full** page content as Markdown |\n| Leave ads, cookie banners, nav, and related-links chrome in the output | Strips boilerplate locally so only the content reaches your context |\n\nIt does **not** try to defeat JavaScript/Cloudflare challenge pages or CAPTCHAs —\nthat's out of scope by design. When a page is a hard block, the HTTP status is\nsurfaced honestly rather than faked.\n\n## Installation\n\n**npm (via npx):**\n\n```json\n{\n  \"mcpServers\": {\n    \"netlens\": {\n      \"command\": \"npx\",\n      \"args\": [\"-y\", \"netlens-mcp\"]\n    }\n  }\n}\n```\n\n**PyPI (via uvx):**\n\n```json\n{\n  \"mcpServers\": {\n    \"netlens\": {\n      \"command\": \"uvx\",\n      \"args\": [\"netlens-mcp\"]\n    }\n  }\n}\n```\n\nAdd either to your MCP client config (e.g. `.mcp.json` for Claude Code), then\nrestart the session so the tools load.\n\n## Tools\n\n### `web_search`\n\nSearch the web and return real result links (title, URL, snippet), parsed locally —\n**links, not summaries**. Follow up with `web_fetch` to read a result.\n\n| Argument | Type | Description |\n|---|---|---|\n| `query` | string (required) | The search query |\n| `limit` | integer | Optional cap; default returns the full first page (~10) |\n| `engine` | string | `auto` (default), `duckduckgo`, `bing`, `mojeek`, `searxng` |\n| `page` | integer | Result page, 1-based. SearXNG only |\n| `time_range` | string | `day`, `week`, `month`, `year`. SearXNG only |\n| `categories` | string | e.g. `it`, `science`, `news`. SearXNG only |\n\nThe result reports which engine answered and what happened to any that were\nskipped, so falling through to a different backend is visible rather than silent:\n\n```json\n{\n  \"query\": \"…\",\n  \"engine\": \"mojeek\",\n  \"results\": [ … ],\n  \"engines_skipped\": [ { \"engine\": \"duckduckgo\", \"outcome\": \"HTTP 202\" } ]\n}\n```\n\nOutcomes are observations, not conclusions — `HTTP 202` is what the server sent;\nwhether that is throttling, changed markup or genuinely no matches cannot be\ndetermined from the response. With SearXNG configured, direct `answers`,\n`infoboxes` and `suggestions` appear alongside the results.\n\nA search fetches a **single result page** (~10 results), returned in full by default so\nnothing at position 9/10 is dropped. There's no deep pagination — if the answer isn't in\nthe first page, refine the query.\n\n### `web_fetch`\n\nFetch any page and return its full content as clean Markdown.\n\n| Argument | Type | Description |\n|---|---|---|\n| `url` | string (required) | URL to fetch (scheme optional; `https` assumed) |\n| `mode` | string | `article` (main content only, default), `full` (whole body), `raw` (unconverted HTML), `outline` (heading structure) |\n| `section` | string | Return only this heading's content, plus anything nested under it |\n| `links` | string | `inline` (default) keeps link targets; `none` keeps link text but drops URLs |\n| `max_chars` | integer | Optional cap on returned characters (truncates with a note) |\n\n### Reading part of a long page\n\nA large article can be tens of thousands of characters when you want one part of\nit. `mode=\"outline\"` returns its shape, and `section` returns just that piece —\non a large encyclopedia article that is 153,000 characters full, 1,000 as an\noutline, and 7,400 for the section actually wanted.\n\n```\nweb_fetch(url=…, mode=\"outline\")      → headings with each section's size\nweb_fetch(url=…, section=\"Gameplay\")  → that section and its subsections\n```\n\nOn link-heavy pages the URLs themselves are a large share of the output — around 40%\nof a big encyclopedia article — so `links=\"none\"` roughly halves it when you only need\nthe prose. Links pointing back into the same page are always rendered as plain text.\n\nIf a page turns out to be a client-rendered shell, `web_fetch` says so rather than\nreturning an empty result as a success:\n\n> _(note: Only 10 characters of readable text were found in 6,856 characters of HTML.\n> This page appears to be rendered client-side by JavaScript…)_\n\n**Workflow:** `web_search` to find pages, then `web_fetch` to read them.\n\n## Search engines\n\nSearch is a pluggable, selectable registry. In `auto` mode NetLens tries engines in\norder and returns the first with results, so a rate-limit/challenge page on one\nfalls through to the next.\n\n| Engine | Notes |\n|---|---|\n| `duckduckgo` | Default; `html.duckduckgo.com` endpoint |\n| `bing` | Automatic fallback |\n| `mojeek` | Independent index; automatic fallback |\n| `searxng` | Self-hosted/public SearXNG JSON API — set `NETLENS_SEARXNG_URL` |\n\nPick per call with the `engine` argument, or set a default with\n`NETLENS_SEARCH_ENGINE`.\n\n## Configuration\n\n| Environment Variable | Default | Description |\n|---|---|---|\n| `NETLENS_SEARCH_ENGINE` | `auto` | Default search backend |\n| `NETLENS_SEARXNG_URL` | — | SearXNG base URL, e.g. `http://192.168.1.10:8888` |\n| `NETLENS_SEARXNG_TIMEOUT` | `8` | Seconds before an unreachable SearXNG is skipped |\n| `NETLENS_USER_AGENT` | Chrome UA | Override the request User-Agent |\n| `NETLENS_MAX_BYTES` | `10485760` | Cap on a single response; larger ones are truncated |\n| `NETLENS_CACHE_TTL` | `300` | Seconds to reuse a fetched page; `0` disables caching |\n| `NETLENS_HOST_DELAY` | `0.5` | Minimum seconds between requests to the same host |\n| `NETLENS_REQUEST_TIMEOUT` | `120` | Ceiling on a single tool call |\n\n### Self-hosted SearXNG\n\nPoint `NETLENS_SEARXNG_URL` at an instance with the JSON API enabled (`search.formats`\nmust include `json` in its `settings.yml`). It is then tried **first** in `auto` mode,\nwhich removes the HTML scraping — and the rate limiting that comes with it — from the\ncommon path.\n\nBecause it is tried first, it must fail fast when the box is off: the connect timeout\nis bounded separately so an unreachable instance is skipped in a few seconds rather\nthan stalling every search, and the skip is reported in the result.\n\n## How it works\n\n- **Direct fetch.** Requests go straight to the target site via the system `curl`\n  (better TLS/HTTP-2/compression, so it looks like a real browser), falling back to\n  `urllib`. No third-party proxy or reader is involved.\n- **Local conversion.** HTML → Markdown happens in-process with a hand-rolled\n  `html.parser` converter — headings, lists, links (relative URLs resolved), code\n  blocks, and GFM tables with colspan/rowspan.\n- **Content selection, not deletion.** NetLens picks the page's main content region\n  — the HTML5 landmark (`<main>` / `<article>` / `[role=main]`) when one exists,\n  otherwise the subtree holding the most prose relative to its link density — and\n  converts only that. Because it *selects* a winner rather than deleting anything\n  that matches a name pattern, extraction cannot silently return an empty page.\n  Nothing inspects CSS class or id names to decide what is content.\n- **Pruning by measurement.** Within that region, blocks that are overwhelmingly\n  link anchors (navboxes, tag clouds, \"more from this site\" grids) are dropped\n  based on their link density. Non-rendering elements (`<script>`, `<style>`, …),\n  explicitly hidden elements, and third-party ad-network slots (identified by\n  vendor names like `adsbygoogle`, which cannot collide with real prose) are\n  removed outright.\n- **Response charset** is honored (from `Content-Type` or `<meta>`), so non-UTF-8\n  pages don't come back garbled.\n\n## Usage from the CLI\n\nThe server is also a plain script — handy for testing before a client loads it:\n\n```sh\npython -m netlens_mcp.server search \"http caching best practices\"\npython -m netlens_mcp.server fetch  https://example.com/article\npython -m netlens_mcp.server full   https://example.com   # whole body\npython -m netlens_mcp.server raw    https://example.com   # unconverted HTML\n```\n\n`python -m netlens_mcp` runs the stdio MCP server; `python -m netlens_mcp.server\n<cmd>` runs the CLI.\n\n## Development\n\n```sh\npip install -e \".[dev]\"\npython -m pytest        # run the test suite\nruff check .            # lint\n```\n\n## Requirements\n\n- Python 3.10+ (and the system `curl`, which ships with modern Windows/macOS/Linux;\n  falls back to `urllib` if absent)\n\n## License\n\nApache License 2.0 — see [LICENSE](LICENSE) and [NOTICE](NOTICE).\n\n<!-- mcp-name: io.github.pzalutski-pixel/netlens -->\n\n",
  "bytes": 9571,
  "sha": "9b4186de28bc93866abbf65371c7572df128ffd0dad31a489b0f1fdc875ce53a",
  "repo_slug": "pzalutski-pixel/netlens-mcp",
  "fonte": "repo",
  "truncated": false,
  "api": "https://agentalog.com/api/listings/mcp_io_github_pzalutski_pixel_netlens_55fc7348/readme"
}