{
  "markdown": "<!-- mcp-name: io.github.VelvetSP/web-retrieval-mcp -->\n\n# web-retrieval-mcp — reliable MCP web search and web fetch for AI agents\n\n> **An open-source [Model Context Protocol (MCP)](https://modelcontextprotocol.io) server for source-grounded web research. Connect Codex CLI and the Codex IDE extension, Claude Code, Claude Desktop, Cursor, ChatGPT desktop, or any other compatible MCP client to six read-only tools for Exa or Tavily search, resilient Exa → Camoufox → Tavily → Firecrawl page retrieval, AI/ML paper discovery, and developer-source search—with explicit provenance, optional local caching, and SSRF guards.**\n\n[![PyPI](https://img.shields.io/pypi/v/web-retrieval-mcp.svg)](https://pypi.org/project/web-retrieval-mcp/)\n[![CI](https://github.com/VelvetSP/web-retrieval-mcp/actions/workflows/ci.yml/badge.svg)](https://github.com/VelvetSP/web-retrieval-mcp/actions/workflows/ci.yml)\n[![Python 3.10+](https://img.shields.io/pypi/pyversions/web-retrieval-mcp.svg)](https://pypi.org/project/web-retrieval-mcp/)\n[![License: MIT](https://img.shields.io/badge/license-MIT-16a34a.svg)](https://github.com/VelvetSP/web-retrieval-mcp/blob/main/LICENSE)\n[![Model Context Protocol](https://img.shields.io/badge/MCP-server-7c3aed.svg)](https://modelcontextprotocol.io)\n\n**[Why use it?](#why-use-web-retrieval-mcp) · [Quick start](#quick-start) · [Tools](#six-read-only-tools) · [Routing](#how-retrieval-works) · [Configuration](#configuration-reference) · [Security](#security-model) · [FAQ](#frequently-asked-questions)**\n\n## Why use web-retrieval-mcp?\n\nWeb research is more reliable when an agent can see where every result came from,\nchoose the right retrieval strategy, and recover when one provider or page-access\nmethod fails. `web-retrieval-mcp` makes those controls part of the tool contract:\n\n- **Keep sources separate.** Default search returns one block per result with its own\n  title, URL, highlights, and text, followed by a `Sources` trailer. Exa deep modes\n  additionally return a clearly labeled synthesized answer, with grounding when Exa\n  provides it, before the source-separated result blocks.\n- **Use more than one retrieval path.** Search can use Exa or Tavily and fall back to\n  Firecrawl. Full-page fetches can escalate from indexed content to a local Camoufox\n  browser, optional Tavily Extract, and Firecrawl.\n- **Know what actually served the answer.** Fetch responses identify the serving tier,\n  cache state, semantic mode, and truncation instead of hiding the route.\n- **Ask for the right shape.** Fetch a readable page body, a concise summary, or a\n  grounded answer to a question. Control freshness, rendering, domains, dates, result\n  count, and per-result text size.\n- **Search specialist corpora.** Discover AI/ML papers, inspect query-relevant\n  full-text passages, expand to related papers, and search developer documentation,\n  READMEs, issues, and merged pull requests.\n- **Own the deployment boundary.** Run locally over stdio or host a loopback HTTP\n  service; select providers and secret storage yourself; optionally cache completed\n  fetches in a private Valkey sidecar.\n- **Fail explicitly.** Unsupported provider choices, missing optional packages, and\n  unavailable credentials fail in the tool result. Tavily mappings, Firecrawl fallback\n  filter gaps, and Exa category migrations are caller-visible; other Exa filter drops\n  are written to server diagnostics.\n\n### How it compares with built-in agent web tools\n\nBuilt-in capabilities vary by agent and can improve over time. The useful comparison\nis therefore not “all defaults are bad”; it is whether you need a portable,\ninspectable retrieval layer with controls your current client does not expose.\n\n| Need | Typical bundled search/fetch surface | **web-retrieval-mcp** |\n|---|---|---|\n| Search output | Client-defined result or synthesis format | Default search returns source-separated blocks plus `Sources`; Exa deep modes add a labeled synthesis |\n| Backend choice | Provider and routing are usually managed by the client | Exa or Tavily search, with a disclosed Firecrawl fallback |\n| Difficult pages | One client-specific access path | Indexed content, guarded local browser rendering, Tavily Extract, and Firecrawl tiers |\n| Retrieval intent | Usually search or page text | Full body, concise summary, or query-grounded page answer |\n| Research discovery | General web index | Dedicated paper and developer-source tools in addition to web search |\n| Freshness and filters | Whatever the client exposes | Date windows, hour-level recency, domains, categories, result limits, and forced freshness |\n| Auditability | Client-specific | Fetch tier/cache/truncation disclosures; Tavily and Firecrawl search-fallback labels; explicit filter notices where supported |\n| Operations | Client-defined; often hosted or opaque | Local stdio or self-hosted loopback HTTP, optional private UDS cache |\n| Security policy | Client-specific | Initial URL validation plus guarded browser requests and redirects |\n\nUse it as an **independent complementary retrieval lane** by default. If you want it\nto replace Claude Code's built-in `WebSearch` and `WebFetch`, the package also includes\nan optional, previewable hook installer.\n\n## Quick start\n\nPython 3.10 or newer is required.\n\n### 1. Install\n\nFor the complete feature set in an isolated environment:\n\n```bash\npipx install --include-deps 'web-retrieval-mcp[all]'\ncamoufox fetch\n```\n\n`--include-deps` exposes Camoufox's own command from the isolated pipx environment;\nthe second command downloads its managed browser. If you do not want local rendering,\ninstall the base package or selected non-render extras and skip the browser download.\n\nFor a lean Exa + Firecrawl installation:\n\n```bash\npipx install web-retrieval-mcp\n```\n\nWith [uv](https://docs.astral.sh/uv/guides/tools/) installed, you can run the base\npackage in a temporary isolated environment without a permanent install:\n\n```bash\nuvx web-retrieval-mcp\n```\n\nOr install into an existing virtual environment:\n\n```bash\npython -m pip install 'web-retrieval-mcp[all]'\npython -m camoufox fetch\n```\n\nThe browser download is required whenever you install the `render` extra and want\nCamoufox to serve `render=\"auto\"` or `render=\"always\"` calls.\n\nOptional extras are composable:\n\n| Extra | Adds |\n|---|---|\n| `tavily` | Tavily Search and Tavily Extract |\n| `render` | Local Camoufox/Playwright rendering |\n| `cache` | Valkey client for completed-result caching |\n| `keyring` | Cross-platform native secret-store access |\n| `all` | Every optional runtime capability above |\n\n### 2. Add provider credentials\n\nFor the broadest routing coverage, configure Exa and Firecrawl; add Tavily when you\nwant provider-selectable search or another full-body fetch tier.\n\n```bash\nexport EXA_API_KEY='<your-exa-api-key>'\nexport FIRECRAWL_API_KEY='<your-firecrawl-api-key>'\nexport TAVILY_API_KEY='<your-tavily-api-key>'   # optional\n```\n\nYou do not need all three keys to start the server. Tools resolve credentials lazily\nand report what a selected route is missing. See [Credential storage](#credential-storage)\nfor a key file and OS secret-store alternatives.\n\nAll three providers advertised no-card entry allocations when checked on 2026-09-01;\nplans and unit costs can change, so check their live pricing before estimating a\nworkload:\n\n| Provider | Entry allocation checked 2026-09-01 | Used for |\n|---|---|---|\n| [Exa](https://exa.ai/pricing?tab=api) | $20 signup credit plus $10 monthly API credits | Default web search and first indexed-content fetch tier |\n| [Firecrawl](https://www.firecrawl.dev/pricing) | 1,000 credits per month | Search/fetch fallback and research/developer indexes |\n| [Tavily](https://www.tavily.com/pricing) | 1,000 API credits per month | Optional search provider and full-body extract tier |\n| [Camoufox](https://github.com/daijro/camoufox) | Local open-source browser | JavaScript-rendered page retrieval on your machine |\n\n### 3. Connect an MCP client\n\nCodex CLI and the Codex IDE extension share MCP configuration. Add the server with\nthe CLI, then verify it is registered:\n\n```bash\ncodex mcp add web-retrieval -- web-retrieval-mcp\ncodex mcp list\n```\n\nWith `uvx` and no prior install:\n\n```bash\ncodex mcp add web-retrieval -- uvx web-retrieval-mcp\n```\n\nSee OpenAI's [Codex MCP documentation](https://developers.openai.com/codex/mcp)\nfor the current CLI, IDE, and configuration-file details.\n\nClaude Code:\n\n```bash\nclaude mcp add web-retrieval -- web-retrieval-mcp\n```\n\nWith `uvx` and no prior install:\n\n```bash\nclaude mcp add web-retrieval -- uvx web-retrieval-mcp\n```\n\nClaude Desktop, Cursor, and other clients that accept an `mcpServers` object can use:\n\n```json\n{\n  \"mcpServers\": {\n    \"web-retrieval\": {\n      \"command\": \"web-retrieval-mcp\"\n    }\n  }\n}\n```\n\nThat configuration assumes the executable is on the client's `PATH`. For an `uvx`\nlaunch, use `\"command\": \"uvx\"` and `\"args\": [\"web-retrieval-mcp\"]`.\nIf a desktop client does not inherit your shell environment, use the private key file\ndescribed below or place literal key values in the client's protected environment\nconfiguration. Do not rely on `${VARIABLE}` interpolation unless your client documents\nthat behavior.\n\nChatGPT desktop and clients with different setup surfaces can connect through the MCP\ntransport their current version supports. This server uses stdio by default and can\nalso expose Streamable HTTP; consult the client's documentation for its configuration\nformat and supported transports.\n\n### 4. Give the agent a real task\n\nOnce the MCP server is connected, prompts can stay natural:\n\n```text\nSearch the web for the latest primary documentation about Python package metadata.\nKeep each source separate and preserve its URL.\n\nFetch https://example.com/report, force a fresh retrieval, and answer only:\nWhat methodology did the authors use?\n\nFind AI/ML papers about retrieval reranking, inspect the strongest paper for its\nreported benchmark result, then find newer work that cites it.\n\nSearch merged pull requests and documentation for the origin of this exact error:\n\"transport closed\".\n```\n\n## Six read-only tools\n\nAll tools carry MCP's read-only annotation. MCP clients receive the complete generated\ninput schema and descriptions when they connect.\n\n| Tool | Compact signature | Best for |\n|---|---|---|\n| `web_search` | `web_search(query, num_results=8, mode=\"auto\", …, provider=None)` | General web search through Exa or Tavily, with source-separated results and Firecrawl fallback |\n| `web_fetch` | `web_fetch(url, render=\"auto\", max_chars=None, max_age_hours=None, mode=\"full\", question=None, tavily=None)` | Readable page bodies, concise summaries, or grounded answers through a tiered fetch cascade |\n| `research_papers` | `research_papers(query, k=8)` | Ranked AI/ML paper discovery through Firecrawl's Research Index |\n| `research_paper` | `research_paper(paper_id, query=None)` | Paper metadata and optional query-relevant full-text passages for claim verification |\n| `research_similar` | `research_similar(paper_id, intent, k=8, mode=\"similar\", …)` | Related papers, citers, or references guided by a natural-language intent |\n| `research_github` | `research_github(query, k=8, passages=2, types=None, repos=None)` | Developer documentation, repository READMEs, issues, and merged pull requests |\n\n`web_search` supports relevance or date-oriented filters, domain inclusion/exclusion,\npublication windows, generated per-result summaries, and Exa's fast/deep search modes.\nTavily receives equivalent controls where its API supports them; approximations and\ndropped controls are disclosed in the output.\n\nThe research-paper index is arXiv-oriented and best suited to AI/ML. For scholarly\nwork outside that scope, use `web_search(category=\"publication\")`.\n\n## How retrieval works\n\n### Search routing\n\nExa is the default search provider. Select Tavily for one call with\n`provider=\"tavily\"`, or globally:\n\n```bash\nexport WEB_SEARCH_PROVIDER=tavily\n```\n\nOnly `exa` and `tavily` are accepted. If the selected provider fails, the server tries\nFirecrawl and names both the failed primary provider and serving fallback in the\nresult.\n\n### Fetch routing\n\nFor an ordinary full-body request in automatic mode:\n\n```text\nExa indexed contents\n        ↓ unusable or unavailable\nguarded local Camoufox browser\n        ↓ unusable or unavailable\noptional Tavily Extract\n        ↓ unusable or unavailable\nFirecrawl\n```\n\n- `render=\"auto\"` uses that adaptive cascade.\n- `render=\"never\"` forbids the local browser.\n- `render=\"always\"` starts with the local browser and skips Exa; optional Tavily and\n  Firecrawl remain backstops.\n- `mode=\"concise\"` asks for a compact generated summary.\n- `question=\"…\"` asks for a grounded answer rather than the whole page.\n- `max_age_hours=0` forces fresh provider work; `-1` requests Exa's always-use-cache\n  behavior; positive values set a freshness window.\n- Explicit full-body requests above Exa's body-size ceiling start with a tier capable\n  of satisfying the requested size rather than silently returning a short body.\n\nEnable Tavily Extract per call with `tavily=true`, or globally:\n\n```bash\nexport WEB_FETCH_TAVILY_TIER=1\n```\n\nTavily Extract is a full-body tier. The server skips it when it cannot honor a\nconcise, question-answer, or explicit freshness contract.\n\n## Credential storage\n\nThe server resolves each API key lazily in this order:\n\n1. `EXA_API_KEY`, `FIRECRAWL_API_KEY`, or `TAVILY_API_KEY` in the process environment.\n2. A dotenv-style private key file.\n3. The optional Python `keyring` package.\n4. macOS Keychain or Linux Secret Service command-line clients.\n\nThe default key file is:\n\n- Linux/macOS: `${XDG_CONFIG_HOME:-~/.config}/web-retrieval-mcp/keys.env`\n- Windows: `%APPDATA%\\web-retrieval-mcp\\keys.env`\n\nOverride the file with `WEB_RETRIEVAL_MCP_ENV_FILE`, or its directory with\n`WEB_RETRIEVAL_MCP_CONFIG_DIR`. On POSIX systems the file must not be readable by\ngroup or other users:\n\n```bash\ninstall -d -m 700 ~/.config/web-retrieval-mcp\nprintf '%s\\n' \\\n  'EXA_API_KEY=<your-exa-api-key>' \\\n  'FIRECRAWL_API_KEY=<your-firecrawl-api-key>' \\\n  'TAVILY_API_KEY=<your-tavily-api-key>' \\\n  > ~/.config/web-retrieval-mcp/keys.env\nchmod 600 ~/.config/web-retrieval-mcp/keys.env\n```\n\nFor `keyring`, store each secret under service `web-retrieval-mcp`, using the\nenvironment-variable name as the username:\n\n```bash\nkeyring set web-retrieval-mcp EXA_API_KEY\nkeyring set web-retrieval-mcp FIRECRAWL_API_KEY\nkeyring set web-retrieval-mcp TAVILY_API_KEY\n```\n\nCredentials remain in process memory, are redacted from provider errors and displayed\nURLs, and are never put in provider command arguments.\n\n## Configuration reference\n\n| Setting | Default | Purpose |\n|---|---|---|\n| `EXA_API_KEY` | unset | Exa search and indexed page contents |\n| `FIRECRAWL_API_KEY` | unset | Firecrawl fallback, research papers, and developer search |\n| `TAVILY_API_KEY` | unset | Tavily Search and Extract; also requires the `tavily` extra |\n| `WEB_SEARCH_PROVIDER` | `exa` | Default `web_search` provider: `exa` or `tavily` |\n| `WEB_FETCH_TAVILY_TIER` | `false` | Globally enable Tavily Extract in eligible fetch cascades |\n| `WEB_RETRIEVAL_MCP_ENV_FILE` | platform key-file path | Override the exact dotenv key file |\n| `WEB_RETRIEVAL_MCP_CONFIG_DIR` | platform config directory | Override the directory containing `keys.env` |\n| `WEB_RETRIEVAL_MCP_CACHE` | `auto` | Completed fetch cache: `auto`, `on`, or `off` |\n| `WEB_RETRIEVAL_MCP_VALKEY_SOCKET` | unset | Absolute path to a private Valkey Unix-domain socket |\n| `WEB_RETRIEVAL_MCP_HOST` | `127.0.0.1` | Streamable HTTP bind address |\n| `WEB_RETRIEVAL_MCP_PORT` | `8100` | Streamable HTTP port |\n\nBoolean settings accept `1/0`, `true/false`, `yes/no`, and `on/off`; invalid values\nfail explicitly rather than being guessed.\n\n### Optional completed-result cache\n\nInstall the `cache` extra and point the server at a private, non-persistent Valkey\nUnix-domain socket:\n\n```bash\nexport WEB_RETRIEVAL_MCP_CACHE=on\nexport WEB_RETRIEVAL_MCP_VALKEY_SOCKET=/absolute/private/path/valkey.sock\n```\n\nSuccessful eligible fetches are stored for 24 hours. Credential-bearing URLs,\nuserinfo, forced or positive freshness, and `render=\"always\"` bypass replay. Cache\nerrors fail open to normal provider retrieval. Default `auto` enables caching only\nwhen the Valkey client is installed on a non-Windows host; `on` still attempts the\nconfigured Unix socket and fails open on errors, while `off` disables it. See [the cache guide](https://github.com/VelvetSP/web-retrieval-mcp/blob/main/docs/CACHE.md)\nfor the privacy model, singleflight behavior, and a minimal sidecar configuration.\n\n### Streamable HTTP transport\n\nstdio is the default and recommended transport for a local MCP client:\n\n```bash\nweb-retrieval-mcp\n# equivalent: python -m web_retrieval_mcp\n```\n\nFor a shared local process, stateless Streamable HTTP is available:\n\n```bash\nweb-retrieval-mcp --http --host 127.0.0.1 --port 8100\n```\n\nHTTP transport has no built-in authentication. Keep it on loopback unless you add an\nauthenticated perimeter and suitable network controls.\n\n### Optional Claude Code replacement policy\n\nThe package includes a PreToolUse hook that can deny Claude Code's built-in\n`WebSearch` and `WebFetch`, directing agents to this MCP server instead. Preview the\nexact settings change first:\n\n```bash\nweb-retrieval-mcp-install --print\n```\n\nInstall or remove it explicitly:\n\n```bash\nweb-retrieval-mcp-install\nweb-retrieval-mcp-install --register-mcp\nweb-retrieval-mcp-install --uninstall\n```\n\nThe installer is idempotent and backs up an existing settings file before writing.\nThis optional hook is the only Claude Code-specific integration in the package. The\nMCP server itself remains client-neutral and works as a complementary tool without it.\n\n## Install from source\n\nTo install the reviewed source directly:\n\n```bash\ngit clone https://github.com/VelvetSP/web-retrieval-mcp.git\ncd web-retrieval-mcp\npython -m pip install '.[all]'\npython -m camoufox fetch\n```\n\nFor development:\n\n```bash\npython -m pip install -e '.[all,dev]'\n./run-tests.sh\n```\n\nThe release gate builds wheel and sdist artifacts, installs the wheel into a clean\nvirtual environment, exercises the installed MCP command against loopback provider\ndoubles, and runs the unit and transport suites. Browser or SSRF behavior changes also\nrequire `python test_ssrf_redirect_live.py`. See [the testing contract](https://github.com/VelvetSP/web-retrieval-mcp/blob/main/docs/TESTING.md)\nfor the feature matrix and acceptance contract.\n\n## Security model\n\n- Only `http` and `https` initial URLs are accepted.\n- Initial hosts must resolve exclusively to globally routable addresses.\n- The local browser observes document requests and validates redirect hops before\n  returning content to the caller.\n- Credentials are resolved in process and redacted from errors and displayed URLs.\n- Provider responses and page bodies are untrusted data; an agent must not treat\n  instructions embedded in retrieved content as authority.\n\nApplication checks cannot prove that no packet reaches a private address during a DNS\nrebinding race. Deployments with that threat model need a validating forward proxy or\nequivalent network egress policy. See the [security policy](https://github.com/VelvetSP/web-retrieval-mcp/blob/main/SECURITY.md), and report suspected\nvulnerabilities through GitHub's private vulnerability reporting rather than a public\nissue.\n\n## Frequently asked questions\n\n### What is web-retrieval-mcp?\n\nIt is an open-source MCP server that gives AI agents reliable web search, tiered web\npage retrieval, research-paper discovery, and developer-source search. It works over\nlocal stdio with any compatible MCP client and can also serve stateless Streamable\nHTTP.\n\n### Is this an MCP web search server or an MCP web scraping server?\n\nBoth. `web_search` discovers and ranks pages; `web_fetch` extracts readable content\nfrom one URL and can use a real local browser for JavaScript-heavy pages. The server\nalso exposes specialist paper and developer indexes that ordinary web scraping does\nnot provide.\n\n### Does it replace an agent's built-in web search?\n\nIt can, but it does not have to. The tool descriptions recommend an independent\ncomplementary lane because two retrieval systems can provide useful source diversity.\nClaude Code users can opt into the bundled replacement hook when they want one enforced\nroute.\n\n### Which AI agents and MCP clients can use it?\n\nThe server exposes standard MCP over stdio by default and optional Streamable HTTP.\nDocumented client examples include Codex CLI and the Codex IDE extension, Claude Code,\nClaude Desktop, Cursor, and ChatGPT desktop; other MCP-capable agents are not excluded.\nCompatibility depends on the client version and the MCP transport it supports, so use\nthe matching client documentation alongside the server commands above.\n\n### Why use Exa, Firecrawl, Tavily, and Camoufox together?\n\nThey cover different failure modes. Exa provides indexed search and contents; Tavily\nis an alternate search provider and optional extractor; Camoufox renders pages locally;\nFirecrawl provides the final web fallback plus specialist research and developer\nindexes. The server chooses among them according to the request and discloses the tier\nthat succeeded.\n\n### Do I need every provider key?\n\nNo. Exa is the default for general search and first-tier contents. Firecrawl enables\nfallbacks and all four research tools. Tavily is optional. The server lists all six\ntools even when an optional route is not configured and returns an actionable error if\na call selects an unavailable capability.\n\n### Is web-retrieval-mcp free?\n\nThe project is MIT-licensed. Provider usage is billed under your own accounts; each\nprovider currently offers an entry allocation, but quotas and prices can change.\nCamoufox runs locally without a metered retrieval API.\n\n### Which operating systems are supported?\n\nThe base Python server is platform-independent. Environment and key-file credentials\nwork across Linux, macOS, and Windows. Valkey UDS caching is non-Windows, and local\nbrowser availability follows the supported Camoufox/Playwright platforms.\n\n### Does it support RAG and autonomous research agents?\n\nYes. The source-separated result contract, explicit URLs, controllable text budgets,\ngrounded page questions, specialist indexes, and deterministic provenance make the\ntools suitable as a retrieval layer for RAG pipelines and research agents. The server\nreturns evidence; the calling application remains responsible for evaluation,\ncitation, and prompt-injection handling.\n\n## Project links\n\n- [Python package on PyPI](https://pypi.org/project/web-retrieval-mcp/)\n- [MCP Registry manifest](https://github.com/VelvetSP/web-retrieval-mcp/blob/main/server.json)\n- [Documentation index](https://github.com/VelvetSP/web-retrieval-mcp/blob/main/docs/README.md)\n- [Testing and acceptance contract](https://github.com/VelvetSP/web-retrieval-mcp/blob/main/docs/TESTING.md)\n- [Security policy](https://github.com/VelvetSP/web-retrieval-mcp/blob/main/SECURITY.md)\n- [Publishing runbook](https://github.com/VelvetSP/web-retrieval-mcp/blob/main/PUBLISHING.md)\n- [Agent-readable project index](https://github.com/VelvetSP/web-retrieval-mcp/blob/main/llms.txt)\n- [Issues and feature requests](https://github.com/VelvetSP/web-retrieval-mcp/issues)\n- [Release notes](https://github.com/VelvetSP/web-retrieval-mcp/releases)\n\nContributions are welcome. Keep `stdout` reserved for JSON-RPC, send diagnostics to\n`stderr`, add public-boundary acceptance coverage for changed behavior, and run\n`./run-tests.sh` before opening a pull request.\n\n## License\n\n[MIT](https://github.com/VelvetSP/web-retrieval-mcp/blob/main/LICENSE) © VelvetSP\n",
  "bytes": 23737,
  "sha": "29799ea8c376cdb30d84a8cef08524508a1654a4f6d2b1473d6a92fb5c383267",
  "repo_slug": "velvetsp/web-retrieval-mcp",
  "fonte": "repo",
  "truncated": false,
  "api": "https://agentalog.com/api/listings/mcp_io_github_velvetsp_web_retrieval_mcp_002b974a/readme"
}