{
  "markdown": "# webfetch\n\n<!-- mcp-name: io.github.firish/webfetch -->\n\nWeb search for LLM agents that you run yourself - up to 8x fewer input\ntokens and 3x lower cost than hosted web_search, at the same accuracy.\n\nHosted web-search tools charge $10 per thousand searches and then bill you\nagain for every token of retrieved content they push into your context\nwindow. webfetch replaces them with a local pipeline - multi-engine\nsearch, page fetching and extraction, semantic reranking, sentence-level\ncompression - exposed as a `web_search` tool your model calls like any\nother. And unlike every hosted tool and search API we surveyed, repeated\nand paraphrased queries are served from a semantic cache for free.\n\n(Install with `pip install webfetch-llm`; the import name is `webfetch`.)\n\n**Jump to:**\n[The headline](#same-accuracy-a-third-of-the-cost-an-eighth-of-the-tokens) ·\n[What you get](#what-you-get) ·\n[Getting started](#getting-started) ·\n[Check your setup](#check-your-setup) ·\n[Full benchmark results](#full-benchmark-results) ·\n[Claude Code](#using-it-in-claude-code--claude-desktop) ·\n[Agent loop](#use-it-in-your-own-agent-loop) ·\n[Savings report](#the-savings-report) ·\n[How it works](#how-it-works) ·\n[Caveats](#caveats-honestly)\n\n## Same accuracy. A third of the cost. An eighth of the tokens.\n\nOne agent loop, one model, one judge, 50 SimpleQA questions. The only\nthing that changes between rows is the search tool:\n\n| search tool | accuracy | input tok/query | cost/query |\n|---|---|---|---|\n| Anthropic hosted web_search | 96% | 17,408 | $0.108 |\n| **webfetch** (4-engine fusion) | 92% | **3,467** | **$0.035** |\n| **webfetch** (DDG only, $0 in fees) | 84% | 3,623 | $0.026 |\n\nSwap Opus for gpt-5.6-sol and the same webfetch tool hits **96% - hosted\nparity - at $0.040/query and 2,156 tokens**: an eighth of what the hosted\ntool pushes into your context. [Full results](#full-benchmark-results)\ncover every arm we ran.\n\nThese numbers are the WORST case for webfetch - measured on an empty\ncache. In real use the gap widens on its own: repeats and rewords serve\nfrom cache for free, and the token advantage is paid again on every\nlater turn that keeps search results in context. It adds up to receipts\nlike this one, from an ordinary Claude Code session:\n\n![savings report rendered in Claude Code](docs/saving_tool_1.png)\n\nEvery claim in this README is generated by an eval harness that ships in\nthis repo - the question sets, per-question records, judging protocol,\nand the negative results are all in [evals/](evals/), and every table can\nbe regenerated with one command. Don't take our word for the grading:\n[evals/results/README.md](evals/results/README.md) maps every table row\nto the raw result file that produced it, down to per-question judge\nverdicts.\n\n## What you get\n\n**A search pipeline you own (4-engine RRF fusion, local extraction,\nsentence-level compression).** Results come from reciprocal-rank fusion\nacross DuckDuckGo, Brave, Serper, and Tavily - whichever of them you have\nkeys for. DDG needs no key, so the tool works at literally zero cost out\nof the box; every key you add joins the fusion automatically. Pages are\nfetched and extracted locally (trafilatura, readability, newspaper4k,\nPlaywright rendering for JS pages and 403 walls), chunked, ranked by a\nhybrid BM25 + bi-encoder cascade with a cross-encoder on top, then\ncompressed to the sentences that answer the query - measured 50% fewer\ntokens at zero recall loss.\n\n**Caching nobody else has (exact + semantic matching, volatility-aware\nTTLs).** Two layers in one sqlite file: page text by\nURL, ranked results by query. Identical queries hit an exact cache.\nParaphrased queries hit a semantic cache - an embedding shortlist\nverified by an NLI cross-encoder, tuned eval-first for precision (zero\nwrong-target matches across every live run we have done). Cache lifetimes\nadapt to the query: prices and scores expire in 15 minutes, current-ish\ntopics in 7 days, release notes and specs in 90 - classified by the\ncalling model's hint or a local classifier. The model sees provenance on\nevery cached result (`[cache: semantic match to \"...\", 2h old, recent]`)\nand can send `force_fresh` when it disagrees. No hosted tool or search\nAPI we surveyed offers any client-visible caching at all.\n\n**The model can maintain its own cache (save_finding, labeled\nUNVERIFIED, kill switch included).** If a search comes up empty and\nthe model answers from some other source (a hosted search fallback, say),\nit can call `save_finding` to store what it learned - marked\nmodel-contributed, served with an explicit UNVERIFIED warning and a\nforce_fresh escape hatch, and aged out on the normal TTL rules. A kill\nswitch (`SAVE_FINDING_ENABLED = False`) exists for deployments that never\nwant unverified content cached.\n\n**Levers where models actually need them (full_results, fetch_url,\nfreshness, force_fresh).** `full_results` returns\nuncompressed excerpts for list and ranking queries (compression trims\nparallel list items - we measured it). `fetch_url` pulls one cited page\nin full, instantly if the pipeline has ever fetched it. `freshness` hints\ncontrol cache lifetime; `force_fresh` bypasses it.\n\n**Receipts (session and lifetime, exact counters).** Usage counters\npersist in the cache file;\n`webfetch-savings` (or the `savings_report` tool) shows what you did not\npay hosted-search pricing for, split by this session and lifetime.\n[Sample below.](#the-savings-report)\n\n## Getting started\n\nThe one-liner (uv installs the package on first launch, `@latest` picks\nup new releases automatically):\n\n```\nclaude mcp add webfetch -- uvx --from webfetch-llm@latest webfetch-mcp\n```\n\nOr manage the install yourself, two commands:\n\n```\npip install \"webfetch-llm[all]\"\nclaude mcp add webfetch -- webfetch-mcp\n```\n\nThe slim install (`webfetch-llm` without extras) starts in seconds and\nruns degraded: BM25 ranking, exact-only cache, lexical compression.\n`[all]` pulls the semantic stack (torch - a few minutes once) plus\nPlaywright, PDF, and table extraction; it is the configuration every\nbenchmark number in this README was measured on. Needs pip >= 24 in fresh\nvenvs (`python -m pip install -U pip` - older pips crash on a duplicated\nextra in our dependency tree). If you install into a venv, register the\nabsolute path to `webfetch-mcp` or keep the venv active.\n\nKeys are optional (DDG works with none) but improve recall - all the\nengines have free tiers:\n\n```\ncp .env.example .env    # fill in what you have\n```\n\nThe console scripts and the MCP server pick up a `.env` from the\ndirectory they run in; exported env vars and `--env` flags on\n`claude mcp add` always take precedence. `import webfetch` as a library\nnever reads `.env` - your process env is yours.\n\n## Check your setup\n\n`webfetch-status` (or asking the model to call the `status` tool) answers\n\"is my key being picked up, and what am I actually running?\":\n\n```\nwebfetch 0.1.3\n\nsearch engines:\n  ddg      ready (no key needed)\n  brave    ready (BRAVE_API_KEY is set)\n  serper   off   (SERPER_API_KEY not set)\n  tavily   ready (TAVILY_API_KEY is set)\n\nactive provider: multi(ddg+brave+tavily) (RRF fusion)\n\noptional features:\n  semantic ranking/cache/compression: on\n  JS-page rendering (playwright):     on\n  PDF extraction (pdfplumber):        OFF (pip install 'webfetch-llm[pdf]')\n  HTML tables (pandas+tabulate):      on\n  result compression:                 on (crossencoder)\n\ncache: /Users/you/.webfetch/cache.db\n  5.4 MB, 70 lifetime searches (webfetch-savings for the receipt)\n```\n\nKey names only - values are never printed. Configuration is environment\nvariables, read at server start:\n\n| variable | effect |\n|---|---|\n| `BRAVE_API_KEY`, `SERPER_API_KEY`, `TAVILY_API_KEY` | each key adds an engine to the fusion |\n| `WEBFETCH_PROVIDER` | `multi` (default: fuse everything keyed), `fallback` (DDG serves, keyed engines catch its blocks), or a single engine name |\n| `WEBFETCH_CACHE_DB` | relocate the cache file (default `~/.webfetch/cache.db`) |\n\nWith zero keys the default resolves to plain DDG; the status output says\nso honestly rather than calling it fusion. Library users can skip env\nentirely: `Pipeline(search=get_search_adapter(\"fallback\"), cache=...)`.\n\n## Full benchmark results\n\nEvery arm, same 50 SimpleQA questions, same judge, one same-day run\n(2026-07-14). Agent-loop arms share an identical loop; only the search\ntool differs. Costs include model tokens plus each provider's published\nper-search fees; ours include estimated engine fees.\n\n| search tool | model | accuracy | input tok/q | cost/q |\n|---|---|---|---|---|\n| OpenAI hosted web_search | gpt-5.6-sol | 100% | 10,027 | $0.066 |\n| webfetch (4-engine fusion) | gpt-5.6-sol | 96% | 2,156 | $0.040 |\n| Anthropic hosted web_search | Opus 4.7 | 96% | 17,408 | $0.108 |\n| webfetch (4-engine fusion) | Opus 4.7 | 92% | 3,467 | $0.035 |\n| Exa (search + contents) | Opus 4.7 | 90% | 5,496 | $0.053 |\n| Tavily | Opus 4.7 | 88% | 6,387 | $0.047 |\n| webfetch (DDG only) | Opus 4.7 | 84% | 3,623 | $0.026 |\n| webfetch (4-engine fusion) | Haiku 4.5 | 76%* | 3,021 | $0.031 |\n\n\\* Haiku's failures were mostly re-searching past the turn cap, and its\nlast few questions ran on a degraded engine set after we exhausted a free\ntier mid-benchmark. Treat it as a floor.\n\nNotice the token column: webfetch results cost half the input tokens of\nthe snippet APIs and a fifth to an eighth of the hosted tools - which is\nwhy our cost stays lowest even where per-search fees are similar.\n\nOn a second dataset of 27 questions about events from the two weeks\nbefore the run (hand-written, never published, so no vendor could have\ntuned on them), webfetch scored 100% with fusion and 100% with DDG alone;\nthe hosted tools also scored 100%. Fresh events are not the hard part -\nthe [date-injection trap](#use-it-in-your-own-agent-loop) is.\n\nReproduce: `python evals/run_e2e_eval.py --arms ours-multi,hosted` - the\nharness, datasets, and per-question records are in [evals/](evals/). The\ntable above comes from\n[e2e_eval_20260714_005531.json](evals/results/e2e_eval_20260714_005531.json)\n(plus the Haiku run and the fresh-events set - the full claim-to-file map\nis in [evals/results/README.md](evals/results/README.md)). The questions\nare a seeded sample (seed 42) of OpenAI's public SimpleQA set, so\n`build_datasets.py` regenerates the identical 50 on your machine. One\nhonest caveat: the live web moves between runs, and at n=50 the 96-vs-92\naccuracy gap is 48-vs-46 correct - parity within noise. The token and\ncost columns are the mechanically stable result.\n\n## Using it in Claude Code / Claude Desktop\n\nFastest route in Claude Code - install it as a plugin, two slash\ncommands (runs the same uv one-liner under the hood):\n\n```\n/plugin marketplace add firish/webfetch\n/plugin install webfetch@webfetch\n```\n\nOr [register the MCP server yourself](#getting-started). Either way, a\nnew session shows `webfetch` under `/mcp` with five tools: `web_search`, `fetch_url`,\n`save_finding`, `status`, and `savings_report`. Ask anything recent and\nwatch the tool calls; ask the same thing reworded and the result comes\nback with a `[cache: semantic match ...]` header instead of a fresh\nsearch. First call in a session is slow (encoder warm-up plus real page\nfetches, 10-40s); cached calls are instant.\n\nToggle the server off per-session in `/mcp`; remove it with\n`/plugin uninstall webfetch` or `claude mcp remove webfetch`, matching\nhow you installed it. Your cache and its receipt history live in\n`~/.webfetch/` and survive reinstalls. Run one server per machine - the\nsemantic cache assumes a single process owns its file.\n\nUpdate notice: the server makes one request to pypi.org per process to\ncheck for a newer release, surfaced as one line in `savings_report`\noutput. `UPDATE_CHECK_ENABLED = False` in `webfetch/config.py` disables\nit; nothing else is ever sent anywhere.\n\n## Use it in your own agent loop\n\n```python\nimport time\nimport anthropic\nfrom webfetch import WEB_SEARCH_TOOL, handle_web_search\n\nclient = anthropic.Anthropic()\nmessages = [{\"role\": \"user\", \"content\": \"What did the FOMC decide this week?\"}]\nsystem = (f\"Today's date is {time.strftime('%Y-%m-%d')}. \"\n          \"Use web_search for recent facts.\")\n\nwhile True:\n    response = client.messages.create(\n        model=\"claude-opus-4-7\", max_tokens=2000, system=system,\n        tools=[WEB_SEARCH_TOOL], messages=messages,\n    )\n    if response.stop_reason != \"tool_use\":\n        break\n    messages.append({\"role\": \"assistant\", \"content\": response.content})\n    results = [{\"type\": \"tool_result\", \"tool_use_id\": b.id,\n                \"content\": handle_web_search(b.input)}\n               for b in response.content if b.type == \"tool_use\"]\n    messages.append({\"role\": \"user\", \"content\": results})\n```\n\nA complete version with prompt caching and adaptive thinking is in\n[examples/agent_loop.py](examples/agent_loop.py). The tool schema is\nprovider-agnostic in spirit - OpenAI function calling needs only a\nmechanical reshape of the dict.\n\n**Put today's date in your system prompt.** This is not optional. Models\nrefuse to search for events they believe have not happened yet: on our\nfresh-events dataset, loops without the date declined to even call the\ntool on up to 10 of 27 questions (\"Wimbledon 2026 hasn't taken place\nyet\"). One line fixes it. Hosted search tools do this server-side, which\nis part of why nobody notices until they run their own tool.\n\n`handle_web_search` never raises. Engine failures, empty results, and\nmalformed input come back as readable strings the model can react to,\nbecause an exception mid-conversation kills the whole agent loop.\n\n## The savings report\n\nCounters accumulate in the cache file as you search. Ask the model to run\nthe report, or run `webfetch-savings` yourself. The MCP tool splits this\nsession from lifetime, so you can see what one conversation saved:\n\n![session vs lifetime savings](docs/saving_tool_2.png)\n\nThat one-liner unpacked - a 30-search session on this machine:\n\n| | this session | lifetime |\n|---|---|---|\n| searches served | 30 | 76 |\n| served from cache | 17 (57%) - 10 exact, 7 semantic | - |\n| search fees avoided | ~$0.30 | |\n| content-token cost avoided | ~$2.56 | |\n| **estimated total avoided** | **~$2.86** | **~$7.21** |\n\nCounters are exact; the dollar lines are estimates with the assumptions\n(hosted per-search fee, hosted tokens per call - both measured - and your\nmodel's token price) exposed as arguments to `webfetch.savings_report()`.\n\n## How it works\n\n```\nquery -> multi-engine search (RRF fusion, circuit breakers)\n      -> semantic query cache? serve with provenance header\n      -> fetch pages concurrently (page cache by URL)\n      -> extract (trafilatura -> readability -> newspaper4k -> playwright)\n      -> chunk (400 chars) -> BM25 + bi-encoder fusion -> cross-encoder top 5\n      -> compress to query-relevant sentences -> source-labeled result\n```\n\nEverything heavy is optional: without `[rerank]` you get BM25 ranking,\nexact-match caching, and lexical compression, with a logged warning\ninstead of an ImportError. DDG deserves a special note: it\nfingerprint-blocks automated clients with silent empty responses.\nwebfetch detects that (empty-with-peers in fusion, any empty in the\nfallback chain), benches the engine on a circuit breaker, and routes\naround it; the TLS side is handled by the `ddgs` dependency.\n\nDesign decisions and their measurements are documented as they happened\nin [docs/architecture.md](docs/architecture.md) and\n[docs/ROADMAP.md](docs/ROADMAP.md) - including the negative results.\n\n## Caveats, honestly\n\n- Latency. A fresh search takes 10-40 seconds - real pages get fetched\n  and ranked locally. Hosted search returns in ~10s, snippet APIs in ~6s.\n  Cache hits are instant. If you need sub-second search and do not care\n  about cost, this is not your tool.\n- The cache is single-process. One agent loop, one MCP server, or one\n  notebook at a time; a shared team cache backend is on the roadmap.\n- Engine free tiers are real quotas. We exhausted Brave's monthly tier\n  during benchmarking. The resilience layer degrades gracefully, but your\n  recall degrades with it. Engine fees in the tables are estimates from\n  published prices.\n- Answer quality depends on the model driving the tool - weak models\n  formulate worse queries and re-search instead of reading (see the Haiku\n  row).\n- Semantic-cache recall on cross-form rewords (\"what did X add\" vs\n  \"X new features\") is partial: measured 6/10 on real pairs after tuning,\n  because some paraphrase forms defeat every available verifier. Misses\n  cost a re-search, never a wrong answer.\n\n## Benchmarks and data\n\nThree eval layers, all in [evals/](evals/): an offline matcher eval that\npicks the semantic-cache thresholds, a live retrieval eval (recall,\ntokens, cache diagnostics), and the end-to-end answer eval quoted above\n(SimpleQA protocol: normalized exact match, then an LLM judge). Datasets\nare built deterministically from SimpleQA (MIT), QQP/GLUE, and FreshQA\n(CC-BY-SA), with provenance sidecars next to each file. The fresh-events\nset was hand-written against verified news sources days before the\nbenchmark ran, specifically so no model or vendor pipeline could have\nseen it.\n\n## License\n\nMIT.\n",
  "bytes": 17212,
  "sha": "9e6ce1e7c636c19a4285f20a4e21ad45784eb6511f3f40a6afbf5c4c14770cd6",
  "repo_slug": "firish/webfetch",
  "fonte": "repo",
  "truncated": false,
  "api": "https://agentalog.com/api/listings/mcp_io_github_firish_webfetch_29eb0400/readme"
}