{
  "markdown": "# mcp-webgate\n\n[![Python Version](https://img.shields.io/badge/python-3.11+-blue.svg)](https://www.python.org/downloads/)\n[![License](https://img.shields.io/badge/license-MIT-green.svg)](LICENSE)\n[![MCP Protocol](https://img.shields.io/badge/MCP-Protocol-blueviolet)](https://spec.modelcontextprotocol.io/)\n[![Latest Release](https://img.shields.io/badge/release-v0.1.33-purple.svg)](https://github.com/x-hannibal/mcp-webgate/releases/tag/v0.1.33)\n[![Beta](https://img.shields.io/badge/status-beta-orange.svg)](https://github.com/x-hannibal/mcp-webgate/issues)\n\nWeb search that doesn't wreck your AI's memory.\n\nmcp-webgate is an MCP server that gives your AI clean, bounded web content — across all major AI clients:\n- **IDEs**: Claude Desktop, Claude Code, Zed, Cursor, Windsurf, VSCode\n- **CLI Agents**: Gemini CLI, Claude CLI, custom agents\n\n## 🌱 A Gentle Introduction\n\n**What is mcp-webgate?**\nWhen your AI uses a standard \"fetch URL\" tool, it gets the raw HTML of the page — ads, menus, scripts, cookie banners and all. A single news article can dump **200,000 tokens** of garbage into the AI's memory, wiping out your entire conversation.\n\n**mcp-webgate** is a protective filter that sits between your AI and the web:\n\n1. **Strips the junk** — menus, scripts, ads, footers are removed with surgical HTML parsing; only readable text passes through\n2. **Hard-caps every response** — no page can ever blow up your context window, no matter how big the original was\n3. **Optionally summarizes** — route results through a secondary local LLM that produces a compact Markdown report with citations; your primary AI gets a polished briefing instead of a wall of text\n\nThe result: clean, bounded, useful web content — always.\n\n### 🔬 Real example: what happens under the hood\n\nSearching for *\"mcp model context protocol\"* with LLM features on:\n\n```\nQuery → LLM expands to 5 search variants → 20 pages found, 13 fetched in parallel\n\nRaw HTML downloaded     5.16 MB   (~1,290,000 tokens)\nAfter cleaning          52.1 KB   (   ~13,000 tokens)  — 99% noise stripped\nAfter LLM summary        5.8 KB   (    ~1,450 tokens)  — structured report with citations\n```\n\n**13 sources distilled into ~1,450 tokens.** A single naive fetch of just *one* of those pages (e.g. a security blog at 563 KB) would dump **~140,000 tokens** of raw HTML into your AI's context. webgate processes all 13 and delivers a clean briefing that fits in a footnote.\n\nThis is an intensive case (5 queries × 5 results). A typical search with 3–5 results still saves 95%+ of context compared to raw fetching — and your AI gets structured, ranked content instead of a wall of HTML soup.\n\n## 🚀 Quick Start\n\n### 1. Make sure you have `uvx`\n\n```bash\npip install uv\n```\n\n`uvx` runs Python tools without installing them permanently. You only need to do this once.\n\n### 2. Set up a search backend\n\nThe easiest option is **SearXNG** — free, no account, runs locally:\n\n```bash\ndocker run -d -p 8080:8080 --name searxng searxng/searxng\n```\n\nNo Docker? Use a cloud backend instead (Brave, Tavily, Exa, SerpAPI) — see [Backends](#backends).\n\n### 3. Add webgate to your AI client\n\nSee the [Integrations](#integrations) table for your specific client. As a quick example, for **Claude Desktop**:\n\nOpen the config file:\n- **macOS**: `~/Library/Application Support/Claude/claude_desktop_config.json`\n- **Windows**: `%APPDATA%\\Claude\\claude_desktop_config.json`\n\nAdd this:\n\n```json\n{\n  \"mcpServers\": {\n    \"webgate\": {\n      \"command\": \"uvx\",\n      \"args\": [\"mcp-webgate\"],\n      \"env\": {\n        \"WEBGATE_DEFAULT_BACKEND\": \"searxng\",\n        \"WEBGATE_SEARXNG_URL\": \"http://localhost:8080\"\n      }\n    }\n  }\n}\n```\n\nRestart the client after editing.\n\n### 4. Ask your AI to search!\n\n```\nSearch the web for: latest news on AI regulation\n```\n\nThe AI will use `webgate_query` automatically. You're done.\n\n## 🔍 How it works\n\n```\nYour question\n    ↓\nSearch backend  (SearXNG / Brave / Tavily / Exa / SerpAPI)\n    ↓  [deduplicate URLs, block binary files, filter domains]\nFetch pages in parallel  (streaming — hard size cap per page)\n    ↓  [optional: retry failed pages from reserve pool]\nStrip HTML junk  (menus, ads, scripts, footers — lxml)\n    ↓\nClean up text  (invisible chars, unicode junk, BiDi tricks)\n    ↓\nBM25 reranking  (best-matching results first — always active)\n    ↓  [optional: LLM reranking]\nCap total output to budget\n    ↓  [optional: LLM summarization → compact Markdown report]\nClean result lands in your AI's context\n```\n\n## 🛠️ Tools\n\nwebgate gives your AI three tools:\n\n### `webgate_fetch` — read a single page\n\nUse this when you already know the URL you want. The AI passes the URL and gets back the cleaned text — up to `max_query_budget` characters (default 32,000).\n\n```json\n{ \"url\": \"https://example.com/article\", \"max_chars\": 32000 }\n```\n\n```json\n{\n  \"url\": \"https://example.com/article\",\n  \"title\": \"Article Title\",\n  \"text\": \"cleaned text...\",\n  \"truncated\": true,\n  \"char_count\": 12450\n}\n```\n\n### `webgate_query` — search + fetch + clean\n\nRuns a full search cycle. Pass one query (or several) and get back cleaned, ranked results.\n\n```json\n{ \"queries\": \"how to set up a VPN on Linux\", \"num_results_per_query\": 5 }\n```\n\nMultiple queries run in parallel and are merged:\n\n```json\n{\n  \"queries\": [\"VPN Linux setup\", \"best VPN Linux 2024\"],\n  \"num_results_per_query\": 5\n}\n```\n\n**Output without LLM** — returns cleaned page content for each result:\n\n```json\n{\n  \"sources\": [\n    { \"id\": 1, \"title\": \"...\", \"url\": \"...\", \"content\": \"cleaned text...\", \"truncated\": false }\n  ],\n  \"snippet_pool\": [ { \"id\": 6, \"title\": \"...\", \"url\": \"...\", \"snippet\": \"...\" } ],\n  \"stats\": { \"fetched\": 5, \"total_chars\": 18200, \"per_page_limit\": 6400 }\n}\n```\n\n**Output with LLM summarization** — returns a compact Markdown report:\n\n```json\n{\n  \"summary\": \"## How to set up a VPN on Linux\\n\\nTo install...[1][2]\",\n  \"citations\": [{ \"id\": 1, \"title\": \"...\", \"url\": \"...\" }],\n  \"stats\": { \"fetched\": 5, \"total_chars\": 58000 }\n}\n```\n\n**Output when LLM fails** — error reason shown, full sources returned as fallback:\n\n```json\n{\n  \"llm_summary_error\": \"ReadTimeout: LLM did not respond in time\",\n  \"sources\": [ \"...\" ],\n  \"stats\": { \"...\" : \"...\" }\n}\n```\n\n`snippet_pool` contains extra results from the search that were not fetched (search-engine snippet only). The AI can use these to decide if more fetches are worthwhile.\n\n### `webgate_onboarding` — how-to guide\n\nReturns a JSON guide explaining how to use webgate effectively. The AI should call this once at the start of a session if in doubt about which tool to use.\n\n## 🔧 Using webgate with local or smaller models\n\nMost frontier models follow MCP tool instructions automatically. Smaller or local models sometimes ignore the server-provided guidance and fall back to a built-in fetch tool instead — returning raw HTML that floods the context with noise.\n\nIf you notice this happening, add an explicit instruction block to your system prompt:\n\n```\nYou have access to webgate tools for web search and page retrieval.\nFollow these rules in every session:\n- To search the web: use webgate_query — never use a built-in fetch, browser, or HTTP tool\n- To retrieve a URL: use webgate_fetch — never fetch URLs directly\n- Built-in fetch tools return raw HTML that floods your context; webgate returns clean, bounded text\nAt the start of each session, call webgate_onboarding to read the full operational guide.\n```\n\nThis works because user system prompt instructions take precedence over MCP server-level guidance, making the constraint explicit at the highest-priority layer the model sees.\n\n> **Tip:** if your client supports named system prompts or prompt templates, save the block above as a reusable preset so you don't have to paste it every time.\n\n## 🎛️ Tuning\n\nThis section explains what the key parameters do and when to change them. The defaults work well for most cases — only tweak if you have a specific reason.\n\n### What is a \"character budget\"?\n\nwebgate measures text in **characters** (not tokens). A rough conversion for English text:\n\n> 4 characters ≈ 1 token\n\n| Characters | Approximate tokens |\n|------------|-------------------|\n| 8,000 | ~2,000 |\n| 32,000 | ~8,000 |\n| 96,000 | ~24,000 |\n\n### `webgate_fetch` budget\n\nWhen you fetch a single URL, the ceiling is `max_query_budget` (default **32,000 chars**). The tool parameter `max_chars` can request less, but never more than this ceiling.\n\n**Why `max_query_budget` and not `max_result_length`?** Because you're fetching one page — the \"total output\" IS that one page, so the right limit is the overall context budget, not the per-page cap designed for multi-source queries.\n\n### `webgate_query` budget — without LLM\n\nWith no LLM, the cleaned sources go directly to your AI's context. webgate distributes `max_query_budget` across all fetched pages so the total never exceeds the budget:\n\n> **Per-page limit** = `max_query_budget` ÷ number of results\n> (capped at `max_result_length`)\n\n| Results fetched | Per-page limit | Total output |\n|-----------------|---------------|-------------|\n| 1 | 8,000 (cap) | ≤ 8,000 |\n| 5 | 6,400 | ≤ 32,000 |\n| 10 | 3,200 | ≤ 32,000 |\n| 20 | 1,600 | ≤ 32,000 |\n\nThe total output is always at most `max_query_budget`, regardless of how many results you request — the per-page share automatically shrinks to compensate.\n\n### `webgate_query` budget — with LLM summarization\n\nWhen a secondary LLM is summarizing, it *compresses* the content before passing the result to your primary AI. This means it's safe — and beneficial — to give it more raw material to work from.\n\nwebgate scales up the input using `input_budget_factor` (default **3**):\n\n> **LLM input budget** = `max_query_budget` × `input_budget_factor`\n> Default: 32,000 × 3 = **96,000 chars**\n\n| Results fetched | LLM input / page | Total LLM input | Output to your AI |\n|-----------------|-----------------|----------------|------------------|\n| 1 | 96,000 | 96,000 | compact report |\n| 5 | 19,200 | 96,000 | compact report |\n| 10 | 9,600 | 96,000 | compact report |\n| 20 | 4,800 | 96,000 | compact report |\n\nThe secondary LLM sees much more content per page. Your primary AI sees only the final report — typically **1,000–3,000 tokens** — regardless of how many sources were processed. This is the main efficiency advantage of LLM mode.\n\n### Quick tuning guide\n\n| Symptom | Fix |\n|---------|-----|\n| AI responses feel slow, too much text | Reduce `max_query_budget` (e.g. `16000`) |\n| AI answers are shallow or miss details | Increase `max_query_budget` (e.g. `48000`) |\n| LLM summary is thin or misses things | Increase `input_budget_factor` (e.g. `5`) |\n| LLM summary times out or is very slow | Reduce `input_budget_factor` (e.g. `2`) or reduce `results_per_query` |\n| `fetch` returns too little of a long page | Increase `max_query_budget` (e.g. `64000`) |\n| Pages are slow to download | Reduce `max_download_mb` (e.g. `1`, already default) |\n| Server downloads too much garbage | Reduce `max_download_mb` (e.g. `1`) |\n\n## 🤖 LLM Features\n\nOptional, opt-in. When `llm.enabled = false` (the default), webgate is fully deterministic. Enable the `[llm]` block to unlock three extra capabilities.\n\n### 🤔 When to enable LLM features\n\n| Situation | Recommended setup | Typical latency overhead |\n|-----------|------------------|--------------------------|\n| Fast answers, general research | LLM **disabled** (default) — BM25-ranked clean sources, zero latency overhead | none |\n| Deep research on a complex topic | **Summarization on** — get a cited Markdown report instead of raw pages | +5–30s |\n| Broad topic, one query isn't enough | **Expansion + Summarization** — LLM generates variants and synthesizes all results | +6–35s |\n| Result order matters more than speed | **LLM reranking on** — semantic ordering at the cost of one extra LLM call per query | +1–5s |\n\n**Privacy:** with LLM disabled, no data leaves your machine except web requests. With LLM enabled, cleaned search results (not raw HTML) are sent to the configured `base_url`. Point it at a local Ollama instance to keep everything on-device.\n\n**Latency trade-off:** each enabled feature adds one LLM round-trip per query. Expansion adds ~1–5s; summarization adds ~5–30s depending on model and content volume. For interactive use, summarization with a fast local model (e.g. Gemma 3 4B) is a good starting point.\n\n### Setup\n\n```toml\n[llm]\nenabled  = true\nbase_url = \"http://localhost:11434/v1\"   # Ollama, OpenAI, LM Studio, vLLM, Groq...\napi_key  = \"\"                            # empty for local models\nmodel    = \"gemma3:27b\"\ntimeout  = 60                            # local 27B+ models may need up to 60s\n```\n\nOr with env vars:\n\n```json\n\"env\": {\n  \"WEBGATE_LLM_ENABLED\": \"true\",\n  \"WEBGATE_LLM_BASE_URL\": \"http://localhost:11434/v1\",\n  \"WEBGATE_LLM_MODEL\": \"gemma3:27b\",\n  \"WEBGATE_LLM_TIMEOUT\": \"60\"\n}\n```\n\n`base_url` accepts any OpenAI-compatible endpoint: **OpenAI**, **Ollama**, **LM Studio**, **vLLM**, **Together AI**, **Groq**, and others.\n\n### Query expansion\n\nWhen you send a single query and `expansion_enabled = true`, the LLM automatically generates complementary search variants before hitting the backend. If you already pass multiple queries, this step is skipped.\n\n```\n\"best laptop for programming\"\n    ↓ expansion\n[\"best laptop for programming 2024\", \"developer laptop recommendations\", \"laptop specs for coding\"]\n    ↓ all search in parallel\n```\n\nFalls back silently to your original query if the LLM fails.\n\n### Summarization\n\nWhen `summarization_enabled = true`, the LLM reads all fetched pages and writes a structured Markdown report with inline citations. Your AI receives the report instead of the raw text.\n\n- **Success**: `summary` + `citations` (lean output — no raw content passed to your AI)\n- **Failure**: `llm_summary_error` with the reason + full `sources` as fallback (your AI can still work with the cleaned content)\n\nThe report length target is `max_summary_words`. When `0` (default), it is derived from `max_query_budget / 5` — e.g. with a 32k budget, the target is ~6,400 words.\n\n### Reranking\n\nResults are always reranked by BM25 (keyword overlap, zero cost) before being returned. Optionally, the LLM can do a second pass for semantic relevance:\n\n| Tier | When | Cost |\n|------|------|------|\n| **BM25** (deterministic) | Always | Zero — pure math |\n| **LLM-assisted** | `llm_rerank_enabled = true` | One LLM call per query |\n\nLLM reranking adds latency proportional to your LLM response time. Enable it only if result ordering matters more than speed.\n\nPipeline: `clean → BM25 rerank → (LLM rerank) → (LLM summarize) → output`\n\n## 🔗 Integrations\n\nmcp-webgate works with all major AI clients:\n\n| Platform | Configuration Guide | Notes |\n|----------|---------------------|-------|\n| **Claude Desktop** | [IDE Integration](docs/integrations/IDE.md#claude-desktop) | Desktop application |\n| **Claude Code** | [IDE Integration](docs/integrations/IDE.md#claude-code) | CLI coding agent |\n| **Zed Editor** | [IDE Integration](docs/integrations/IDE.md#zed-editor) | Native MCP support |\n| **Cursor** | [IDE Integration](docs/integrations/IDE.md#cursor) | Requires Agent mode |\n| **Windsurf** | [IDE Integration](docs/integrations/IDE.md#windsurf) | Global config only |\n| **VSCode** | [IDE Integration](docs/integrations/IDE.md#vscode) | Via Copilot or MCP extension |\n| **Gemini CLI** | [Agent Integration](docs/integrations/AGENT.md#gemini-cli) | Google's CLI agent |\n| **Claude CLI** | [Agent Integration](docs/integrations/AGENT.md#claude-cli) | Anthropic's CLI agent |\n\n## 📦 Installation\n\n### Via uvx (recommended — no install needed)\n\n```bash\nuvx mcp-webgate\n```\n\n### Via pip / uv\n\n```bash\npip install mcp-webgate\n# or\nuv add mcp-webgate\n```\n\n## ⚙️ Full Configuration\n\nReady-to-use config files are in [`examples/`](examples/).\n\n### Resolution order\n\n```\nCLI args  >  env vars  >  webgate.toml  >  defaults\n```\n\nConfig is read once at startup; restart the server to apply changes.\n\nYou can configure webgate in three ways — mix and match as needed:\n\n- **`webgate.toml`** — checked at startup in `./webgate.toml` then `~/webgate.toml`\n- **Env vars** — `WEBGATE_*` prefix, always strings (MCP JSON requirement)\n- **CLI args** — `--kebab-case`, integers stay integers, ideal for multi-instance setups\n\n### Config file (`webgate.toml`)\n\n```toml\n[server]\nmax_download_mb    = 1        # how many MB to download per page before cutting off\nmax_result_length  = 8000     # max chars per page in multi-source queries (no LLM)\nmax_query_budget   = 32000    # total char budget for a fetch, or input pool for a query\nmax_search_queries = 5        # max parallel queries per call\nresults_per_query  = 5        # results to fetch per query\nsearch_timeout     = 8        # seconds before giving up on a page\noversampling_factor = 2       # fetch 2× more candidates than needed (dedup reserve)\nauto_recovery_fetch = false   # retry failed fetches from reserve pool\nmax_total_results  = 20       # hard cap: never fetch more than this many pages total\nblocked_domains    = [\"reddit.com\", \"pinterest.com\"]\nallowed_domains    = []       # if non-empty, only these domains are allowed\nadaptive_budget    = false   # [EXPERIMENTAL] proportional char allocation based on BM25 rank\nadaptive_budget_fetch_factor = 3  # generous pre-rank fetch multiplier\n\n[backends]\ndefault = \"searxng\"\n\n[backends.searxng]\nurl = \"http://localhost:8080\"\n\n[backends.brave]\napi_key = \"BSA...\"\n\n[backends.tavily]\napi_key = \"tvly-...\"\nsearch_depth = \"basic\"\n\n[llm]\nenabled  = true\nbase_url = \"http://localhost:11434/v1\"\napi_key  = \"\"\nmodel    = \"llama3.2\"\ntimeout  = 60\nexpansion_enabled     = true\nsummarization_enabled = true\nllm_rerank_enabled    = false\nmax_summary_words     = 0     # 0 = max_query_budget / 5 (e.g. 6400 with budget 32000)\ninput_budget_factor   = 3     # LLM input = max_query_budget × factor (default: 96000)\n```\n\n### MCP client config examples\n\n**With env vars** (all values must be strings):\n\n```json\n{\n  \"mcpServers\": {\n    \"webgate\": {\n      \"command\": \"uvx\",\n      \"args\": [\"mcp-webgate\"],\n      \"env\": {\n        \"WEBGATE_DEFAULT_BACKEND\": \"searxng\",\n        \"WEBGATE_SEARXNG_URL\": \"http://localhost:8080\",\n        \"WEBGATE_LLM_ENABLED\": \"true\",\n        \"WEBGATE_LLM_TIMEOUT\": \"60\"\n      }\n    }\n  }\n}\n```\n\n**With CLI args** (integers stay integers — ideal for running independent instances in Zed, Cursor, etc.):\n\n```json\n{\n  \"mcpServers\": {\n    \"webgate\": {\n      \"command\": \"uvx\",\n      \"args\": [\n        \"mcp-webgate\",\n        \"--searxng-url\", \"http://localhost:8080\",\n        \"--llm-enabled\",\n        \"--llm-model\", \"gemma3:27b\",\n        \"--llm-timeout\", \"60\"\n      ]\n    }\n  }\n}\n```\n\nBoolean flags support `--flag` / `--no-flag` syntax (e.g. `--llm-enabled`, `--no-llm-rerank-enabled`).\n\n### Full reference\n\n| CLI argument | Env var | Default | Description |\n|---|---|---|---|\n| `--default-backend` | `WEBGATE_DEFAULT_BACKEND` | `searxng` | Active backend |\n| `--searxng-url` | `WEBGATE_SEARXNG_URL` | `http://localhost:8080` | SearXNG instance URL |\n| `--brave-api-key` | `WEBGATE_BRAVE_API_KEY` | _(empty)_ | Brave Search API key |\n| `--tavily-api-key` | `WEBGATE_TAVILY_API_KEY` | _(empty)_ | Tavily API key |\n| `--exa-api-key` | `WEBGATE_EXA_API_KEY` | _(empty)_ | Exa API key |\n| `--serpapi-api-key` | `WEBGATE_SERPAPI_API_KEY` | _(empty)_ | SerpAPI key |\n| `--serpapi-engine` | `WEBGATE_SERPAPI_ENGINE` | `google` | SerpAPI engine (`google`, `bing`, ...) |\n| `--serpapi-gl` | `WEBGATE_SERPAPI_GL` | `us` | SerpAPI country code |\n| `--serpapi-hl` | `WEBGATE_SERPAPI_HL` | `en` | SerpAPI language |\n| `--max-download-mb` | `WEBGATE_MAX_DOWNLOAD_MB` | `1` | Per-page download size cap (MB) |\n| `--max-result-length` | `WEBGATE_MAX_RESULT_LENGTH` | `8000` | Per-page char cap (no-LLM queries) |\n| `--max-query-budget` | `WEBGATE_MAX_QUERY_BUDGET` | `32000` | Total char budget for fetch and query |\n| `--max-search-queries` | `WEBGATE_MAX_SEARCH_QUERIES` | `5` | Max queries per call |\n| `--results-per-query` | `WEBGATE_RESULTS_PER_QUERY` | `5` | Default results fetched per query |\n| `--search-timeout` | `WEBGATE_SEARCH_TIMEOUT` | `8` | HTTP request timeout (seconds) |\n| `--oversampling-factor` | `WEBGATE_OVERSAMPLING_FACTOR` | `2` | Search result multiplier for dedup reserve |\n| `--auto-recovery-fetch` | `WEBGATE_AUTO_RECOVERY_FETCH` | `false` | Enable gap-filler (Round 2 fetch) |\n| `--max-total-results` | `WEBGATE_MAX_TOTAL_RESULTS` | `20` | Hard cap on total results per call |\n| `--debug` | `WEBGATE_DEBUG` | `false` | Enable structured debug logging |\n| `--log-file` | `WEBGATE_LOG_FILE` | _(empty)_ | Log file path (empty = stderr) |\n| `--trace` | `WEBGATE_TRACE` | `false` | Include content in summarized citations; also activates debug logging |\n| `--adaptive-budget` | `WEBGATE_ADAPTIVE_BUDGET` | `false` | [EXPERIMENTAL] Proportional char allocation based on BM25 rank |\n| `--adaptive-budget-fetch-factor` | `WEBGATE_ADAPTIVE_BUDGET_FETCH_FACTOR` | `3` | [EXPERIMENTAL] Generous pre-rank fetch multiplier |\n| `--llm-enabled` | `WEBGATE_LLM_ENABLED` | `false` | Enable LLM features |\n| `--llm-base-url` | `WEBGATE_LLM_BASE_URL` | `http://localhost:11434/v1` | OpenAI-compatible endpoint |\n| `--llm-api-key` | `WEBGATE_LLM_API_KEY` | _(empty)_ | API key (empty for local models) |\n| `--llm-model` | `WEBGATE_LLM_MODEL` | `llama3.2` | Model name |\n| `--llm-timeout` | `WEBGATE_LLM_TIMEOUT` | `30` | LLM request timeout (seconds) |\n| `--llm-expansion-enabled` | `WEBGATE_LLM_EXPANSION_ENABLED` | `true` | Auto-expand queries into variants |\n| `--llm-summarization-enabled` | `WEBGATE_LLM_SUMMARIZATION_ENABLED` | `true` | LLM summary with citations |\n| `--llm-rerank-enabled` | `WEBGATE_LLM_RERANK_ENABLED` | `false` | LLM-assisted reranking |\n| `--llm-max-summary-words` | `WEBGATE_LLM_MAX_SUMMARY_WORDS` | `0` | Summary word target (0 = auto) |\n| `--llm-input-budget-factor` | `WEBGATE_LLM_INPUT_BUDGET_FACTOR` | `3` | LLM input budget multiplier |\n\n## 🔌 Backends\n\n| Backend | Auth | Notes |\n|---------|------|-------|\n| **SearXNG** | none | Self-hosted, recommended |\n| **Brave Search** | API key | High quality, [free tier available](https://brave.com/search/api/) |\n| **Tavily** | API key | AI-oriented snippets, [free tier available](https://tavily.com/) |\n| **Exa** | API key | Neural/semantic search, [free tier available](https://exa.ai/) |\n| **SerpAPI** | API key | Proxy for Google, Bing, DuckDuckGo and more, [free tier available](https://serpapi.com/) |\n\n### SearXNG quickstart (Docker)\n\n```bash\ndocker run -d -p 8080:8080 --name searxng searxng/searxng\n```\n\nThen set `WEBGATE_SEARXNG_URL=http://localhost:8080`.\n\n### Exa notes\n\nExa uses neural (semantic) search by default — the primary reason to use it over keyword backends. `use_autoprompt` is hardcoded to `false` (not user-configurable) because mcp-webgate handles query expansion via its own LLM expander.\n\n### SerpAPI notes\n\n`engine` selects the underlying search engine (`google`, `bing`, `duckduckgo`, `yandex`, `yahoo`). `gl` and `hl` significantly affect result quality for non-English queries.\n\n## 🐛 Debug mode\n\nWhen enabled, every tool call logs a structured entry:\n\n- **`fetch`**: URL, raw KB downloaded, clean KB returned, elapsed ms\n- **`query`**: queries used, results requested/fetched/failed, raw MB, clean KB, total elapsed ms\n\n```bash\nexport WEBGATE_DEBUG=true             # log to stderr\nexport WEBGATE_LOG_FILE=/tmp/wg.log  # or log to file\n```\n\n## 🛡️ Protections summary\n\nThese protections are always active — they are the core value proposition and cannot be disabled.\n\n| What could go wrong | How webgate stops it |\n|---------------------|---------------------|\n| Page dumps 2 MB of HTML | `max_download_mb` hard cap — download stops mid-stream, never buffered |\n| Cleaned text is still huge | `max_result_length` char cap per page |\n| Many results flood the context | `max_query_budget` distributes a fixed total across all results |\n| Too many pages fetched | `max_total_results` hard cap |\n| PDF / ZIP / DOCX requested | Binary extension filter runs *before* any network request |\n| Slow or hanging connections | `search_timeout` + 5s connect timeout |\n| Invisible Unicode tricks in content | Full regex sterilization pipeline (zero-width, BiDi, etc.) |\n| Rate limiting (429 / 502 / 503) | Exponential retry backoff, respects `Retry-After` header |\n| Unwanted domains | `blocked_domains` / `allowed_domains` filter |\n\n## 📚 Documentation Structure\n\n### Integration Guides\n- **[IDE Integration](docs/integrations/IDE.md)** — Claude Desktop, Claude Code, Zed, Cursor, Windsurf, VSCode\n- **[Agent Integration](docs/integrations/AGENT.md)** — Gemini CLI, Claude CLI, custom agents\n- **[Advanced Features](ADVANCED.md)** — BM25/LLM reranking internals, adaptive budget allocation\n\n<!-- RECENT_CHANGES_START -->\n<!-- RECENT_CHANGES_END -->\n\n## 🧪 Beta Status\n\nmcp-webgate is in **beta**. Core functionality is stable and the server is used in production,\nbut the configuration API may still change before 1.0.\n\n**Feedback is very welcome.** If something doesn't work as expected, behaves oddly,\nor you have a use case that isn't covered:\n\n→ [Open an issue on GitHub](https://github.com/x-hannibal/mcp-webgate/issues)\n\nBug reports, configuration questions, and feature requests all help shape the roadmap.\n\n## 🤝 Contributing\n\nContributions are welcome! Please see [CONTRIBUTING.md](CONTRIBUTING.md) for detailed guidelines on:\n- Development setup and workflow\n- Code style and conventions\n- Testing requirements\n- Documentation standards\n- Pull request process\n\n## 📄 License\n\nMIT License — see [LICENSE](LICENSE) for details.\n\n## 🔗 Links\n\n- **[GitHub Repository](https://github.com/x-hannibal/mcp-webgate)** — Source code and issues\n- **[PyPI Package](https://pypi.org/project/mcp-webgate/)** — Python Package Index\n- **[MCP Registry](https://registry.modelcontextprotocol.io/?q=mcp-webgate&all=1)** — Model Context Protocol Registry\n- **[MCP Protocol](https://modelcontextprotocol.io/specification/2025-11-25)** — Model Context Protocol specification\n\n---\n\n**Need help?** Check the [documentation](docs/) or open an [issue](https://github.com/x-hannibal/mcp-webgate/issues) on GitHub.\n\n<!-- mcp-name: io.github.x-hannibal/mcp-webgate -->\n",
  "bytes": 26209,
  "sha": "23ddc805f22228b7b5a48dc8b35d380f035faaac83662f634f66e3ab7a41b0c1",
  "repo_slug": "x-hannibal/mcp-webgate",
  "fonte": "repo",
  "truncated": false,
  "api": "https://agentalog.com/api/listings/mcp_io_github_x_hannibal_mcp_webgate_3758d5e3/readme"
}