{
  "markdown": "# WebShift\r\n\r\n[![Crates.io](https://img.shields.io/crates/v/webshift.svg)](https://crates.io/crates/webshift)\r\n[![docs.rs](https://img.shields.io/docsrs/webshift)](https://docs.rs/webshift)\r\n[![Latest Release](https://img.shields.io/badge/release-v0.2.14-purple.svg)](https://github.com/x-hannibal/webshift/releases/tag/v0.2.14)\r\n[![Beta](https://img.shields.io/badge/status-beta-blue.svg)](https://github.com/x-hannibal/webshift/issues)\r\n[![License](https://img.shields.io/badge/license-MIT-green.svg)](https://github.com/x-hannibal/webshift/blob/main/LICENSE)\r\n<!--[![MCP Protocol](https://img.shields.io/badge/MCP-Protocol-blueviolet)](https://modelcontextprotocol.io/specification/2025-11-25)-->\r\n\r\n---\r\n\r\n## What is WebShift\r\n\r\nWebShift is a Rust library and MCP server that shifts noisy web pages into\r\nclean, right-sized text for LLM consumption.\r\n\r\nRaw HTML is mostly junk: scripts, ads, navigation menus, cookie banners,\r\ntracking pixels. Feeding it directly to an LLM floods the context window\r\nwith tens of thousands of useless tokens and leaves no room for reasoning.\r\nWebShift strips all that noise, sterilizes the text, and enforces strict\r\nsize budgets so the model receives only the content that matters.\r\n\r\n### What you get\r\n\r\nDepending on the features you enable, WebShift can be four things:\r\n\r\n| Use case | Crate | Feature flags | What it does |\r\n|----------|-------|---------------|--------------|\r\n| **HTML denoiser** | `webshift` | `default-features = false` | `clean()` — pure Rust HTML-to-text pipeline. Strips noise elements, sterilizes Unicode/BiDi, collapses whitespace. Zero network, zero config. Drop into any Rust project that processes web content for LLMs. |\r\n| **HTML text rewriter** | `webshift` | `features = [\"text-map\"]` | `extract_text_nodes()` + `replace_text_nodes()` — extract individual text nodes from HTML, manipulate them (translate, rewrite, simplify), and rebuild the HTML with structure intact. Tags, attributes, and links are never touched. |\r\n| **Web content client** | `webshift` | `default` or `features = [\"llm\"]` | `fetch()` + `query()` — streaming HTTP fetcher with size caps, 8 search backends, BM25 reranking, optional LLM query expansion and summarization. Full pipeline from search query to structured results. |\r\n| **MCP server** | `webshift-mcp` | all features | Native binary (`mcp-webshift`) that exposes `webshift_query`, `webshift_fetch`, and `webshift_onboarding` over MCP stdio. Single static binary, zero runtime dependencies. |\r\n\r\n### When to use WebShift\r\n\r\n- You're building an AI agent that needs web search and you want clean,\r\n  budget-controlled text — not raw HTML.\r\n- You're processing web pages in a Rust pipeline and need a reliable\r\n  HTML-to-text cleaner that strips noise without losing real content.\r\n- You need an LLM to translate, rewrite, or simplify text inside HTML\r\n  without corrupting the markup — text-map gives you a safe round-trip.\r\n- You want an MCP web search server that works as a single binary —\r\n  no Python, no pip, no venv, no Docker (unless you want it).\r\n- You need hard guarantees on output size: per-page caps, total budget\r\n  caps, streaming download limits.\r\n\r\n### When NOT to use WebShift\r\n\r\n- You need a headless browser that renders JavaScript-heavy SPAs.\r\n  WebShift parses static HTML — it doesn't execute JS.\r\n- You need to render or screenshot a page preserving its visual layout.\r\n  WebShift processes HTML structure but does not render CSS or compute layout.\r\n  (Note: `text-map` does preserve the DOM structure for text rewriting use cases.)\r\n- You're building a web scraper that needs to navigate across pages,\r\n  fill forms, or handle authentication flows.\r\n\r\n---\r\n\r\n## How it works\r\n\r\n```\r\nQuestion\r\n  |\r\n  +- (optional) LLM query expansion -> multiple search variants\r\n  |\r\n  +- Search via backend (SearXNG, Brave, Tavily, Exa, SerpAPI, Google, Bing, HTTP)\r\n  |\r\n  +- Deduplicate + filter binary URLs\r\n  |\r\n  +- Streaming fetch with per-page size cap\r\n  |\r\n  +- HTML cleaning -> plain text (noise elements, scripts, nav removed)\r\n  |\r\n  +- Unicode/BiDi sterilization\r\n  |\r\n  +- BM25 deterministic reranking\r\n  |   +- (optional) LLM-assisted tier-2 reranking\r\n  |\r\n  +- Budget-aware truncation across all sources\r\n  |\r\n  +- (optional) LLM Markdown summary with inline citations\r\n  |\r\n  +- Structured JSON output\r\n```\r\n\r\nFor a detailed explanation of each pipeline stage, BM25 parameters, adaptive budget allocation, and real compression metrics see [Under the Hood](https://github.com/x-hannibal/webshift/blob/main/docs/UNDER_THE_HOOD.md). For the full configuration reference (TOML, env vars, CLI args) see [Configuration](https://github.com/x-hannibal/webshift/blob/main/docs/CONFIGURATION.md). For ready-to-use examples see [Use Cases](https://github.com/x-hannibal/webshift/blob/main/docs/USE_CASES.md).\r\n\r\n---\r\n\r\n## Installation\r\n\r\n### Binary (MCP server)\r\n\r\n```bash\r\ncargo install webshift-mcp\r\n```\r\n\r\nThe binary is called `mcp-webshift`.\r\n\r\n### From source\r\n\r\n```bash\r\ncargo install --path crates/webshift-mcp\r\n```\r\n\r\n### As a library\r\n\r\n```toml\r\n# Full pipeline (search + fetch + clean + rerank)\r\nwebshift = \"0.2\"\r\n\r\n# Cleaner + fetcher only (no search backends)\r\nwebshift = { version = \"0.2\", default-features = false }\r\n\r\n# Text-map only (extract/replace text nodes in HTML)\r\nwebshift = { version = \"0.2\", default-features = false, features = [\"text-map\"] }\r\n\r\n# Everything including LLM features\r\nwebshift = { version = \"0.2\", features = [\"llm\"] }\r\n```\r\n\r\n---\r\n\r\n## Quick start\r\n\r\n### 1. Set up a search backend\r\n\r\nThe easiest option is [SearXNG](https://docs.searxng.org/) — free, self-hosted, no API key:\r\n\r\n```bash\r\ndocker run -d -p 8080:8080 searxng/searxng\r\n```\r\n\r\nNo Docker? Use a cloud backend — see [Search backends](#search-backends).\r\n\r\n### 2. Configure your MCP client\r\n\r\n```json\r\n{\r\n  \"mcpServers\": {\r\n    \"webshift\": {\r\n      \"command\": \"mcp-webshift\",\r\n      \"args\": [\"--default-backend\", \"searxng\"]\r\n    }\r\n  }\r\n}\r\n```\r\n\r\nThat's it. The agent now has `webshift_query`, `webshift_fetch`, and `webshift_onboarding`.\r\n\r\nFor client-specific setup see [docs/integrations/](https://github.com/x-hannibal/webshift/tree/main/docs/integrations).\r\n\r\n---\r\n\r\n## MCP tools\r\n\r\n| Tool | Description |\r\n|------|-------------|\r\n| `webshift_query` | Full search pipeline: search + fetch + clean + rerank + (optional) summarize |\r\n| `webshift_fetch` | Single page fetch and clean |\r\n| `webshift_onboarding` | Returns a JSON guide for the agent (budgets, backends, tips) |\r\n\r\n### `webshift_query` parameters\r\n\r\n| Parameter | Type | Default | Description |\r\n|-----------|------|---------|-------------|\r\n| `queries` | string or list | required | Search query or list of queries |\r\n| `num_results_per_query` | integer | 5 | Results per query |\r\n| `lang` | string | none | Language filter (e.g. `\"en\"`) |\r\n| `backend` | string | config default | Override search backend |\r\n\r\n---\r\n\r\n## Configuration\r\n\r\nResolution order (highest priority first):\r\n\r\n1. **CLI args** — `--default-backend`, `--brave-api-key`, etc.\r\n2. **Environment variables** — `WEBSHIFT_*` prefix\r\n3. **Config file** — `webshift.toml` (current dir, then `~/webshift.toml`)\r\n4. **Built-in defaults**\r\n\r\n### Config file\r\n\r\n```toml\r\n[server]\r\nmax_query_budget    = 32000   # total char budget across all sources\r\nmax_result_length   = 8000    # per-page char cap\r\nmax_total_results   = 20      # hard cap on results per call\r\nmax_download_mb     = 1       # streaming cap per page (MB)\r\nsearch_timeout      = 8       # seconds\r\nresults_per_query   = 5\r\noversampling_factor = 2\r\nadaptive_budget     = \"auto\"  # \"auto\" | \"on\" | \"off\" — budget allocation mode\r\n\r\n[backends]\r\ndefault = \"searxng\"\r\n\r\n[backends.searxng]\r\nurl = \"http://localhost:8080\"\r\n\r\n[backends.brave]\r\napi_key = \"BSA-...\"\r\n\r\n[backends.tavily]\r\napi_key = \"tvly-...\"\r\n\r\n[backends.exa]\r\napi_key = \"...\"\r\n\r\n[backends.serpapi]\r\napi_key = \"...\"\r\nengine  = \"google\"    # google | bing | duckduckgo | yandex\r\n\r\n[backends.google]\r\napi_key = \"...\"\r\ncx      = \"...\"       # Custom Search Engine ID\r\n\r\n[backends.bing]\r\napi_key = \"...\"\r\nmarket  = \"en-US\"\r\n\r\n[backends.http]\r\nurl           = \"https://my-search.example.com/api/search\"\r\nquery_param   = \"q\"\r\ncount_param   = \"limit\"\r\nresults_path  = \"data.items\"     # dot-path to results array in JSON response\r\ntitle_field   = \"title\"\r\nurl_field     = \"link\"\r\nsnippet_field = \"description\"\r\n\r\n[backends.http.headers]\r\n\"Authorization\" = \"Bearer my-token\"\r\n\r\n[llm]\r\nenabled               = false\r\nbase_url              = \"http://localhost:11434/v1\"   # OpenAI-compatible\r\napi_key               = \"\"\r\nmodel                 = \"gemma3:27b\"\r\ntimeout               = 60\r\nexpansion_enabled     = true\r\nsummarization_enabled = true\r\nllm_rerank_enabled    = false\r\n```\r\n\r\nFor every setting with all three config methods (TOML, env vars, CLI args)\r\nand plain-language descriptions, see the full [Configuration Reference](https://github.com/x-hannibal/webshift/blob/main/docs/CONFIGURATION.md).\r\nReady-to-use config examples are in [Use Cases](https://github.com/x-hannibal/webshift/blob/main/docs/USE_CASES.md) and [`examples/`](https://github.com/x-hannibal/webshift/tree/main/examples).\r\n\r\n### Key environment variables\r\n\r\n```bash\r\nWEBSHIFT_DEFAULT_BACKEND=searxng\r\nWEBSHIFT_SEARXNG_URL=http://localhost:8080\r\nWEBSHIFT_BRAVE_API_KEY=BSA-xxx\r\nWEBSHIFT_GOOGLE_API_KEY=xxx\r\nWEBSHIFT_GOOGLE_CX=xxx\r\nWEBSHIFT_BING_API_KEY=xxx\r\nWEBSHIFT_LLM_ENABLED=true\r\nWEBSHIFT_LLM_BASE_URL=http://localhost:11434/v1\r\nWEBSHIFT_LLM_MODEL=gemma3:27b\r\n```\r\n\r\n---\r\n\r\n## Search backends\r\n\r\n| Backend | Auth | Notes |\r\n|---------|------|-------|\r\n| **SearXNG** | none | Self-hosted, free. Default: `http://localhost:8080` |\r\n| **Brave** | API key | Free tier. [brave.com/search/api](https://brave.com/search/api/) |\r\n| **Tavily** | API key | AI-oriented. [tavily.com](https://tavily.com/) |\r\n| **Exa** | API key | Neural search. [exa.ai](https://exa.ai/) |\r\n| **SerpAPI** | API key | Multi-engine proxy (Google, Bing, DDG...). [serpapi.com](https://serpapi.com/) |\r\n| **Google** | API key + CX | Custom Search. Free: 100 req/day. [programmablesearchengine.google.com](https://programmablesearchengine.google.com/) |\r\n| **Bing** | API key | Web Search API. Free: 1,000 req/month. [Microsoft Azure](https://www.microsoft.com/en-us/bing/apis/bing-web-search-api) |\r\n| **HTTP** | configurable | Generic REST backend — no code required, TOML-only config |\r\n\r\n---\r\n\r\n## LLM features (optional)\r\n\r\nAll opt-in — disabled by default, no data leaves your machine unless enabled.\r\n\r\n| Feature | What it does |\r\n|---------|-------------|\r\n| **Query expansion** | Single query -> N complementary search variants |\r\n| **Summarization** | Markdown report with inline `[1]` `[2]` citations |\r\n| **LLM reranking** | Tier-2 reranking on top of deterministic BM25 |\r\n\r\n> **Cross-language normalization (bonus):** when BM25 reranking surfaces pages in\r\n> foreign languages (e.g. Chinese, Japanese, Arabic), the LLM summarizer still\r\n> produces the final report in the prompt language. The agent receives clean,\r\n> readable output regardless of the language mix in the source pages.\r\n\r\nWorks with any OpenAI-compatible API (OpenAI, Ollama, vLLM, LM Studio, etc.):\r\n\r\n```toml\r\n[llm]\r\nenabled  = true\r\nbase_url = \"http://localhost:11434/v1\"\r\nmodel    = \"gemma3:27b\"\r\n```\r\n\r\n---\r\n\r\n## Anti-flooding protections\r\n\r\nAlways active — the core value proposition:\r\n\r\n| Protection | Description |\r\n|------------|-------------|\r\n| `max_download_mb` | Streaming cap — never buffers full response |\r\n| `max_result_length` | Hard cap on characters per cleaned page |\r\n| `max_query_budget` | Total character budget across all sources |\r\n| `max_total_results` | Hard cap on results per call |\r\n| Binary filter | `.pdf`, `.zip`, `.exe`, etc. filtered **before** any network request |\r\n| Unicode sterilization | BiDi control chars, zero-width chars removed |\r\n\r\n---\r\n\r\n## Library usage\r\n\r\n```rust\r\nuse webshift::{Config, clean, fetch, query};\r\n\r\n// Clean raw HTML — cap output at 8000 chars\r\nlet result = clean(\"<html><body><p>Hello world</p></body></html>\", 8000);\r\nprintln!(\"{}\", result.text);\r\n\r\n// Pass 0 to disable the per-page cap entirely (no truncation)\r\nlet full = clean(\"<html><body><p>Hello world</p></body></html>\", 0);\r\nassert!(!full.truncated);\r\n\r\n// Fetch and clean a single page\r\nlet config = Config::default();\r\nlet page = fetch(\"https://example.com\", &config).await?;\r\n\r\n// Full search pipeline\r\nlet results = query(&[\"rust async programming\"], &config).await?;\r\nfor source in &results.sources {\r\n    println!(\"[{}] {} — {} chars\", source.id, source.title, source.content.len());\r\n}\r\n\r\n// Backend partial failures (CAPTCHA, rate-limits, engine outages) are surfaced\r\n// in `warnings` rather than as Err. Empty sources + non-empty warnings means\r\n// \"all backends blocked\" — distinct from a legitimate \"no matches found\".\r\nif results.sources.is_empty() && !results.warnings.is_empty() {\r\n    eprintln!(\"all backends failed: {:?}\", results.warnings);\r\n}\r\n```\r\n\r\n### Text-map: rewrite HTML content without breaking markup\r\n\r\nExtract text nodes, manipulate them (translate, rewrite, simplify), and rebuild\r\nthe HTML with structure, attributes, and links intact.\r\n\r\n```rust\r\nuse webshift::{extract_text_nodes, replace_text_nodes, TextReplacement};\r\n\r\nlet html = r#\"<p>Hello <a href=\"https://example.com\">world</a></p>\"#;\r\nlet map = extract_text_nodes(html);\r\n// map.nodes = [(0, \"Hello\"), (1, \"world\")]\r\n\r\nlet replacements = vec![\r\n    TextReplacement { id: 0, text: \"Ciao\".into() },\r\n    TextReplacement { id: 1, text: \"mondo\".into() },\r\n];\r\nlet result = replace_text_nodes(html, &replacements).unwrap();\r\n// → <p>Ciao <a href=\"https://example.com\">mondo</a></p>\r\n// href untouched, tags intact, only text changed.\r\n```\r\n\r\nRequires `features = [\"text-map\"]`. See [Use Cases #11](https://github.com/x-hannibal/webshift/blob/main/docs/USE_CASES.md#11-translate-html-without-breaking-layout-text-map) for a full translation example.\r\n\r\n### Feature flags\r\n\r\n| Feature | Default | Enables |\r\n|---------|---------|---------|\r\n| `backends` | on | All search backends + query pipeline |\r\n| `llm` | off | LLM client, expander, summarizer, LLM reranking |\r\n| `text-map` | off | `extract_text_nodes()` + `replace_text_nodes()` — DOM round-trip for content rewriting |\r\n\r\n---\r\n\r\n## Integrations\r\n\r\n| Platform | Guide |\r\n|----------|-------|\r\n| Claude Desktop, Claude Code, Cursor, Windsurf, VS Code | [IDE Integration](https://github.com/x-hannibal/webshift/blob/main/docs/integrations/IDE.md) |\r\n| **Zed** — native extension with auto-download and Configure Server modal | [Zed Extension](https://github.com/x-hannibal/webshift/blob/main/docs/integrations/ZED_EXTENSION.md) |\r\n| Gemini CLI, Claude CLI, custom agents | [Agent Integration](https://github.com/x-hannibal/webshift/blob/main/docs/integrations/AGENT.md) |\r\n\r\n---\r\n\r\n## Beta Status\r\n\r\nWebShift is in **beta**. Core functionality is stable and the server is used daily,\r\nbut the API surface may still change before 1.0.\r\n\r\n**Feedback is very welcome.** If something doesn't work as expected, behaves oddly,\r\nor you have a use case that isn't covered:\r\n\r\n> [Open an issue on GitHub](https://github.com/x-hannibal/webshift/issues)\r\n\r\nBug reports, configuration questions, and feature requests all help shape the roadmap.\r\n\r\n## Contributing\r\n\r\nContributions are welcome! Please see [CONTRIBUTING.md](https://github.com/x-hannibal/webshift/blob/main/CONTRIBUTING.md) for detailed guidelines on:\r\n- Development setup and workflow\r\n- Code style and conventions\r\n- Testing requirements\r\n- Documentation standards\r\n- Pull request process\r\n\r\n## License\r\n\r\nMIT License — see [LICENSE](https://github.com/x-hannibal/webshift/blob/main/LICENSE) for details.\r\n\r\n## Links\r\n\r\n- **[GitHub Repository](https://github.com/x-hannibal/webshift)** — Source code and issues\r\n- **[Docs.rs](https://docs.rs/webshift)** — API documentation \r\n- **[MCP Registry](https://registry.modelcontextprotocol.io/?q=mcp-webshift&all=1)** — WebShift on Model Context Protocol Registry\r\n- **[MCP Protocol](https://modelcontextprotocol.io/specification/2025-11-25)** — Model Context Protocol specification\r\n\r\n---\r\n\r\n**Need help?** Check the [documentation](https://github.com/x-hannibal/webshift/tree/main/docs) or open an [issue](https://github.com/x-hannibal/webshift/issues) on GitHub.\r\n\r\n<!-- mcp-name: io.github.x-hannibal/mcp-webshift -->\r\n",
  "bytes": 16385,
  "sha": "222767451be389fdc186cb1a17246d745f68aac29ce666f3c4eb878ce9780897",
  "repo_slug": "annibale-x/webshift",
  "fonte": "repo",
  "truncated": false,
  "api": "https://agentalog.com/api/listings/mcp_io_github_annibale_x_webshift_a27ba842/readme"
}