{
  "markdown": "<h1 align=\"center\">mcp-retrieval</h1>\n\n<p align=\"center\">\n  <b>An MCP server that gives an LLM three web tools: search, image search, and page scraping — no API keys required.</b>\n</p>\n\n<p align=\"center\">\n  <a href=\"LICENSE\"><img src=\"https://img.shields.io/badge/License-MIT-8bc34a?style=for-the-badge\" alt=\"License MIT\"></a>\n  <img src=\"https://img.shields.io/badge/Go-1.25+-00ADD8?style=for-the-badge&logo=go&logoColor=white\" alt=\"Go\">\n  <img src=\"https://img.shields.io/badge/MCP-Server-6E56CF?style=for-the-badge&logo=anthropic&logoColor=white\" alt=\"MCP\">\n  <img src=\"https://img.shields.io/badge/Transport-stdio_%7C_http-26A5E4?style=for-the-badge\" alt=\"Transport\">\n  <img src=\"https://img.shields.io/badge/DuckDuckGo-Search-DE5833?style=for-the-badge&logo=duckduckgo&logoColor=white\" alt=\"DuckDuckGo\">\n  <img src=\"https://img.shields.io/badge/Bing-Images-008373?style=for-the-badge&logo=microsoftbing&logoColor=white\" alt=\"Bing Images\">\n  <img src=\"https://img.shields.io/badge/uTLS-Fingerprint-1f6feb?style=for-the-badge\" alt=\"uTLS\">\n</p>\n\n<p align=\"center\">\n  <a href=\"#tools\">Tools</a> ·\n  <a href=\"#quick-start\">Quick start</a> ·\n  <a href=\"#configuration\">Configuration</a> ·\n  <a href=\"#retrieval-engine\">Retrieval engine</a> ·\n  <a href=\"#architecture\">Architecture</a> ·\n  <a href=\"CONTRIBUTING.md\">Contributing</a>\n</p>\n\n<!-- TODO: add a demo here once it is recorded. GitHub renders a <video> tag\n     inline when the src points at an uploaded asset URL\n     (https://github.com/user-attachments/assets/...). -->\n\n---\n\n## What it is\n\n`mcp-retrieval` is a [Model Context Protocol](https://modelcontextprotocol.io) server written in Go. It exposes web retrieval capabilities to any MCP-compatible client (Claude Desktop, IDE agents, custom LLM apps) as three read-only tools. Under the hood it uses the [`retrieval-go`](https://github.com/free-llms-foundation/retrieval-go) library to search the web and fetch pages, returning results as clean Markdown ready to hand to a model.\n\nThe library needs **no API keys**: web search goes through DuckDuckGo Lite, image search through Bing Images, and page fetching runs the HTML through a readability extractor before converting it to Markdown. To stay reliable against bot protection it impersonates real browsers at the TLS level and can rotate both browser fingerprints and proxies — see [Retrieval engine](#retrieval-engine).\n\nBoth transports the MCP SDK supports are available and expose the identical tool set:\n\n- **stdio** — the client launches the binary and talks over stdin/stdout (the default, ideal for desktop clients).\n- **http** — a long-running streamable HTTP server (useful for remote/shared deployments).\n\n---\n\n## Tools\n\n| Tool | Description |\n| :--- | :--- |\n| **`web_search`** | Runs one or more queries in parallel and returns per-query deduplicated, reranked snippets with links. |\n| **`web_search_images`** | Runs one or more image queries in parallel and returns per-query deduplicated image results. |\n| **`web_scrape`** | Downloads one or more pages in parallel and returns the main article text as Markdown. |\n\nAll three are annotated as **read-only**. Each tool returns a structured JSON payload that matches its output schema; the SDK mirrors the same JSON into the text content block for clients that do not read `structuredContent`.\n\n### `web_search`\n\n| Parameter | Type | Default | Notes |\n| :--- | :--- | :--- | :--- |\n| `queries` | `[]string` | — | **Required.** Executed in parallel. |\n| `max_results` | `int` | `5` | Snippets per query, capped at `max_results` config (`20`). |\n| `timeout_ms` | `int64` | `5000` | Whole-call timeout; clamped to `[min, max]` from config. |\n| `date` | `string` | — | Freshness filter: `d` (day), `w` (week), `m` (month), `y` (year). |\n\n### `web_search_images`\n\n| Parameter | Type | Default | Notes |\n| :--- | :--- | :--- | :--- |\n| `queries` | `[]string` | — | **Required.** Executed in parallel. |\n| `max_images` | `int` | `5` | Images per query, capped at `max_images` config (`10`). |\n| `timeout_ms` | `int64` | `5000` | Whole-call timeout; clamped to `[min, max]` from config. |\n| `date` | `string` | — | Freshness filter: `d` / `w` / `m` / `y`. |\n\n### `web_scrape`\n\n| Parameter | Type | Default | Notes |\n| :--- | :--- | :--- | :--- |\n| `urls` | `[]string` | — | **Required.** Downloaded in parallel. |\n| `robots_txt` | `bool` | `false` | Respect the page's `robots.txt`. |\n| `timeout_ms` | `int64` | `5000` | Whole-call timeout; clamped to `[min, max]` from config. |\n| `remove_links` | `bool` | `false` | Strip Markdown links from the text. |\n| `max_chars` | `int` | `20000` | Truncate page text to N characters, capped at `max_document_chars` config (`20000`). |\n\n> Both `queries`/`urls` lists are capped at `max_queries` (`10`) items per call. Queries must be ≤ 512 characters; URLs ≤ 2048 characters and `http`/`https` only.\n\n### Results and counts\n\nEvery call fans out across the input list and returns one entry per query/URL, each with its own `status` — `success`, `failed`, or `timeout` — so a partial failure still returns the items that did work.\n\n`count` is the number of items actually returned, and it can be **lower than the requested `max_results` / `max_images`**: duplicates within a single query's results are removed before the limit is applied, and the upstream may simply have fewer items to give. A smaller `count` is a normal outcome, not an error.\n\nDeduplication is **per query, not across queries**. Each entry is deduplicated on its own, so a link found by two of the queries in the same call appears in both entries — dedupe the union yourself if you need it.\n\n### Errors\n\nRequest-level failures are returned as a tool result with `isError: true` and a plain-text message, not as a JSON-RPC error — the model reads the message and can correct the call itself. Per-item failures never do this; they stay inside the payload as `status: \"failed\"` / `\"timeout\"`.\n\nA call fails outright only when the input is rejected before any work starts, or when **every** item in it fails:\n\n| Message | Meaning |\n| :--- | :--- |\n| `invalid request` | The arguments did not pass validation. |\n| `too many queries` / `too many urls` | The list exceeds `MAX_QUERIES`. |\n| `query must not be empty` | An empty query, or an empty `queries` list. |\n| `query is too long` | A query exceeds 512 characters. |\n| `invalid url` | A URL is malformed, over 2048 characters, or not `http`/`https`. |\n| `robots.txt denied` | `robots_txt: true` and the page disallows fetching. |\n| `upstream service unavailable` | The upstream answered with an unexpected status code. |\n| `every url failed to be scraped; the pages may be unreachable or hold no extractable text` | All URLs failed. Individual causes are logged to `stderr`, not returned. |\n| `every query failed; the search upstream may be unreachable` | All queries failed. |\n| `internal server error` | Anything unclassified. |\n\nThe all-failed messages deliberately do not distinguish timeouts from other causes: a mixed batch can fail for several reasons at once, and the per-item `status` already carries that detail whenever at least one item survives.\n\n### Known limitations\n\n- **`web_scrape` handles HTML only.** Pages are run through a readability extractor, which needs article markup, so `text/plain` responses yield nothing and come back as `status: \"failed\"`. Raw-file hosts are the common case: `raw.githubusercontent.com`, `github.com/.../raw/...`, `cdn.jsdelivr.net`. Scrape the rendered page instead of the raw file.\n- **`web_search_images` relevance is not guaranteed.** For some queries Bing Images serves a page that is not a result set, and it is parsed as though it were — the tool then returns unrelated images with `status: \"success\"`. Treat image results as best-effort and verify them before showing them to a user.\n- **No JavaScript.** Pages are fetched as-is; content rendered client-side is invisible to the extractor.\n\n---\n\n## Quick start\n\n### Install\n\nPick whichever fits — all of them give the identical server.\n\n**Container** (no Go toolchain needed):\n\n```bash\ndocker pull ghcr.io/role1776/mcp-retrieval:latest\n```\n\n**Prebuilt binary** — grab the archive for your platform from the [latest release](https://github.com/Role1776/mcp-retrieval/releases/latest), unpack it, and put `mcp-retrieval` on your `PATH`.\n\n**MCP Bundle** — for clients that install `.mcpb` files, download `mcp-retrieval_<version>_<os>_<arch>.mcpb` from the [latest release](https://github.com/Role1776/mcp-retrieval/releases/latest) and open it with your client. The bundle carries the compiled binary, so it needs neither Docker nor Go. Pick the file matching your OS *and* CPU architecture: a bundle holds one native binary.\n\n**From source:**\n\n```bash\ngo install github.com/Role1776/mcp-retrieval/app/cmd/mcp-retrieval@latest   # needs Go 1.25.5+\n```\n\nOr build the binary in place (the Go module lives in `app/`):\n\n```bash\nmake build          # -> bin/mcp-retrieval\n```\n\n### Run\n\n```bash\n# defaults: stdio transport, no configuration needed\n./bin/mcp-retrieval\n\n# with an explicit env file\n./bin/mcp-retrieval -env /absolute/path/to/.env\n```\n\nThe one flag is optional:\n\n| Flag | Meaning |\n| :--- | :--- |\n| `-env` | Path to a `.env` file. If omitted — or if the file does not exist — the server starts on defaults and whatever is already in the environment. There is no implicit lookup: under stdio the working directory is chosen by the MCP client, so a relative default would be unpredictable. |\n\n### Connecting an MCP client (stdio)\n\nPoint your client at the built binary. Example Claude Desktop config:\n\n```json\n{\n  \"mcpServers\": {\n    \"retrieval\": {\n      \"command\": \"/absolute/path/to/mcp-retrieval\",\n      \"env\": {\n        \"MAX_RESULTS\": \"20\"\n      }\n    }\n  }\n}\n```\n\nThe `env` block is optional — `\"command\"` alone is enough.\n\n### Connecting an MCP client (container)\n\nRun the image on stdio. Configuration still travels through the `env` block, but Docker needs each variable named on the command line with `-e` for it to reach the process:\n\n```json\n{\n  \"mcpServers\": {\n    \"retrieval\": {\n      \"command\": \"docker\",\n      \"args\": [\n        \"run\", \"-i\", \"--rm\",\n        \"-e\", \"MAX_RESULTS\",\n        \"-e\", \"DEFAULT_TIMEOUT_MS\",\n        \"ghcr.io/role1776/mcp-retrieval:latest\"\n      ],\n      \"env\": {\n        \"MAX_RESULTS\": \"20\",\n        \"DEFAULT_TIMEOUT_MS\": \"5000\"\n      }\n    }\n  }\n}\n```\n\n`-i` is required — without it the container gets no stdin and the client sees the server die immediately. Clients that install from the [MCP Registry](https://registry.modelcontextprotocol.io) build this invocation themselves and prompt for the variables declared in [`server.json`](server.json).\n\n### Running over HTTP\n\nSet `MCP_TRANSPORT=http` and the server listens on `SERVER_PORT` at `MCP_PATH` (default `http://localhost:8080/mcp`).\n\n---\n\n## Configuration\n\nEverything is configured through **environment variables**, and each value is validated before startup: a non-numeric or non-positive value is a startup error. Relationships *between* limits are not checked at startup — see [Limits](#limits). Variables already present in the environment win over a `.env` file, so an MCP client's `env` block always takes effect. Every field has a sensible default, so the server runs with no configuration at all (stdio transport).\n\nSee [`.env.example`](.env.example) for the full list at its default values, ready to copy to `.env`.\n\n### MCP server\n\n| Env | Default | Notes |\n| :--- | :--- | :--- |\n| `MCP_TRANSPORT` | `stdio` | `stdio` or `http`. |\n| `MCP_NAME` | `mcp-retrieval` | Server name advertised to clients. |\n| `MCP_PATH` | `/mcp` | HTTP route (http transport only). |\n\nThe version advertised to clients is not configurable: it is stamped into the binary at build time from the git tag.\n\n### HTTP server (http transport only)\n\n| Env | Default |\n| :--- | :--- |\n| `SERVER_PORT` | `8080` |\n| `SERVER_READ_TIMEOUT` | `60s` |\n| `SERVER_WRITE_TIMEOUT` | `60s` |\n\n### HTTP client and proxy\n\n| Env | Default | Notes |\n| :--- | :--- | :--- |\n| `MAX_IDLE_CONNS_PER_HOST` | `100` | HTTP connection pooling. |\n| `PROXY_HOST` | — | Optional. If set, requests are routed through a rotating-session proxy. |\n| `PROXY_PORT` | — | Required when `PROXY_HOST` is set. |\n| `PROXY_SCHEME` | — | Required when `PROXY_HOST` is set. |\n| `PROXY_LOGIN` | — | Required when `PROXY_HOST` is set. |\n| `PROXY_PASSWORD` | — | Required when `PROXY_HOST` is set. |\n\nWhen a proxy is configured, each outbound request gets a unique session id appended to the login, so the upstream provider rotates the exit IP per request.\n\n### Limits\n\n| Env | Default |\n| :--- | :--- |\n| `MAX_QUERIES` | `10` |\n| `DEFAULT_RESULTS` | `5` |\n| `MAX_RESULTS` | `20` |\n| `DEFAULT_TIMEOUT_MS` | `5000` |\n| `MAX_TIMEOUT_MS` | `10000` |\n| `MIN_TIMEOUT_MS` | `1000` |\n| `DEFAULT_IMAGES` | `5` |\n| `MAX_IMAGES` | `10` |\n| `DEFAULT_DOCUMENT_CHARS` | `20000` |\n| `MAX_DOCUMENT_CHARS` | `20000` |\n\nEach value is checked on its own — it must be greater than zero — but the `DEFAULT_*`, `MIN_*` and `MAX_*` triples are **not** cross-checked against each other at startup. An inconsistent set does not stop the server; it is reconciled per request instead:\n\n- a value the caller omits, or passes as zero or negative, falls back to the matching `DEFAULT_*`;\n- the result is then clamped into `[MIN_*, MAX_*]`, so a `DEFAULT_*` larger than its `MAX_*` simply yields `MAX_*`;\n- if `MIN_*` exceeds `MAX_*`, the maximum wins.\n\nThe effective limit is therefore always within the configured maximum, and misconfiguration degrades to a working server rather than a failed start. The trade-off is that it degrades **silently**: a typo such as `MAX_RESULTS=2` instead of `20` produces no warning, only quietly smaller responses. Worth double-checking these values when results look truncated.\n\n### Logging\n\n| Env | Default | Notes |\n| :--- | :--- | :--- |\n| `LOG_MODE` | `local` | `local` → text handler at debug level; `prod` → JSON handler at info level. Logs go to `stderr`. |\n\n---\n\n## Architecture\n\nThe project follows a clean, layered structure. Dependencies point inward toward the domain, and each layer talks to the next through interfaces.\n\n```\napp/                       the Go module: sources plus its build files\n                           (Dockerfile, .dockerignore, .goreleaser.yaml)\n\ncmd/mcp-retrieval/main.go  entry point: parse flags, load config, run app\n\ninternal/\n  app/                     wiring + lifecycle (build server, run, graceful shutdown)\n  config/                  config loading (.env → env vars → validate)\n  domain/                  core types (Query, Link, Document, Snippet, Image) and errors\n  dto/web/                 request/response shapes for the MCP tools\n  transport/mcp/           MCP layer\n    router/                registers every tool group on the MCP server\n    web/                   tool handlers\n    utils/                 schema helpers and error → tool-result mapping\n  usecase/web/             business logic: validation, parallelism, timeouts, dedupe/limit/rerank\n  adapter/web/             retrieval-go client wiring (search, images, scrape, proxy)\n  pkg/                     reusable building blocks (mcpserver, server, logger, validator)\n```\n\nRequest flow for a tool call:\n\n```\nMCP client → transport/mcp/web (handler) → usecase/web → adapter/web → retrieval-go → the web\n                     ↑ maps errors               ↑ validates, fans out, limits results\n```\n\nSearch and scrape both fan out across the input list concurrently and aggregate per-item results, each with its own status (`success`, `failed`, `timeout`). A call only fails outright when **every** item in it fails.\n\n---\n\n## Retrieval engine\n\nAll network work is delegated to [`retrieval-go`](https://github.com/free-llms-foundation/retrieval-go), configured in [`app/internal/adapter/web`](app/internal/adapter/web/retrieval.go). Worth knowing:\n\n- **Sources.** Web search uses **DuckDuckGo Lite**; image search uses **Bing Images**; page fetching runs the raw HTML through a **readability** extractor and converts the main article to **Markdown** (tables included). No search-engine API keys are required.\n- **Browser impersonation.** The adapter enables `WithBrowserRotation()`, so each request is sent from one of ~11 real browser profiles picked at random. Every profile pairs a genuine **TLS/JA3 fingerprint** (via [uTLS](https://github.com/refraction-networking/utls)) with a matching `User-Agent` and client-hint headers — Chrome 133/131/120 (Windows/macOS/Linux), Edge 131, Firefox 120 (Windows/macOS), Safari 18.4 (macOS), and iOS 18.4 Safari. This makes the traffic look like ordinary browsers rather than a Go HTTP client, which is what keeps the free sources reachable.\n- **Proxy rotation.** When `PROXY_HOST` is configured, the adapter installs a proxy factory that appends a unique `session-<id>` to the proxy username on every request. With a session-based residential/rotating proxy provider, that yields a **fresh exit IP per request**, spreading load and avoiding rate limits. Without a proxy, requests go out directly.\n- **Response handling.** Responses are transparently decompressed (`gzip`, `br`, `zstd`, `deflate`), and keep-alive is disabled (`WithDisableKeepAlive()`) so pooled connections don't pin a single fingerprint/IP across requests.\n\nNone of this needs configuration to work — the defaults above are applied automatically. Only proxy credentials are optional extras.\n\n## Development\n\nEverything Go lives in `app/`, so either use the makefile from the repository\nroot or pass `-C app` to the toolchain:\n\n```bash\nmake build          # compile the binary\nmake test           # run tests\n\ngo -C app build ./...      # compile everything\ngo -C app test ./...       # run tests\ngo -C app vet ./...        # static checks\n```\n\nSee [`CONTRIBUTING.md`](CONTRIBUTING.md) for pull-request guidelines.\n\n## License\n\nReleased under the [MIT License](LICENSE).\n",
  "bytes": 17922,
  "sha": "ae57fbb542d2b5ec379f8922cadf81045804337d7a7082146b15daf0a2226541",
  "repo_slug": "role1776/mcp-retrieval",
  "fonte": "repo",
  "truncated": false,
  "api": "https://agentalog.com/api/listings/mcp_io_github_role1776_mcp_retrieval_1592baf9/readme"
}