{ "markdown": "
\n\n# SearchForge\n\n### Open-source web search API and MCP server for LLMs, agents, and RAG.\n\n[![CI](https://github.com/divyanshu-iitian/SearchForge/actions/workflows/ci.yml/badge.svg)](https://github.com/divyanshu-iitian/SearchForge/actions/workflows/ci.yml)\n[![License: MIT](https://img.shields.io/badge/license-MIT-22c55e.svg)](LICENSE)\n[![Node.js](https://img.shields.io/badge/node-%3E%3D20-339933?logo=nodedotjs&logoColor=white)](package.json)\n[![MCP](https://img.shields.io/badge/MCP-3_tools-8b5cf6)](https://modelcontextprotocol.io)\n[![Website](https://img.shields.io/badge/docs-SearchForge-087a54)](https://divyanshu-iitian.github.io/SearchForge/)\n[![Official MCP Registry](https://img.shields.io/badge/MCP_Registry-active-087a54)](https://registry.modelcontextprotocol.io/v0.1/servers?search=io.github.divyanshu-iitian/searchforge)\n\n**One local gateway. Intent-aware search. Clean Markdown. REST, MCP, CLI, and TypeScript.**\n\n[Website](https://divyanshu-iitian.github.io/SearchForge/) · [Quick start](#quick-start) · [Free tools](#free-tools) · [MCP](#mcp) · [API](#rest-api) · [Design](#how-it-works)\n\n
\n\n---\n\n**SearchForge is a free, open-source web search API and MCP server for LLMs, AI agents, and retrieval-augmented generation (RAG) pipelines.** It provides a predictable retrieval layer without forcing every project to integrate a paid search vendor. SearchForge routes each query to the right source, isolates provider failures, deduplicates URLs, fuses rankings, and turns public pages into LLM-ready Markdown.\n\nIt does **not** generate answers, hide citations, scrape public SearXNG instances, or send telemetry.\n\n## What you get\n\n| Capability | Default source | Cost / credentials |\n| ----------- | ----------------------------------- | ------------------------------ |\n| `auto` | Intent-routed source mix | No key by default |\n| `web` | Wikipedia; optional private SearXNG | No key / self-hosted |\n| `code` | GitHub repository search | No key; token optional |\n| `academic` | Crossref works and DOI metadata | No key |\n| `community` | Hacker News via Algolia | No key, community service |\n| `read_url` | Jina Reader | No key, currently rate-limited |\n\nSearchForge starts with all no-key adapters enabled. `auto` is the default and routes code, research, and current/community intent to relevant sources while retaining a web fallback. A GitHub token only raises the public API quota, and Brave remains an optional keyed backend. Broad, independent web metasearch is provided by the included SearXNG stack.\n\n## Quick start\n\n### Try it without cloning\n\n```bash\nnpx --yes --package github:divyanshu-iitian/SearchForge \\\n searchforge search \"latest open-source agent frameworks\"\n```\n\nThe first run downloads and builds the package from GitHub. Searches use intent-aware `auto` routing unless you select a category.\n\n### Zero-key local CLI\n\n```bash\ngit clone https://github.com/divyanshu-iitian/SearchForge.git\ncd SearchForge\nnpm install\nnpm run build\n\nnode dist/cli.js search \"latest open-source agent frameworks\"\nnode dist/cli.js search \"retrieval augmented generation\" --category academic\nnode dist/cli.js search \"local LLM tooling\" --category community\nnode dist/cli.js read \"https://example.com\"\nnode dist/cli.js doctor\n```\n\n### Full web search with private SearXNG\n\n```bash\ndocker compose up --build\n```\n\n```bash\ncurl -s http://localhost:3000/v1/search \\\n -H \"content-type: application/json\" \\\n -d '{\"query\":\"open source vector databases\",\"category\":\"web\",\"limit\":5}'\n```\n\nThis starts SearchForge on port `3000` and a private, JSON-enabled SearXNG on port `8080`. Before exposing the stack, change the SearXNG secret, set `SEARCHFORGE_API_KEY`, and terminate TLS at a trusted proxy.\n\n## Free tools\n\n### Search by capability\n\n```bash\nsearchforge search \"latest open-source agent frameworks\"\nsearchforge search \"browser agent\" --category code\nsearchforge search \"semantic reranking\" --category academic --json\nsearchforge search \"Show HN search engine\" --category community\n```\n\nThe default `auto` category detects code, academic, and current/community signals and queries the matching source families alongside the web fallback. Explicit categories prevent irrelevant providers from being queried. An explicit `providers` list overrides category routing, which is useful for evaluations.\n\n### Read a URL as Markdown\n\n```bash\nsearchforge read \"https://example.com/article\"\n```\n\n`read_url` accepts public HTTP(S) URLs only. Credentials, localhost, private IP literals, and non-web protocols are rejected. Responses are size-bounded, timed out, and cached.\n\n### Diagnose the whole retrieval path\n\n```bash\nsearchforge doctor\n```\n\nDoctor performs real, bounded probes and reports each provider's access tier, capability, latency, and error. A failed source produces `degraded`, not a misleading all-or-nothing status.\n\n## MCP\n\nSearchForge exposes three stdio tools:\n\n- `web_search` — routed, citation-ready structured search\n- `read_url` — clean Markdown from a public URL\n- `search_status` — live capability and latency report\n\n```json\n{\n \"mcpServers\": {\n \"searchforge\": {\n \"command\": \"node\",\n \"args\": [\"/absolute/path/to/SearchForge/dist/mcp.js\"],\n \"env\": {\n \"SEARCHFORGE_SEARXNG_URL\": \"http://localhost:8080\"\n }\n }\n }\n}\n```\n\nThe search and status tools return MCP structured content as well as readable text.\n\nRun the MCP server straight from GitHub without a clone:\n\n```json\n{\n \"mcpServers\": {\n \"searchforge\": {\n \"command\": \"npx\",\n \"args\": [\n \"--yes\",\n \"--package\",\n \"github:divyanshu-iitian/SearchForge\",\n \"searchforge-mcp\"\n ]\n }\n }\n}\n```\n\nSearchForge is also published in the\n[official MCP Registry](https://registry.modelcontextprotocol.io/v0.1/servers?search=io.github.divyanshu-iitian/searchforge)\nas `io.github.divyanshu-iitian/searchforge`. To run the registry-backed OCI image\ndirectly from any MCP client that supports a Docker command:\n\n```json\n{\n \"mcpServers\": {\n \"searchforge\": {\n \"command\": \"docker\",\n \"args\": [\n \"run\",\n \"--rm\",\n \"-i\",\n \"ghcr.io/divyanshu-iitian/searchforge-mcp:0.2.0\"\n ]\n }\n }\n}\n```\n\n## REST API\n\n### Search\n\n```http\nPOST /v1/search\nContent-Type: application/json\n\n{\n \"query\": \"open source reranking models\",\n \"category\": \"academic\",\n \"limit\": 8,\n \"language\": \"en\",\n \"freshness\": \"month\",\n \"safeSearch\": \"moderate\"\n}\n```\n\n```json\n{\n \"schemaVersion\": \"1.0\",\n \"query\": \"open source reranking models\",\n \"category\": \"academic\",\n \"results\": [\n {\n \"title\": \"Example work\",\n \"url\": \"https://doi.org/10.0000/example\",\n \"snippet\": \"Authors · Publisher · journal-article\",\n \"source\": \"crossref\",\n \"sources\": [\"crossref\"],\n \"score\": 0.016393\n }\n ],\n \"providers\": [\n {\n \"provider\": \"crossref\",\n \"ok\": true,\n \"latencyMs\": 241,\n \"resultCount\": 8\n }\n ],\n \"tookMs\": 243,\n \"cached\": false\n}\n```\n\n### Read\n\n```http\nPOST /v1/read\nContent-Type: application/json\n\n{\"url\":\"https://example.com/article\"}\n```\n\nOther endpoints:\n\n```text\nGET /healthz Process liveness\nGET /v1/providers Configured capabilities and access tiers\nGET /v1/doctor Live dependency health\n```\n\nSee the full [OpenAPI contract](openapi.yaml).\n\n## TypeScript SDK\n\n```ts\nimport {\n CrossrefProvider,\n GithubProvider,\n JinaReader,\n SearchForge,\n} from \"searchforge-rag\";\n\nconst forge = new SearchForge({\n providers: [new GithubProvider(), new CrossrefProvider()],\n reader: new JinaReader(),\n timeoutMs: 8_000,\n});\n\nconst evidence = await forge.search({\n query: \"agentic retrieval\",\n category: \"academic\",\n limit: 10,\n});\n\nconst page = await forge.read(\"https://example.com/research\");\n```\n\nUntil an npm release is published:\n\n```bash\nnpm install github:divyanshu-iitian/SearchForge\n```\n\n## Provider details\n\n| Provider | Capability | Access | Enabled |\n| ------------------------------------------------------------------------------ | ---------------------- | ------------------------------------------------------------------------ | ------------------------- |\n| [SearXNG](https://docs.searxng.org/) | Web | Self-hosted, no vendor fee | `SEARCHFORGE_SEARXNG_URL` |\n| Wikipedia | Web knowledge fallback | No key | Always |\n| [GitHub](https://docs.github.com/rest/search/search) | Code repositories | No key; 60 unauthenticated REST requests/hour, search has tighter limits | Always |\n| [Crossref](https://www.crossref.org/documentation/retrieve-metadata/rest-api/) | Academic metadata | No key; `mailto` recommended | Always |\n| [HN Algolia](https://hn.algolia.com/api) | Community | No key; community-operated availability | Always |\n| [Jina Reader](https://jina.ai/reader/) | URL to Markdown | No key; documented no-key quota currently 20 RPM | Always |\n| [Brave Search](https://brave.com/search/api/) | Web | API key | `BRAVE_SEARCH_API_KEY` |\n\nSearchForge intentionally does not configure public SearXNG instances. They often disable JSON or limit automated traffic; the Docker stack is the stable free path.\n\n## How it works\n\n```text\nAgent / RAG / MCP client\n |\n validate + route\n |\n +--------+---------+-----------+\n | | | |\n web code academic community read_url\n | | | | |\nSearXNG GitHub Crossref Hacker News Jina Reader\nWikipedia\n +--------+---------+-----------+\n |\n normalize -> canonicalize -> deduplicate -> reciprocal-rank fusion\n |\n versioned evidence + provenance + per-source health\n```\n\nEach idempotent provider call has its own abortable timeout. One outage cannot erase healthy results. Tracking parameters are removed before deduplication, and every contributing provider remains in `sources`.\n\nThis capability-first design is inspired by [Agent Reach](https://github.com/Panniantong/Agent-Reach). Agent Reach helps an agent operate many upstream tools directly; SearchForge complements that approach with one stable, embeddable retrieval API for RAG applications.\n\n## Configuration\n\n| Variable | Default | Purpose |\n| ------------------------------- | ----------: | ------------------------------ |\n| `SEARCHFORGE_SEARXNG_URL` | unset | Private SearXNG base URL |\n| `GITHUB_TOKEN` | unset | Optional GitHub quota increase |\n| `CROSSREF_MAILTO` | unset | Crossref polite-pool identity |\n| `BRAVE_SEARCH_API_KEY` | unset | Optional Brave backend |\n| `SEARCHFORGE_API_KEY` | unset | REST bearer or `x-api-key` |\n| `SEARCHFORGE_PORT` | `3000` | REST port |\n| `SEARCHFORGE_HOST` | `127.0.0.1` | Bind address |\n| `SEARCHFORGE_TIMEOUT_MS` | `8000` | Per-dependency timeout |\n| `SEARCHFORGE_CACHE_TTL_MS` | `300000` | In-memory cache TTL |\n| `SEARCHFORGE_CACHE_MAX_ENTRIES` | `500` | Cache entry bound |\n| `SEARCHFORGE_RATE_LIMIT` | `60` | Requests/client/minute |\n\n## Production boundary\n\n- Set an API key before binding to a public interface.\n- Search results and page content are untrusted input; delimit them and apply prompt-injection defenses.\n- The built-in cache and rate limiter are process-local. Use shared infrastructure for multiple replicas.\n- Provider bodies and credentials are excluded from surfaced errors.\n- `healthz` proves the process is alive; `/v1/doctor` checks dependencies.\n\nSee [SECURITY.md](SECURITY.md), [CONTRIBUTING.md](CONTRIBUTING.md), and [CHANGELOG.md](CHANGELOG.md).\n\n## Principles\n\n1. Evidence over generated answers\n2. Free and self-hosted paths before vendor lock-in\n3. Partial results over total failure\n4. Honest capability and quota reporting\n5. Stable contracts and explicit provenance\n6. No telemetry by default\n\n## License\n\nMIT © Divyanshu.\n\nIf SearchForge helps your agent, star the repository and share your integration in [Discussions](https://github.com/divyanshu-iitian/SearchForge/discussions).\n", "bytes": 13191, "sha": "9fef37259ef2ac6d4ca4ee4083e4747df64ba82de019753060277bdcc2af0276", "repo_slug": "divyanshu-iitian/searchforge", "fonte": "repo", "truncated": false, "api": "https://agentalog.com/api/listings/mcp_io_github_divyanshu_iitian_searchforge_e5c38a60/readme" }