{
  "markdown": "# qsearch\n\n> I built this for my own daily research. After running 100+ research sprints, my agent kept hallucinating because it read 200-char snippets. qsearch gives it full content with multi-engine provenance — running locally, owned by me.\n\n![License: Apache-2.0](https://img.shields.io/badge/license-Apache--2.0-blue.svg)\n![CI](https://github.com/theYahia/qsearch/actions/workflows/test.yml/badge.svg)\n![Status: v0.4.0 live](https://img.shields.io/badge/status-v0.4.0%20live-brightgreen.svg)\n![Demo: qsearch.pro](https://img.shields.io/badge/demo-qsearch.pro-ef4444.svg)\n![MCP](https://img.shields.io/badge/MCP-ready-8b5cf6.svg)\n[![MCP Registry](https://img.shields.io/badge/MCP%20Registry-io.github.theYahia%2Fqsearch-8b5cf6.svg)](https://registry.modelcontextprotocol.io/v0/servers?search=io.github.theYahia/qsearch)\n\nAI agents lose **17–33% of facts to hallucination** because they read 200-character snippets, not full pages ([Stanford 2024](https://dho.stanford.edu/wp-content/uploads/Legal_RAG_Hallucinations.pdf)). Existing search APIs hide *which engines agreed* on a result. Existing knowledge graphs are enterprise-priced or vendor-locked.\n\n**qsearch is the open-source search layer that gives agents full content with multi-engine provenance** — running on your machine, owned by you, ready for MCP today.\n\n> ✅ **v0.4.0 live at [qsearch.pro](https://qsearch.pro).** Multi-engine attribution, trust corpus with per-URL provenance (`engines[]`, `sweep_count`, `trust_score`), corpus viewer at `/ui`, MCP-over-HTTP for Claude Code and any spec-compliant client.\n> 📖 **Architecture:** [ARCHITECTURE.md](./ARCHITECTURE.md) · **Vision:** [docs/VISION.md](./docs/VISION.md) · **Technical spec:** [docs/TRUST_MESH.md](./docs/TRUST_MESH.md) · **Federation deep-dive:** [docs/FEDERATION_ARCHITECTURE.md](./docs/FEDERATION_ARCHITECTURE.md)\n\n## Demo\n\n![qsearch terminal demo — quickstart, multi-engine search, citation verify](assets/demo.gif)\n\n> 60-second tour: `/health` (it's up, local) → `/sweep` (multi-engine `engines[]` attribution) → `/verify` (does the cited source actually support the claim?). Reproduce it locally with [`bash scripts/record-demo.sh`](scripts/record-demo.sh) after `npm start` — see [docs/launch/README-demo-embed.md](docs/launch/README-demo-embed.md).\n\n## Quick start\n\n**Run it in 5 minutes — free tier, no API key:**\n\n```bash\ngit clone https://github.com/theYahia/qsearch.git && cd qsearch\ncp .env.example .env.local       # works as-is on the $0 SearXNG tier\ndocker compose up -d             # Meilisearch + Qdrant + SearXNG\nnpm install && npm start         # → qsearch on http://localhost:8080\n\n# multi-engine attribution in one call\ncurl -X POST http://localhost:8080/sweep \\\n  -H \"Content-Type: text/plain\" \\\n  --data-binary $'t1|self-hosted search engine\\n'\n# → parsed_snippets.md with \"Engines: google, duckduckgo, brave (count=3)\"\n```\n\nThe `broad` sweep tier runs on self-hosted SearXNG and costs nothing. Add a Brave key only when you want the `focused`/`critical` tiers. Full setup (Brave key, Ollama, MCP server) below.\n\n<details>\n<summary><b>Full setup</b> — Brave BYOK + local LLM + MCP server</summary>\n\n```bash\n# 1. Clone\ngit clone https://github.com/theYahia/qsearch.git\ncd qsearch\n\n# 2. Get a Brave Search API key (BYOK, $5/mo for ~1000 queries)\n#    → https://brave.com/search/api/ → sign up → copy key\n\n# 3. Configure\ncp .env.example .env.local\n# Set BRAVE_API_KEY=your_key\n# Set SEARXNG_URL=http://localhost:8888 (for multi-engine attribution)\n\n# 4. Start infrastructure (Meilisearch + Qdrant + SearXNG)\ndocker compose up -d\n\n# 5. (Optional) Pull Ollama models for local LLM cleaning + embedding rerank\n#    Without them, search still works — just no cleaned_markdown and no rerank.\nollama pull qwen2.5:7b-instruct   # ~5GB, cleaner (used by /sweep_context)\nollama pull nomic-embed-text      # 274MB, embedding rerank (Phase B)\n\n# 6. Install & run\nnpm install\nnpm start            # → qsearch v0.4.0 on http://localhost:8080\n\n# 7. (Optional) MCP server for Claude Code / Workbench / OpenClaw\nnpm run start:mcp    # → http://0.0.0.0:8081\n\n# 8. Test multi-engine attribution\ncurl -X POST http://localhost:8080/sweep \\\n  -H \"Content-Type: text/plain\" \\\n  --data-binary $'t1|self-hosted search engine\\n'\n# → parsed_snippets.md with \"Engines: google, duckduckgo, brave (count=3)\"\n```\n\n**BYOK design:** Brave key + SearXNG + Ollama all stay on your machine. No data exfiltration.\n\n</details>\n\n---\n\n## How I use it daily\n\nEvery research sprint I run a dual sweep:\n\n```bash\n# Brave sweep (primary, authoritative)\npython research/scripts/brave_sweep.py queries.txt _raw_data/topic_2026-04-28/brave/\n\n# qsearch sweep (secondary, auto-indexes into corpus)\ncurl -X POST http://localhost:8080/sweep?topic=my_topic \\\n  -H \"Content-Type: text/plain\" --data-binary @queries.txt\n```\n\nAfter 10+ sprints on the same domain, `/corpus/top?min_engines=3` shows which URLs survived multiple independent search engines across multiple sessions. Those are the ones I actually trust.\n\n---\n\n## Why qsearch exists\n\nEvery AI agent today hits the same broken loop:\n\n```\nAgent → Tavily/Exa/Serper API → 200-char snippets → hallucinated answer\n```\n\nThree failures:\n\n1. **Snippets aren't enough.** [Stanford's 2024 production RAG audit](https://dho.stanford.edu/wp-content/uploads/Legal_RAG_Hallucinations.pdf) measured 17–33% hallucination on Lexis+ AI and Westlaw despite \"hallucination-free\" claims. On Wikipedia QA, full content beats snippet-RAG by **+7.3pp** ([arxiv 2501.01880](https://arxiv.org/html/2501.01880v1)).\n\n2. **No trust signal.** Search APIs return ranked lists without telling you *which engines agreed*. SEO-spam at position 3 looks identical to authoritative source at position 4.\n\n3. **No memory.** Every search starts from zero. The same trash gets surfaced again. The same authority goes unrecognized.\n\nqsearch addresses all three:\n- **Full content fetched and cleaned**, not just snippets.\n- **`engines[]` field per result** — Google + DDG + Brave + Qwant + Startpage attribution exposed (via SearXNG aggregation).\n- **Local corpus accumulates** — every URL grows a trust profile across sweeps.\n\n## How it works\n\n```mermaid\nflowchart LR\n    A[Your agent] -->|query| Q[qsearch]\n    Q -->|fan out| B[Brave Search API]\n    Q -->|fan out| S[\"SearXNG\\n(Google, DDG, Brave, Qwant, …)\"]\n    B -->|results| Q\n    S -->|results + engines[]| Q\n    Q -->|index by URL| C[\"Local corpus\\n(Meilisearch + Qdrant)\"]\n    C -->|trust score| Q\n    Q -->|re-ranked + full content + provenance| A\n\n    style C fill:#fde68a,stroke:#d97706,color:#000\n    style Q fill:#93c5fd,stroke:#2563eb,color:#000\n    style S fill:#86efac,stroke:#16a34a,color:#000\n```\n\nThe yellow node is your private corpus. URLs found by 5 engines + 3 sweeps + 4 topics get a trust score that emerges naturally — no human ranking, no centralized authority, no cloud round-trip.\n\n## How qsearch compares\n\n|  | Tavily | Exa | Serper | Brave API | SearXNG | **qsearch** |\n|---|---|---|---|---|---|---|\n| Open source core | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ |\n| Full content (not snippets) | partial | partial | ❌ | ❌ | ❌ | ✅ |\n| Multi-engine attribution | ❌ | ❌ | ❌ | ❌ | partial | ✅ (`engines[]`) |\n| Persistent local corpus | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |\n| Trust score per URL | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |\n| Self-hostable | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ |\n| MCP-native | partial | ✅ | ❌ | ✅ | ❌ | ✅ |\n| BYOK upstream | ❌ | ❌ | ❌ | N/A | ✅ | ✅ |\n\n## API — v0.4.0\n\n### Search endpoints\n\n| Endpoint | Description | Backend |\n|----------|-------------|---------|\n| `POST /search` | Web search + corpus first, trust-weighted re-rank | Brave or SearXNG |\n| `POST /sweep` | Batch search with priority/domain routing (see below) | SearXNG / Brave / Academic |\n| `POST /cached_sweep` | Same as `/sweep`, with SQLite memcache layer | SearXNG / Brave / Academic |\n| `POST /academic_search` | Peer-reviewed papers via arxiv + PubMed + Semantic Scholar | Academic (free, no auth) |\n| `POST /sweep_context` | Local LLM page extraction (analogue of Brave LLM Context) | Ollama qwen2.5 |\n| `POST /news` | News search | Brave (requires key) |\n| `POST /context` | Deep page extraction | Brave (requires key) |\n| `POST /verify` | Citation honesty check — does the cited URL actually support a claim? Returns `Supported`/`Partial`/`Unsupported`/`Fabricated`/`Error` + verbatim excerpt | LLM-as-judge (local Ollama qwen2.5 or DeepSeek) |\n| `POST /index` | Crawl URL or index local `.md` glob | Crawl4AI |\n| `GET /trust/:url` | Trust score + provenance for any URL in corpus | — |\n| `GET /corpus/top` | Top URLs ranked by trust (`?limit=20&min_engines=3`) | — |\n| `GET /corpus/stats` | Corpus size + counts | — |\n| `GET /economy_report` | Sprint cost breakdown by backend + savings vs all-Brave | — |\n| `GET /ui` | Corpus browser — search, trust scores, provenance modal | — |\n| `GET /health` | Service status | — |\n\n`/search` accepts: `query`, `n_results` (1–20), `freshness` (`pd`/`pw`/`pm`/`py`), `search_lang`, `country`, `corpus_first` (default `true`), `corpus_only` (default `false`).\n\n`/sweep` accepts text/plain body with one query per line in the format `label|query[|priority][|domain]`:\n\n- **priority** ∈ `broad` (default, SearXNG, $0) / `focused` (Brave, ~$0.005) / `critical` (Brave + LLM Context, ~$0.01)\n- **domain** ∈ `general` (default) / `scholarly` (arxiv+PubMed+S2, $0) / `ru` (SearXNG with `language=ru-RU` bias, $0)\n\n```\n# Examples\nbench_a|qdrant production latency benchmarks|focused\nsch_a|crispr cas9 off target effects|broad|scholarly\nru_a|tadviser сро рейтинг 2025|broad|ru\ncrit_a|self-hosted vector DB choice 2026|critical\ngen|simple search|broad        # 2-field still works — defaults broad/general\n```\n\nAuto-indexes results into Meilisearch with `engines[]` and `engine_count` filterable.\n\n`/academic_search` accepts JSON: `{ query, n_results (1-20), sources?: [\"arxiv\",\"pubmed\",\"semanticscholar\"] }`. Fans out to all three in parallel, dedupes by DOI/title, returns interleaved top-N.\n\n### Multi-engine attribution example\n\n```bash\ncurl -X POST http://localhost:8080/sweep \\\n  -H \"Content-Type: text/plain\" \\\n  --data-binary $'t1|self-hosted search engine 2026\\n'\n```\n\nOutput excerpt (`parsed_snippets.md`):\n\n```markdown\n**1. GitHub - searxng/searxng**\n- URL: https://github.com/searxng/searxng\n- Engines: google, duckduckgo, brave, qwant (count=4)\n  > A privacy-respecting, hackable metasearch engine...\n\n**2. random-blog.io/seo-spam-2026**\n- URL: https://random-blog.io/seo-spam-2026\n- Engines: google (count=1)\n  > Best self-hosted search engines you must try...\n```\n\nURL #1 has `engine_count=4` — found by 4 independent engines. URL #2 has `engine_count=1` — found by only one. The trust signal is built into the data, not bolted on.\n\n### Filter by trust in Meilisearch\n\n```bash\ncurl -H \"Authorization: Bearer masterKey\" \\\n  \"http://localhost:7700/indexes/qsearch_corpus/documents?filter=engine_count%20%3E%3D%203\"\n```\n\nReturns only URLs found by 3+ engines — your high-trust subset.\n\n## MCP integration\n\nThe MCP server lives in this repo — there is no npm package for it. Run it from source:\n\n```bash\ngit clone https://github.com/theYahia/qsearch.git && cd qsearch\nnpm install\nnpm start            # REST API on :8080 — must be running first\nnpm run start:mcp    # MCP server (Streamable HTTP) on :8081\n```\n\n### Claude Code\n\nAdd to `~/.claude/settings.json`:\n\n```json\n{\n  \"mcpServers\": {\n    \"qsearch\": {\n      \"type\": \"http\",\n      \"url\": \"http://localhost:8081\"\n    }\n  }\n}\n```\n\nAvailable tools:\n- `mcp__qsearch__web_search` — web search via Brave or SearXNG\n- `mcp__qsearch__sweep` — batch research sweep with multi-engine attribution\n- `mcp__qsearch__academic_search` — peer-reviewed papers via arxiv + PubMed + Semantic Scholar\n- `mcp__qsearch__sweep_context` — Phase 3 local LLM page extraction (free, Ollama)\n- `mcp__qsearch__verify_citation` — does the cited URL actually support the claim? (`Supported`/`Partial`/`Unsupported`/`Fabricated`)\n- `mcp__qsearch__economy_report` — cost breakdown vs all-Brave baseline\n- `mcp__qsearch__index_research` — index local `.md` files by glob\n- `mcp__qsearch__news_search` — news search (Brave key required)\n- `mcp__qsearch__context_search` — deep page content (Brave key required)\n\n### Other MCP-over-HTTP clients\n\nqsearch publishes Streamable HTTP transport at `/` on port `:8081`. Compatible with Claude Desktop (HTTP mode), OpenClaw, and any spec-compliant MCP client.\n\n## Stack\n\n| Component | Tech |\n|-----------|------|\n| Runtime | Node.js ≥20 |\n| Web search | Brave Search API (BYOK) |\n| Meta-search | SearXNG (self-hosted, optional) |\n| Academic | arxiv + PubMed E-utilities + Semantic Scholar API (free, no auth) |\n| Full-text corpus | Meilisearch v1.7 |\n| Vector corpus | Qdrant v1.17.1 |\n| Crawler | Crawl4AI 0.8.6 (Python subprocess) |\n| Embedder (optional) | Ollama `nomic-embed-text` (default) or llama.cpp `/v1/embeddings` |\n| LLM cleaner (optional) | Ollama `qwen2.5:7b-instruct` (default; configurable via `OLLAMA_CLEAN_MODEL`) |\n| MCP | `@modelcontextprotocol/sdk` |\n| License | Apache-2.0 |\n\n## Roadmap\n\n| Version | Feature | When |\n|---------|---------|------|\n| **v0.3.1** | Multi-engine `engines[]` attribution + dual sweep + corpus + MCP | shipped |\n| **v0.4.0** | Trust layer: `/trust/:url`, `/corpus/top`, `/ui` viewer, trust-weighted re-rank, sort/pagination, corpus merge-on-upsert, snippet sanitization | shipped |\n| **v0.4.1** | Phase A — academic backend (arxiv + PubMed + S2), 4-field queries (`label\\|q\\|priority\\|domain`), `/academic_search` JSON + MCP tool | shipped |\n| **v0.4.2** | Phase B — embedding rerank (Ollama nomic-embed-text, gated `QSEARCH_RERANK_ENABLED`); Phase C — RU coverage via SearXNG `language=ru-RU` | shipped |\n| **v0.4.3** | QVAC SDK ripped out, all local LLM via Ollama (`qwen2.5:7b-instruct` + `nomic-embed-text`) | shipped |\n| **v0.5** | Launch: awesome list PRs, MCP Registry publish, Show HN, newsletter distribution | in progress |\n| **v0.6** | Phase B Stage 2 — LLM scoring rerank for critical queries; direct Yandex backend; Layer 8 quality gate (rejection threshold) | next |\n| **v0.7+** | Optional federation (research direction — no timeline until v0.5 validated) | open |\n\nSee [docs/VISION.md](./docs/VISION.md) for the full picture and why federation is research-direction-only until we can ship it without overpromise.\n\n## Honest trade-offs\n\n- **Cold start.** First sweep takes 5–10 seconds (engine fan-out + corpus indexing). Best run as long-lived daemon.\n- **Vector search Windows-blocked.** Qdrant requires bare-runtime; not all platforms supported. Full-text Meilisearch works everywhere.\n- **SearXNG rate limits.** Self-host required — public instances get blocked by Google. Our docker-compose handles this.\n- **`engines[]` requires SearXNG.** Pure-Brave mode still works but loses the multi-engine signal.\n- **Full content has latency cost.** ~31s vs ~3s naive snippet retrieval ([Bidirectional RAG study](https://arxiv.org/html/2512.22199v1)). qsearch makes this opt-in via `/context` endpoint.\n\n## Follow\n\n- 🌐 **Live demo:** [qsearch.pro](https://qsearch.pro)\n- ⭐ **Star:** [github.com/theYahia/qsearch](https://github.com/theYahia/qsearch)\n- 🐦 **X:** [@TheTieTieTies](https://x.com/TheTieTieTies)\n\n## License\n\nApache-2.0 — see [LICENSE](./LICENSE). Independent. BYOK. Self-hostable. No vendor lock-in.\n\n---\n\nЧасть [WWmcp](https://github.com/theYahia/WWmcp) · Telegram: [@vhodvai](https://t.me/vhodvai)\n",
  "bytes": 15416,
  "sha": "63a29598b30113022112209d44234e755bbb5ecbc401d05ae8b981e358a88d70",
  "repo_slug": "theyahia/qsearch",
  "fonte": "repo",
  "truncated": false,
  "api": "https://agentalog.com/api/listings/mcp_io_github_theyahia_qsearch_302c5f43/readme"
}