{
  "markdown": "# Enterprise Internal Knowledge Base — Production-Ready RAG + MCP\n\nA public Retrieval-Augmented Generation pipeline exposed as an MCP server. Sample content from Veterans Affairs education manuals.\n\nThe repo implements evaluation, observability, and structure-aware ingestion. Cost/latency tuning, tenant-level access control, and other production concerns are discussed in the article linked below.\n\n**📖 Full writeup in *Towards AI*:** [Enterprise Internal Knowledge Base RAG MCP: POC-to-Production](https://medium.com/towards-artificial-intelligence/poc-to-production-rag-af49476f4ddc)\n\n---\n\n## Why this exists\n\nRAG demos tend to focus on the quality of the retrieval pipeline, without recognizing that production RAG fails on the next ten steps: prompt or model changes that pass code review but tank answer quality, cost and latency drift that cannot be traced to specific queries, cross-tenant leakage that only surfaces in audit. This repo shows what catching them looks like in practice.\n\nThe corpus is public (VA Education manuals — 238 documents, 9,000+ chunks) so anyone can clone, run, and adapt the pipeline.\n\n---\n\n## Quickstart\n\n```bash\ngit clone https://github.com/kimsb2429/internal-knowledge-base\ncd internal-knowledge-base\n\n# 1. Start Postgres + pgvector\ndocker compose up -d\n\n# 2. Python env + dependencies\npython3 -m venv .venv && source .venv/bin/activate\npip install -r requirements.txt\n\n# 3. Restore corpus fixture (~2 min — 238 docs + 9k chunks pre-embedded)\ndocker exec -i ikb_pgvector pg_restore -U ikb -d ikb < evals/fixture_v1.dump\n\n# 4. Smoke-test the MCP server\npython scripts/test_mcp_server.py     # 7/7 tests pass\n\n# 5. Start the MCP server (stdio transport)\npython scripts/mcp_server.py\n```\n\n### Consuming from Claude Desktop\n\nAdd to `~/Library/Application Support/Claude/claude_desktop_config.json`:\n\n```json\n{\n  \"mcpServers\": {\n    \"ikb\": {\n      \"command\": \"python\",\n      \"args\": [\"/absolute/path/to/internal-knowledge-base/scripts/mcp_server.py\"]\n    }\n  }\n}\n```\n\nThen ask Claude things like *\"What RPO handles GI Bill claims in Texas?\"* — the MCP server returns ranked chunks with citations.\n\n---\n\n## Architecture\n\n**Ingestion (one-time per corpus):**\n\n```mermaid\ngraph LR\n    A[KnowVA crawler<br/>HTML + PDF] --> B[Source-specific<br/>preprocessor]\n    B --> C[Structure-aware<br/>chunker]\n    C --> D[mxbai-embed-large<br/>local, 1024-dim]\n    D --> E[(pgvector)]\n    F[Anthropic Contextual<br/>Retrieval] -.-> E\n    E -.-> F\n    style E fill:#e1f5fe\n```\n\n**Query (per MCP tool call):**\n\n```mermaid\ngraph LR\n    A[Claude Desktop<br/>MCP client] --> B[FastMCP server]\n    B --> C[pgvector top-K]\n    C --> D[Reranker<br/>mxbai or FlashRank]\n    D --> E[Claude Sonnet<br/>generation]\n    E --> A\n    E --> F[Langfuse trace]\n    style F fill:#fff9c4\n```\n\n**Stack:**\n- **Vector store:** Postgres + pgvector (Docker, port 5433); `content_tsv` GIN index for hybrid-ready\n- **Embeddings:** mxbai-embed-large (1024 dims, local via sentence-transformers) — $0 API cost\n- **Reranker:** mxbai-rerank-base-v2 (full eval) / FlashRank MiniLM (CI fast mode, 22M ONNX, ~2s/query)\n- **Generation:** Claude Sonnet\n- **MCP server:** FastMCP 3.2.4 — Tools (`query`), Resources (`document://{source_id}`), Prompts (`cite_from_chunks`)\n- **Observability:** Langfuse Cloud, per-trace public sharing\n- **Eval:** DeepEval + 110-query golden set + GitHub Actions merge gate\n\n---\n\n## Eval scores\n\nFull 110-question golden set, contextualized chunks + reranker:\n\n| Metric | Score |\n|---|---|\n| Faithfulness | 0.95 |\n| Answer Relevance | 0.91 |\n| Context Precision | 0.61 |\n| Context Recall | 0.52 |\n| Context Relevance | 0.56 |\n\n🔗 **[Live Langfuse trace](https://us.cloud.langfuse.com/project/cmo0wah7a00pfad071nk6x84c/traces/a574193bbff7d5438f7fae9e27f4bb83)** (public, no login).\n\n**Notable result:** Anthropic's Contextual Retrieval pattern produced modest lift on top of reranking (+4.8pp AnsRel, +4.1pp CtxPrec) at this scale — well short of the +35% recall their published numbers suggested. Reported as found; juiced numbers would defeat the point.\n\n---\n\n## Eval-in-CI as a merge gate\n\nEvery PR runs the golden set in fast mode (FlashRank reranker, ~3-4 min wall, $0.30 in Sonnet calls) against a fixture DB. PRs that regress more than ±5pp on top1/topk/keyword_recall, or +10pp on `idk_rate`, are blocked.\n\n**Forever-artifact:** [PR #5](https://github.com/kimsb2429/internal-knowledge-base/pull/5) — a deliberate failing-then-passing PR. Red CI catches a 20pp top1 regression; green CI confirms the fix. The Actions tab is the proof.\n\nWorkflow: [`.github/workflows/eval-gate.yml`](.github/workflows/eval-gate.yml).\n\n---\n\n## What this repo doesn't cover\n\nA few production-shape items are seams, not implementations:\n\n- **Multi-tenant scoping** — `auth_context` parameter present on every MCP tool, typed, currently unused (labels the SSO/ACL seam)\n- **Ingestion concurrency** — single-threaded chunker + embedder; production would use a modulus-distributed worker pool\n- **Hybrid search wiring** — `content_tsv` GIN index is live; BM25 + RRF fusion at query time stays a post-launch addition\n\nThe writeup linked above covers these topics.\n\n---\n\n## Repo layout\n\n```\ndocs/                    Research, evidence base, deep-dives\ndata/                    Crawled corpus + golden query set\nscripts/\n  crawl_knowva.py            eGain v11 API crawler\n  enrich_metadata.py         Headings, ACL, authority tier, content_category\n  knowva_preprocess.py       Source-specific HTML normalization\n  chunk_documents.py         Structure-aware splitter (preserves table colspan/rowspan)\n  embed_and_store.py         mxbai-embed-large → pgvector\n  contextualize_chunks.py    Anthropic Batches API for Contextual Retrieval\n  rerank.py                  mxbai-rerank + FlashRank\n  retrieve.py / generate.py  RAG path\n  mcp_server.py              FastMCP exposure\n  run_eval.py / score_eval.py / check_regression.py   Eval harness + CI gate\nevals/                   Fixture DB dump + baseline JSON\n.github/workflows/       eval-gate.yml — merge-gate workflow\n```\n\n---\n\n## Reproducing from raw corpus (~30 min)\n\nEach script is idempotent and resume-safe.\n\n```bash\npython scripts/crawl_knowva.py            # Crawl raw HTML (skip if data/knowva_manuals/articles/ exists)\npython scripts/enrich_metadata.py         # Add headings, ACL, authority tier\npython scripts/knowva_preprocess.py       # Normalize HTML quirks\npython scripts/chunk_documents.py         # Structure-aware split\npython scripts/embed_and_store.py         # mxbai → pgvector\npython scripts/contextualize_chunks.py    # Anthropic Batches API (~$12, optional but recommended)\n```\n\nThen `python scripts/run_eval.py --fast` to verify the eval baseline reproduces.\n\n---\n\n## Further reading\n\n- **Full demo writeup**: [Enterprise Internal Knowledge Base RAG MCP: POC-to-Production](https://medium.com/towards-artificial-intelligence/poc-to-production-rag-af49476f4ddc) (*Towards AI*, Medium)\n- [`docs/2026-04-11-engineering-rag-evidence-and-howtos.md`](docs/2026-04-11-engineering-rag-evidence-and-howtos.md) — engineering analysis, evidence base, Zero-to-MCP plan\n- [`docs/2026-04-12-rag-pipeline-buy-vs-build.md`](docs/2026-04-12-rag-pipeline-buy-vs-build.md) — buy-vs-build map per pipeline stage\n- [`docs/deep-dive/2026-04-16-docs-vs-code-rag-adjudication.md`](docs/deep-dive/2026-04-16-docs-vs-code-rag-adjudication.md) — when unified RAG stops working\n\n---\n\n## License\n\nMIT — see [LICENSE](LICENSE).\n",
  "bytes": 7468,
  "sha": "4c5bb1af0603342a2c689d34808dbe1395b338b8c6d02a218a98bce0ea8a508d",
  "repo_slug": "kimsb2429/internal-knowledge-base",
  "fonte": "repo",
  "truncated": false,
  "api": "https://agentalog.com/api/listings/mcp_io_github_kimsb2429_internal_knowledge_b_ff7e3d6a/readme"
}