{
  "markdown": "# hiivmind-corpus\n\nA Claude Code plugin for shareable, linkable LLM knowledge bases. If you have been following Andrej Karpathy's LLM wiki idea or building Obsidian-based PKM setups, hiivmind-corpus is the productised version of that pattern — raw sources compiled into a curated, interlinked index — but any corpus can be published as a plain git repo and registered by anyone else with a single command, queryable remotely via `gh api` with sparse-cloned embeddings and cross-corpus bridges that link concepts across independently maintained knowledge bases. Every other tool in this space (ObsidianRAG, Neural Composer, obsidian-notes-rag, Karpathy's LLM wiki) is single-user, single-vault, local-only; this one enables your knowledge base to become a public library, not just a personal notebook.\n\n**Quick links:** [Using a Corpus](#using-a-corpus) | [Building a Corpus](#building-a-corpus) | [Semantic Search](#semantic-search-rag) | [Published Corpora](#published-corpora)\n\n## The Idea\n\nWithout structured indexing, Claude investigates libraries by relying on training data (outdated), web searching (hit-or-miss), or fetching URLs one at a time (no context). Every session rediscovers the same things.\n\nA corpus solves this. You build a curated index once — collaboratively, around your actual use case — and Claude searches it across sessions. The index tracks where everything came from, how fresh it is, and uses semantic search to find relevant entries even when queries don't match exact keywords.\n\nThis follows the [\"just in time\" context pattern](https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents) from Anthropic: maintain lightweight identifiers, dynamically load content at runtime.\n\n## Getting Started\n\n```\n/hiivmind-corpus\n```\n\nOne command, natural language:\n\n| You say... | What happens |\n|---|---|\n| *\"Create a corpus for Polars\"* | Scaffolds a new corpus, clones docs |\n| *\"What corpora do I have?\"* | Discovers all installed corpora |\n| *\"How do lazy frames work?\"* | Searches your corpora with semantic + keyword matching |\n| *\"Refresh my React corpus\"* | Checks for upstream changes, updates stale entries |\n| *\"Add the TanStack Query docs\"* | Extends an existing corpus with new sources |\n\n## Installation\n\n**From the command line:**\n\n```bash\nclaude plugin marketplace add hiivmind/hiivmind-corpus\nclaude plugin add hiivmind-corpus@hiivmind\n```\n\n**From within a Claude Code session:**\n\n```\n/plugin marketplace add hiivmind/hiivmind-corpus\n/plugin install hiivmind-corpus@hiivmind\n```\n\nOr use `/plugin` to browse and install interactively.\n\n## Using a Corpus\n\nMost users start here — someone else built the corpus, you just want to query it.\n\n### 1. Register a corpus with your project\n\n```\n/hiivmind-corpus register github:hiivmind/hiivmind-corpus-data/hiivmind-corpus-polars\n```\n\nThis adds the corpus to your project's registry (`.hiivmind/corpus/registry.yaml`). Register as many as you need — they're lightweight references, not copies.\n\n### 2. Ask questions\n\nJust ask naturally. Claude routes your question to the right corpus:\n\n```\n\"How do I filter rows in Polars?\"\n\"What's the difference between select and with_columns?\"\n\"Show me lazy frame optimization techniques\"\n```\n\nOr be explicit: `/hiivmind-corpus navigate polars \"group by aggregations\"`\n\n### How navigation works\n\nWhen you ask a question, the navigate skill:\n\n1. **Routes to the right corpus** — matches your query against registered corpora using semantic similarity (if embeddings are cached) or keyword matching\n2. **Finds relevant entries** — searches the corpus index using vector search with optional SQL filtering, boosted by concept graph relationships\n3. **Fetches documentation** — retrieves the actual content from the source repo via `gh api`\n4. **Presents the answer** — with source citations and related doc suggestions\n\nFor remote GitHub corpora, embeddings are automatically cached locally on first query (~5 seconds, then instant on subsequent queries).\n\n### Check what you have\n\n```\n/hiivmind-corpus discover        # List all available corpora\n/hiivmind-corpus status           # Check freshness and health\n```\n\n### Cross-corpus queries\n\nWith 2+ corpora registered, you can create **bridges** — links between related concepts across corpora:\n\n```\n/hiivmind-corpus bridge           # Detect and create cross-corpus links\n```\n\nNavigate then uses bridges and aliases to route queries that span multiple documentation sets. Search for \"lazy evaluation\" and it finds relevant entries in both Polars and Ibis.\n\n---\n\n## Building a Corpus\n\nIf you want to create a new corpus from scratch — for a library, framework, or internal project.\n\n### Quick start\n\n```\n/hiivmind-corpus init             # Scaffold from a GitHub repo\n/hiivmind-corpus add-source       # Add git repos, local files, web pages, PDFs\n/hiivmind-corpus build            # Scan sources, build index collaboratively\n```\n\nThe build process is a conversation — Claude scans the docs and you guide what matters: \"I care about data modeling and ETL, skip the deployment stuff.\" That curation persists across sessions.\n\n### After building\n\n```\n/hiivmind-corpus enhance          # Deepen coverage on specific topics\n/hiivmind-corpus refresh          # Sync with upstream changes\n/hiivmind-corpus graph add-concept  # Add zettelkasten concept clusters\n```\n\n---\n\n## What a Corpus Looks Like\n\nEach corpus is a data-only repository — just files, no server, no database:\n\n```\nhiivmind-corpus-{project}/\n├── config.yaml                # Source definitions + keywords\n├── index.yaml                 # Structured index (entries with summaries, tags, concepts)\n├── index.md                   # Human-readable index (rendered from index.yaml)\n├── graph.yaml                 # Concept graph — zettelkasten relationships (optional)\n├── index-embeddings.lance/    # Semantic search embeddings (committed, not gitignored)\n├── render-index.sh            # Deterministic index.yaml → index.md renderer\n├── .source/                   # Git clones (gitignored)\n├── .cache/                    # Web/llms-txt cached content (gitignored)\n└── uploads/                   # Local document sources\n```\n\nThe index is the product. Everything else supports building and maintaining it.\n\n## Three Layers\n\nThe corpus architecture is a value-add pipeline — each layer builds on the previous:\n\n```\nLayer 1: INDEX (foundation)\n  source files → index.yaml (entries with title, summary, tags, keywords, concepts)\n  → index.md (rendered for humans)\n\nLayer 2a: GRAPH (zettelkasten)          Layer 2b: RAG (semantic search)\n  Concept definitions + relationships    Vector embeddings of entry metadata\n  References entries by concept ID       Enriched by concept membership\n  Pure relationship store                Entries embed: title|summary|tags|concepts\n\n         ↕ mutual enrichment ↕\n  Graph candidate detection uses RAG similarity\n  RAG graph-boost uses graph relationships\n  Bridge detection queries per-corpus RAG\n```\n\n**Layer 1 (index.yaml)** is always built. Layers 2a and 2b are optional — they add value for larger corpora and cross-corpus scenarios.\n\n## Skills\n\n### Build & Maintain\n\n| Skill | Purpose |\n|---|---|\n| **init** | Scaffold a new corpus from a GitHub repo URL or local project |\n| **add-source** | Add git repos, local files, web pages, PDFs, Obsidian vaults, llms.txt |\n| **build** | Scan sources, build index.yaml collaboratively with user, generate embeddings |\n| **enhance** | Deepen coverage on specific topics within an existing index |\n| **refresh** | Compare against upstream commits, flag stale entries, re-embed |\n\n### Query & Discover\n\n| Skill | Purpose |\n|---|---|\n| **navigate** | Search across corpora — semantic pre-filter, graph-boost, reranking |\n| **discover** | Find all installed/registered corpora and their status |\n| **register** | Connect a corpus to the current project via registry.yaml |\n| **status** | Check corpus health — freshness, embedding status, upstream changes |\n\n### Concepts & Relationships\n\n| Skill | Purpose |\n|---|---|\n| **graph** | View, validate, and edit concept graphs (graph.yaml) |\n| **bridge** | Create cross-corpus concept bridges and query-routing aliases |\n\n### Lifecycle\n\n```\ninit → add-source → build → refresh/enhance (as needed)\n                       ↓\n              graph (concepts) ←→ embeddings (RAG)\n                       ↓\n         register → navigate (query) → bridge (cross-corpus)\n```\n\n## Source Types\n\n| Type | Storage | Example |\n|---|---|---|\n| **git** | `.source/{source_id}/` | Library docs, framework APIs |\n| **local** | `uploads/{source_id}/` | Team standards, internal docs |\n| **web** | `.cache/web/{source_id}/` | Blog posts, articles |\n| **llms-txt** | `.cache/llms-txt/{source_id}/` | Sites with llms.txt manifests |\n| **generated-docs** | `.source/{source_id}/` + web | Hybrid git+web (e.g., docs built from source) |\n| **pdf** | `uploads/{source_id}/` | PDF books, split into chapters |\n| **obsidian** | `.source/{source_id}/` | Obsidian vaults with wikilinks and tags |\n| **self** | (current repo) | Embedded corpus — index the repo's own docs |\n\n## Semantic Search (RAG)\n\nCorpora can include **entry-level semantic embeddings** for retrieval that goes beyond keyword matching.\n\n**How it works:**\n- During `build`, entries are embedded using [fastembed](https://github.com/qdrant/fastembed) with `BAAI/bge-small-en-v1.5` (ONNX, no PyTorch, ~120MB)\n- Embeddings are stored in [Lance format](https://lancedb.com/) — flat files, no server, committed to the repo\n- At query time, `navigate` searches by cosine similarity with optional SQL predicates for hybrid search (vector + keyword)\n- An FTS index on metadata enables full-text keyword matching alongside semantic search\n- Optional reranking with CrossEncoder for better precision on ambiguous queries\n\n**What gets embedded:**\n```\n\"passage: {title} | {summary} | {tags} | {concepts}\"\n```\n\nConcept labels in the embedding text mean searching for \"lazy evaluation\" finds entries assigned to that concept even if those words don't appear in the title or summary. The zettelkasten structure directly improves RAG recall.\n\n**Remote corpora:** For GitHub-hosted corpora, navigate automatically sparse-clones the Lance directory to a local cache (`.hiivmind/corpus/cache/`) on first query, with TTL-based freshness tracking.\n\n**Opt-in:** Embeddings are suggested during build when `entry_count > 150` or the corpus has tiered indexes. Below that threshold, the LLM scanning the full index directly is effective enough.\n\n**Dependencies:** `pip install fastembed lancedb pyyaml` (~260MB). If not installed, navigate falls back to keyword/yq pre-filtering — embeddings are an enhancement, not a requirement.\n\n## Concept Graphs\n\nEach corpus can have a **concept graph** (`graph.yaml`) — a zettelkasten-style knowledge structure:\n\n```yaml\nschema_version: 2\nconcepts:\n  lazy-evaluation:\n    label: \"Lazy Evaluation\"\n    description: \"Deferred query execution for optimization\"\n    tags: [performance, lazy]\n  query-optimization:\n    label: \"Query Optimization\"\n    description: \"Techniques for faster query execution\"\n    tags: [performance, indexing]\nrelationships:\n  - from: lazy-evaluation\n    to: query-optimization\n    type: depends-on\n    origin: manual\n```\n\nConcept membership is bidirectional: entries in `index.yaml` declare their concepts via a `concepts[]` field, and graph.yaml defines concept definitions and relationships. The graph skill lets you add concepts, add relationships (with embedding-powered candidate detection), and validate the graph.\n\n## Cross-Corpus Bridges\n\nProjects with 2+ registered corpora can create **bridges** — links between concepts in different corpora:\n\n```yaml\n# .hiivmind/corpus/registry-graph.yaml\nbridges:\n  - concept_a: \"polars:lazy-evaluation\"\n    concept_b: \"ibis:deferred-execution\"\n    type: see-also\n    note: \"Both implement deferred query execution\"\naliases:\n  \"lazy evaluation\":\n    - corpus: polars\n      concept: lazy-evaluation\n    - corpus: ibis\n      concept: deferred-execution\n```\n\nBridge candidate detection queries each corpus's embeddings to find semantically similar concepts across corpora — even when they use different terminology.\n\n## Per-Project Registry\n\nProjects register which corpora they use:\n\n```yaml\n# .hiivmind/corpus/registry.yaml\ncorpora:\n  - id: polars\n    source:\n      type: github\n      repo: hiivmind/hiivmind-corpus-data\n      path: hiivmind-corpus-polars\n      ref: main\n  - id: flyio\n    source:\n      type: github\n      repo: hiivmind/hiivmind-corpus-flyio\n      ref: main\n```\n\nNavigate uses the registry to search across all registered corpora and route queries to the right one.\n\n## Dependencies\n\n| Tool | Required | Purpose |\n|---|---|---|\n| **git** | Yes | Clone source repos, track commits |\n| **gh** (GitHub CLI) | Recommended | Fetch content from GitHub repos (preferred over raw URLs) |\n| **yq** 4.0+ | Recommended | Parse YAML config files (grep fallback available) |\n| **fastembed + lancedb** | Optional | Semantic search embeddings (~260MB pip install) |\n| **pyyaml** | Optional | YAML parsing in embedding scripts |\n\n## Design Principles\n\n- **Human-curated indexes** — You decide what matters, not an algorithm\n- **Collaborative building** — The build process is a conversation, not a batch job\n- **Layered value-add** — Index first, then optionally concepts and embeddings\n- **Graceful degradation** — Works without embeddings, without graphs, without yq\n- **Portable** — Corpora are just files. Commit, diff, review, share with your team\n- **Known freshness** — Commit SHA tracking tells you exactly how old your sources are\n- **Works without local clone** — Falls back to `gh api` for remote content fetching\n\n## Published Corpora\n\n*Already use Obsidian? Register the Obsidian help corpus to get started: `/hiivmind-corpus register github:hiivmind/hiivmind-corpus-obsidian`*\n\n| Corpus | Source |\n|---|---|\n| [hiivmind-corpus-obsidian](https://github.com/hiivmind/hiivmind-corpus-obsidian) | Obsidian help docs |\n| [hiivmind-corpus-polars](https://github.com/hiivmind/hiivmind-corpus-data) | Polars documentation |\n| [hiivmind-corpus-ibis](https://github.com/hiivmind/hiivmind-corpus-data) | Ibis documentation |\n| [hiivmind-corpus-narwhals](https://github.com/hiivmind/hiivmind-corpus-data) | Narwhals documentation |\n| [hiivmind-corpus-substrait](https://github.com/hiivmind/hiivmind-corpus-data) | Substrait specification |\n| [hiivmind-corpus-flyio](https://github.com/hiivmind/hiivmind-corpus-flyio) | Fly.io platform docs |\n| [hiivmind-corpus-lancedb](https://github.com/hiivmind/hiivmind-corpus-lancedb) | LanceDB documentation |\n| [hiivmind-corpus-claude-agent-sdk](https://github.com/hiivmind/hiivmind-corpus-claude) | Claude Agent SDK |\n\nRegister any of these with:\n```\n/hiivmind-corpus register github:hiivmind/hiivmind-corpus-flyio\n```\n\n## License\n\nMIT\n",
  "bytes": 14868,
  "sha": "f111defaf8ef9b6e514a748b6edde46fa1b2f04f52822e86a047e602147d82f0",
  "repo_slug": "hiivmind/hiivmind-corpus",
  "fonte": "repo",
  "truncated": false,
  "api": "https://agentalog.com/api/listings/plg_hiivmind_hiivmind_corpus_9a3770c0/readme"
}