{
"markdown": "# foodini-okf-ingest\n\n[](https://github.com/Foodini-NG/foodini-okf-ingest/actions/workflows/ci.yml)\n[](#conformance-tests)\n[](#deterministic-by-design--no-agents)\n[](#install)\n[](LICENSE)\n\n> ### This is a fork\n>\n> **This file was modified by Foodini on 2026-08-22.** This repository is a\n> Foodini fork of [**okf-ingest**](https://github.com/travisjakel/okf-ingest) by\n> **Travis Jakel**, taken at commit `f3b58994` (release 0.11.0) and licensed\n> under Apache-2.0. See [`NOTICE`](NOTICE) for attribution and the full list of\n> changes.\n>\n> **What differs from the original.** The original ships five bindings — R,\n> Python, Rust, C++ and MATLAB — held byte-identical by a shared conformance\n> corpus. This fork keeps **Python only**, because Foodini does not write the\n> other four and holding them in lockstep would cost several implementations\n> per change. The catalog schema and the conformance corpus are retained\n> unchanged. The distribution is `foodini-okf-ingest` and the command is\n> `okf-ingest` (the original's bare `okf` collides with okf-generator).\n>\n> **This fork is independent.** Travis Jakel has not endorsed it and is not\n> responsible for it, so report problems *with this fork* here rather than to him.\n>\n> **If you want the tool rather than our build of it, go\n> [upstream](https://github.com/travisjakel/okf-ingest).** Five bindings instead\n> of one, actively developed, properly packaged — for most people it is simply the\n> better choice, and a fix contributed there helps far more users than the same\n> fix here. We forked to avoid maintaining four bindings we do not write, not\n> because we thought we could do it better.\n\nA unified, open-source **ingestion tool for [Open Knowledge Format](https://github.com/GoogleCloudPlatform/knowledge-catalog) (OKF) bundles** — read any OKF bundle, validate its conformance (permissively, per the spec), build the concept graph, and load it into a portable, queryable **DuckDB catalog**. One catalog format, one idiomatic **Python** package.\n\n> **Point it at your `[[wikilink]]` vault.** As of 0.6, okf-ingest resolves both\n> markdown `](path)` links *and* `[[wikilink]]` references (Obsidian / Logseq /\n> Foam) — by **name** (id / alias / title), so links **survive file renames**.\n> Your existing notes become a deterministic, queryable, renderable knowledge\n> graph with no rewriting. See [`[[wikilinks]]` & aliases](#wikilinks--aliases).\n\n\n\n> The image is `okf-ingest graph` run on [okf-ingest's own docs](docs/okf-bundle/) — the\n> project dogfoods OKF: that folder is a conformant bundle you can `ingest`,\n> `html`/`graph`, and `doctor` with the tool itself.\n\nOKF (Google Cloud; spec v0.2, v0.1 bundles fully supported) is a directory of markdown files with YAML frontmatter — one concept per file, markdown links as a graph. Validators and parsers already exist (Node, a web tool, a pure-Rust crate). **What no other tool does — and what this one is for — is load a bundle into a SQL-queryable DuckDB catalog with built-in semantic search (RAG) from Python**. See [Related tools](#related-tools).\n\n## Deterministic by design — no agents\n\nokf-ingest is **pure, deterministic machinery**: the same bundle in always\nproduces the same catalog, the same graph, the same clusters, and the same\nrendered HTML out — byte-for-byte, on any machine, with no network and no API\nkey. **There are no LLM agents anywhere in it.** It never asks a model to\nsummarize a page, infer a \"layer,\" guess a relationship, or decide anything. It\nreads exactly the structure the author wrote — the frontmatter, the markdown\nlinks — and surfaces *that*.\n\nThis is a deliberate line. A wave of tools will read your knowledge base by\nturning agents loose to summarize and \"understand\" it; their output is\nnon-reproducible, costs tokens, ships your corpus to a model, and quietly\ninvents structure. okf-ingest does the opposite — it's the boring, auditable\nsubstrate underneath:\n\n- **Reproducible** — deterministic enough to assert on in CI; a parity test locks\n R and Python to byte-identical catalogs. No \"re-ran it and got different\n edges.\"\n- **Free & offline** — no tokens, no keys, no calls. Parsing, validation, the\n link graph, community clustering (deterministic label propagation), backlinks,\n impact, and HTML/graph rendering are all plain code.\n- **Private** — your content never leaves the machine. Nothing is sent anywhere.\n- **Composable with agents, not replaced by them** — when you *do* want an LLM,\n okf hands it the curated graph to reason over (`okf-ingest context`) rather than\n pretending to be the reasoner. You bring the model; okf brings the ground truth.\n\n**The two honest exceptions**, both opt-in and explicit: the `embed`/`rag` layer\ncalls a *local, pluggable* embedding model (default Ollama — swap in your own) to\nadd vector search; and `ingested_at` is a wall-clock metadata field you can\noverride (the conformance suite does, which is how it stays byte-stable). The\nknowledge representation itself — concepts, graph, clusters, render — is 100%\ndeterministic and model-free.\n\n| | Wiki / docs site | Vector DB | Agent \"understand my wiki\" | **okf-ingest** |\n|---|---|---|---|---|\n| Reproducible (same in → same out) | ✅ | ⚠️ re-embed drift | ❌ non-deterministic | ✅ byte-locked |\n| Offline, no API key / tokens | ✅ | ⚠️ | ❌ | ✅ |\n| Content stays local / private | ✅ | ⚠️ | ❌ sent to a model | ✅ |\n| SQL / programmatic access | ❌ | ⚠️ vectors only | ⚠️ | ✅ DuckDB |\n| Explicit concept graph | ⚠️ implicit | ❌ | ✅ inferred | ✅ author-written |\n| Your `[[wikilink]]` notes → queryable graph | ⚠️ renders only | ❌ ignores links | ⚠️ non-deterministic | ✅ + rename-safe |\n| Renders to HTML + interactive graph | ⚠️ | ❌ | ✅ | ✅ |\n| Semantic search | ❌ | ✅ | ✅ | ✅ (opt-in) |\n| Invents structure with an LLM | ❌ | ❌ | ✅ | ❌ by design |\n\n### How it fits together\n\n```mermaid\ngraph LR\n B[\"OKF bundle (dir · git · tar/zip)\"] --> I[ingest]\n I --> C[(DuckDB catalog concepts · links · validation · chunks)]\n C --> CX[\"context (LLM-wiki blob)\"]\n C --> H[\"html (site / single)\"]\n C --> G[\"graph / export (interactive · JSON · Mermaid)\"]\n C --> D[\"doctor (health · --fix)\"]\n C --> DF[\"diff (drift · snapshot changelog)\"]\n C --> RK[\"rank (Personalized PageRank)\"]\n C --> R[\"embed / rag (opt-in, local model)\"]\n```\n\n## Do you actually need RAG?\n\nOften you don't — by design. OKF bundles are meant to be read *directly* by an\nagent: load `index.md`, follow the curated links, pull the few relevant concept\nfiles into context. For a small, well-linked bundle (dozens to low-hundreds of\nconcepts), that index-first traversal is the intended pattern — cf. Karpathy's\n\"LLM wiki\"; Google's own framing is that OKF **complements** RAG, it doesn't\nrequire it. No catalog, no embeddings, no okf-ingest — just let the agent\nnavigate the markdown.\n\nReach for okf-ingest when direct reading isn't enough:\n\n- **Programmatic / SQL access** (any size) — query concepts, the link graph, or\n conformance findings from code or CI, in R or Python. → the DuckDB catalog.\n- **Large bundles** — thousands of concepts, where loading the index or the whole\n bundle into context isn't practical.\n- **Semantic / cross-corpus retrieval** — feeding OKF into a wider RAG pipeline,\n or similarity search over a big/heterogeneous knowledge base. → the optional\n `embed`/`rag` layer.\n\nFor small curated bundles, **skip `embed`/`rag`** — the explicit graph the author\nwrote beats fuzzy vector matches, and following links costs nothing. If you want\ntooling *for* that wiki pattern (rather than against it), use\n[`okf-ingest context`](#context--the-index-first-no-embeddings-primitive): it assembles\nthe index-first, link-following slice for an agent to read directly — no\nembeddings involved.\n\n## Quickstart\n\nInstall from a clone (see [Install](#install)), then:\n\n```bash\nuv pip install -e \".[html]\" # from a clone; see Install\n\nokf-ingest ingest ./my-bundle --db kb.duckdb # dir, git URL, or tar/zip\nokf-ingest embed kb.duckdb\nokf-ingest rag kb.duckdb --query \"how is revenue computed?\" -k 5\n# [0.71] metrics/revenue.md#1 — Revenue\n# [0.64] orders.md#1 — Orders\n```\n\nThe catalog is plain DuckDB — query it from Python, or with the bare `duckdb`\nCLI, or anything else that speaks DuckDB.\n\n## Use it from an AI agent\n\nokf is deterministic and **composes with your agent** — give the agent the graph,\nlet it reason. Drop this into your agent's instructions\n(`AGENTS.md` / `CLAUDE.md` / Cursor rules) so it drives okf instead of grepping:\n\n```\nThe docs at are an OKF bundle. Use `okf-ingest` to navigate them:\n- `okf-ingest context --start .md --depth 1` → index.md + that concept +\n its linked neighbours as one markdown blob. Read that; don't grep files ad hoc.\n- `okf-ingest query --search \"\"` (substring) or `--sql \"\"` for lookups.\nStart from index.md to see the map.\n```\n\nThat's it — one paste and the agent reads the bundle index-first, the way OKF is\nmeant to be consumed (`okf-ingest` runs locally; no data leaves the machine). For\nsemantic instead of substring lookup, ingest once (`okf-ingest ingest --db kb.duckdb`),\n`okf-ingest embed kb.duckdb`, then point the agent at `okf-ingest rag kb.duckdb --query \"…\"`. See\n[`context`](#context--the-index-first-no-embeddings-primitive).\n\n## The core is a contract, not compiled code\n\nTwo language-neutral artifacts define behaviour, and this fork keeps both unchanged:\n\n1. **`schema/catalog.sql`** — the DuckDB catalog schema. You can query the catalog with the bare `duckdb` CLI, no library at all, and a catalog written here is readable by any conformant implementation of OKF ingestion — including the original's other bindings.\n2. **`conformance/`** — language-agnostic golden bundles plus expected outputs that the implementation must reproduce, including a `content_hash` parity lock and a hidden-directory guard.\n\nIn the original these existed to hold five independent implementations byte-identical. Here there is one implementation, so they serve as a **behavioural regression gate** instead — and an unusually strict one, precisely because every asserted value had to be reproducible in five languages. Any change that moves a conformance-asserted value must update the expected JSON deliberately and say why.\n\n## What it enforces (and tolerates)\n\nPer OKF §6, a bundle is **conformant** iff every non-reserved `.md` has parseable YAML frontmatter with a **non-empty `type`** (a free string — no enum). Everything else is permissive: missing recommended fields, unknown types/keys, broken links, and missing `index.md` produce **findings, never rejection**. See [`docs/SPEC_NOTES.md`](docs/SPEC_NOTES.md).\n\n## Install\n\nPython **3.14+**. The project is uv-managed and is not published to any index —\ninstall from a clone.\n\n```bash\ngit clone git@github.com:Foodini-NG/foodini-okf-ingest.git\ncd foodini-okf-ingest\nuv venv --python 3.14\nuv pip install -e \".[html]\"\n```\n\nThat puts the **`okf-ingest`** command on PATH inside `.venv`, plus the\nimportable `okf` package. The `[html]` extra adds the markdown engine for\n`okf-ingest html`; embeddings and `rag` use a local Ollama server by default and\nneed no extra Python dependency (the embedder is pluggable — see\n[Semantic search](#semantic-search-rag)).\n\nWithout installing: `PYTHONPATH=src python -m okf …`.\n\n> The command is **`okf-ingest`**, not `okf`. Both OKF tools claimed the bare\n> `okf`, so on a machine with okf-generator installed `okf --help` could run the\n> other tool. Renaming it is one of this fork's changes.\n\n## Usage\n\n```python\nimport okf.okf as okf\ncon, summary = okf.ingest(\"path/to/bundle\", db_path=\"catalog.duckdb\")\nokf.search(con, \"revenue\")\n```\n\nBoth produce the same `okf_bundle / okf_concept / okf_link / okf_validation` tables.\n\n### Semantic search (RAG)\n\n> Optional, and overkill for small curated bundles — see\n> [Do you actually need RAG?](#do-you-actually-need-rag). It pays off for large\n> or cross-corpus knowledge bases, not a hand-linked folder of a few dozen concepts.\n\n`embed` chunks concept bodies (paragraph-merged to ~600 chars), embeds each via\na **pluggable embedder** (default: local Ollama `nomic-embed-text`, 768-dim;\nswap in any `texts -> list[vector]` callable), and stores vectors in `okf_chunk`.\n`rag` embeds a query and ranks chunks by cosine similarity using DuckDB's native\n`list_cosine_similarity` — **no vector-DB extension required**. Embeddings are\npart of the shared catalog, so you can embed with one binding and query with the\nother.\n\n## Without the catalog — the lean parse / lint / graph layer\n\nThe DuckDB catalog is the *queryable materialization*; it isn't on the critical\npath for parsing, linting, or graphing a bundle. If you don't want to touch\nDuckDB at all, three functions hand you the whole model as plain data structures\n(R: data frames / lists; Python: dicts / dataclasses) and **never go through the\ncatalog**:\n\n| | R | Python | Rust | Gives you |\n|---|---|---|---|---|\n| Parse | `okf_read()` | `read_bundle()` | `read_bundle()` | concepts (frontmatter + body) |\n| Lint / health | `okf_validate()` | `validate()` | `validate()` | the *same* findings `doctor` reports — broken links, orphans, missing fields, non-ISO timestamps |\n| Graph | `okf_links()` | `links()` | `links()` | resolved + broken edges (markdown **and** `[[wikilinks]]`) |\n| Diff | `okf_diff(a, b)` | `diff(a, b)` | `diff(a, b)` | dir-vs-dir changelog (concepts, types, edges) — the catalog only enters if a side *is* one |\n\n```r\nrd <- okf_read(\"my-bundle\") # no DuckDB\nval <- okf_validate(rd) # data.frame of findings (broken_link, orphan, …)\nlk <- okf_links(rd) # src_path / dst_raw / dst_path / resolved\nsubset(lk, !resolved) # every dangling link, as data — your call what to do\n```\n\n```python\nb = okf.read_bundle(\"my-bundle\") # no DuckDB\nval = okf.validate(b) # list of findings\nlk = okf.links(b) # the edge graph\n[e for e in lk if not e[\"resolved\"]]\n```\n\n```rust\n// The Rust crate IS this layer (plus rank/seeds/diff/fetch), catalog-free:\nlet ing = okf_ingest::ingest(\"my-bundle\")?; // summary + findings + links in memory\nlet broken: Vec<_> = ing.links.iter().filter(|l| !l.resolved).collect();\n```\n\nReach for the catalog (`okf_ingest`) when you want what a SQL engine is *for*:\nSQL `query`, the rendered `html`/`graph`, `context` blobs, or vector `rag`. So\nDuckDB is effectively opt-in by usage — build it when you need it, ignore it when\nyou don't.\n\n## CLI\n\nAfter install, `okf-ingest …` (the console script). Without installing,\n`PYTHONPATH=src python -m okf …` takes the same arguments.\n\n```bash\nokf-ingest validate [--subdir