{
  "markdown": "# neo4j-okf — Google's Open Knowledge Format meets Neo4j\n\nDemo + talk assets showing how to map [OKF (Open Knowledge Format)](https://github.com/GoogleCloudPlatform/knowledge-catalog/tree/main/okf)\nbundles onto a Neo4j property graph, ingest them deterministically, and query them with\n[**neo4j-graphrag**](https://neo4j.com/docs/neo4j-graphrag-python/current/) — including the places where the\ngraph visibly beats vector-only RAG (governance-aware retrieval, impact analysis, Text2Cypher).\n\n```\nokf bundle (markdown + frontmatter)          Neo4j property graph\n────────────────────────────────────         ─────────────────────────────────────────────\nmetrics/gross-margin.md      ────────▶       (:Concept:Metric {status, trust_tier, …})\n  [link](../computations/x.md)  ─────▶         -[:LINKS_TO {section}]->(:Concept:AttestedComputation)\n  sources: [...]                ─────▶         -[:DERIVES_FROM]->(:Source)-[:RESOLVES_TO]->(:Concept)\n  verified: {by: human:…}       ─────▶         -[:VERIFIED_BY {at}]->(:Actor {kind:'human'})\n  # Definition …                ─────▶         -[:HAS_SECTION]->(:Section {embedding})-[:CITES]->(:Source)\n```\n\n…and back out again. The graph is not a dead end: it **projects** to a portable\nOKF bundle, and arbitrary documents can be turned into one.\n\n```\n                    parse                              project\n  OKF bundle  ─────────────▶  ┌───────────────┐  ◀─────────────  Neo4j\n                              │ ParsedBundle  │                    ▲\n  documents,  ─────────────▶  └───────────────┘  ─────────────┐    │\n  web pages       wiki               │                emit    │    │ ingest\n                                     └────────────────────────┴────┘\n                                            OKF bundle (files)\n```\n\nThree things follow from making `ParsedBundle` the hub rather than the graph:\n\n* **Projection is a query, not an export.** `okf-graph project` takes a\n  selection — trust tier, staleness, tags, a seed concept and a hop radius —\n  so the bundle you get out is one the graph decided on, not one that ever\n  existed on disk. \"Assemble the human-reviewed, non-stale context for gross\n  margin, as a tarball\" is a Cypher query.\n* **LLM-authored knowledge enters through the same door.** `okf-graph wiki`\n  turns documents into an OKF bundle on disk, then the *existing* deterministic\n  parser ingests it. One mapping, nothing to drift, and the model's output is a\n  git-diffable artifact a human can review in a PR.\n* **Round-trip is a tested property.** `parse → ingest → project → emit → parse`\n  returns an equivalent model, and the projection is byte-identical to\n  serializing the parse directly (`tests/test_roundtrip.py`, `tests/test_project.py`).\n\n## Quickstart\n\nDependencies are managed with [uv](https://docs.astral.sh/uv/) — one `uv sync` creates the venv, installs\neverything, and installs `okf_graph` itself (editable) with the `okf-graph` CLI.\n\n```bash\ndocker compose up -d                   # Neo4j 2025.x at bolt://localhost:7687 (neo4j/demodemo)\nuv sync                                # deps + package + CLI (.venv, uv.lock)\ncp .env.example .env                   # add your OPENAI_API_KEY\n\n# ingest the sample bundle (deterministic — runs with NO api key)\nuv run okf-graph ingest bundles/acme_retail --reset\n\n# + embeddings + vector/fulltext indexes (needs OPENAI_API_KEY)\nuv run okf-graph ingest bundles/acme_retail --reset --embed\n\nuv run jupyter lab notebooks/okf_graphrag_demo.ipynb    # the demo\n```\n\nThen the other two directions — both run offline, with no API key:\n\n```bash\n# graph → OKF. A governed subset, as a portable tarball.\nuv run okf-graph project /tmp/servable --bundle acme_retail \\\n    --min-trust human-reviewed --status stable --exclude-stale --format tar\n\n# a context pack: everything one hop from gross margin, and whatever its\n# Attested Computations need in order to actually be runnable\nuv run okf-graph project /tmp/pack --bundle acme_retail \\\n    --seed metrics/gross-margin --hops 1\n\n# documents → an LLM-authored OKF bundle → the same graph\nuv run okf-graph wiki bundles/acme_wiki --path corpus/acme_intranet --ingest\n```\n\n`--extractor heuristic` is the default and needs neither a key nor the network;\n`--extractor openai` or `--extractor anthropic` swaps in a real model.\n\nOffline smoke test, no key (also safe to rehearse with — the index dimension guard re-embeds when you switch\nback to OpenAI): `uv run okf-graph ingest bundles/acme_retail --reset --embed --embedding-provider hash`\n\nTests: `uv run pytest -q` (the projection tests skip themselves when no Neo4j is running).\n\n## What's here\n\n| path | what |\n|---|---|\n| `okf_graph/parser.py` | OKF v0.2 bundle → in-memory model (frontmatter families, links, sections, footnote citations, logs). Deterministic, permissive per SPEC §11. |\n| `okf_graph/ingest.py` | model → Neo4j: idempotent batched MERGEs, secondary labels from `type`, constraints, embeddings pass, vector + fulltext indexes. No APOC needed here — only the optional `SimpleKGPipeline` appendix uses APOC (neo4j-graphrag's writer/resolver call it; `docker-compose.yml` installs the plugin). |\n| `okf_graph/emit.py` | model → OKF markdown. The inverse of the parser: frontmatter in SPEC key order, regenerated `index.md` (§8), `log.md` (§9), path safety and collision detection. |\n| `okf_graph/project.py` | Neo4j → model → files. Selection filters, closure over the Attested Computation consumer contract, directory / `.tar.gz` / `.zip` writers, and a `.okf/projection.json` manifest recording exactly what was cut. |\n| `okf_graph/documents.py` | source acquisition: local `.md`/`.txt`/`.html`/`.pdf`, URL fetch and shallow crawl, HTML → markdown with headings preserved. Deny-by-default fetch policy (see [Fetching the web](#fetching-the-web)). |\n| `okf_graph/wiki.py` | documents → concept drafts → an OKF bundle. Pluggable extractor: `heuristic` (offline, deterministic), `openai`, `anthropic`. |\n| `okf_graph/queries.py` | governance Cypher (trust tiers, staleness, impact, co-citation) + the `GOVERNED_RETRIEVAL_QUERY` for `VectorCypherRetriever` + Text2Cypher schema/examples. |\n| `okf_graph/embedding.py` | OpenAI embedder factory + deterministic hash embedder for offline rehearsal. |\n| `okf_graph/openwiki.py` | OpenWiki adapter: OKF v0.1 wikis → the same graph, plus groundings, `WikiRun` watermark, impact/staleness/coverage Cypher, wiki retrieval query. |\n| `okf_graph/mcp_server.py` | `okf-graph-mcp` — read-only stdio MCP server (search, change surface, staleness, coverage) over ingested wikis. |\n| `notebooks/okf_graphrag_demo.ipynb` | the live demo: parse → ingest → explore → baseline RAG trap → graph-aware RAG → Text2Cypher → projection → LLM-authored wiki → (appendix) SimpleKGPipeline domain layer. |\n| `notebooks/openwiki_neo4j_demo.ipynb` | the OpenWiki demo: ingest OpenWiki's own wiki → page graph → groundings → impact analysis → staleness → GraphRAG → MCP finale. |\n| `tests/` | `uv run pytest -q` — parser semantics, round-trip equivalence, document normalization, fetch-policy refusals, the wiki pipeline, the OpenWiki adapter, and (Neo4j-gated) projection. |\n| `bundles/acme_retail/` | sample OKF v0.2 bundle vendored from [GoogleCloudPlatform/knowledge-catalog](https://github.com/GoogleCloudPlatform/knowledge-catalog) (Apache-2.0) so the demo runs offline. |\n| `bundles/openwiki_self/` | OpenWiki's own dogfooded wiki (OKF v0.1, MIT) + real `git` sidecars (`.git-changes*.json`, `.repo-files.sample.json`) so the OpenWiki demo runs offline. |\n| `corpus/acme_intranet/` | five documents of simulated org exhaust — a wiki page, a finance memo, a warehouse README, a data dictionary, an on-call runbook — in HTML, markdown and plain text. Input for `okf-graph wiki`. |\n| `slides/okf-neo4j.pptx` | the talk deck (diagrams in `slides/diagrams/`). |\n\n## The mapping\n\n| OKF construct (SPEC v0.2) | Graph element |\n|---|---|\n| bundle | `(:Bundle {name, okf_version})` |\n| concept file | `(:Concept)` + secondary label from `type` (`BigQuery Table` → `:BigQueryTable`) |\n| concept id = path sans `.md` | `Concept.id`, `uid = bundle + ':' + id` |\n| frontmatter scalars | node properties (`status` defaults to `stable`, dates typed as `date()`/`datetime()`) |\n| unknown frontmatter keys | preserved in `Concept.extra_frontmatter` (JSON) — consumers MUST NOT reject (§11) |\n| markdown link | `[:LINKS_TO {section, text, resolved}]` — section-scoped, so prose context survives |\n| broken link (§6.1) | stub `(:Concept {stub:true})` — \"not-yet-written knowledge\" becomes a queryable backlog |\n| directory tree | `Concept.dir` property (+ `IN_BUNDLE`); the tree is derivable, the graph is what's indexed |\n| `sources[]` (§5.1) | `(:Source)` deduped per bundle by resource; intrinsic signals (`author`, `last_modified`) on the node, declaration-scoped (`usage_count`, `usage_window`) on `[:DERIVES_FROM]` |\n| source → internal path | `(:Source)-[:RESOLVES_TO]->(:Concept/:Artifact)` — provenance chains become traversable |\n| `generated` / `verified` (§5.2) | `(:Actor {kind: human/agent/process})` + `[:GENERATED_BY {at}]` / `[:VERIFIED_BY {at}]` |\n| trust tier (§5.3) | derived in Cypher from verifier kinds; also materialized as `Concept.trust_tier` |\n| `status`, `stale_after` (§5.4–5.5) | properties; staleness = `date() >= stale_after` at query time |\n| Attested Computation (§10) | `:AttestedComputation` label + `runtime`, `parameters_json`, `receipt`; `[:EXECUTED_BY]->(:Skill)`, `[:ATTESTED_BY]->(:Artifact)` |\n| body `# sections` (§4.2) | `(:Section {heading, order, text, embedding})` + `[:HAS_SECTION]`, `[:NEXT]` — OKF's conventional headings are the chunking |\n| links, section-scoped | `(:Section)-[:MENTIONS]->(:Concept)` — which section grounds which relationship |\n| footnote refs `[^id]` (§5.1) | `(:Section)-[:CITES]->(:Source)` — claim-level provenance |\n| `computation:` file form (§10.3) | `[:COMPUTATION_FILE]->(:Artifact)` |\n| `tags` | `(:Tag)` + `[:TAGGED]` (and kept as array property) |\n| `log.md` (§9) | `(:LogEntry {date, kind, dir})-[:REFERENCES]->(:Concept)`; any frontmatter kept verbatim on `Bundle.log_frontmatter` |\n| `index.md` (§8) | not ingested — it's derivable (progressive disclosure is a *serving* concern), and the projection regenerates it; root `okf_version` lifted to Bundle |\n| non-`.md` files | `(:Artifact {path, kind, sha256, size, text})` — every one, referenced or not, with contents carried under 64 KB so a projection is runnable rather than just structurally complete |\n\n### Sync semantics\n\n`ingest()` is a **sync**, not an append: for the bundle being ingested it clears replaceable\nrelationships (trust, links, provenance, tags), removes vanished sections/stubs/log entries, refreshes\nsecondary labels, and rebuilds from the parse — so the derived trust tier can never drift from the\nmaterialized `trust_tier`, and a re-run never duplicates edges. Sections keep their embeddings unless\ntheir text changed (changed text nulls the vector so the next embed pass picks it up). `reset(bundle)`\nremains the hard wipe. The demo is single-tenant for clarity; for multi-bundle estates add a `bundle`\npredicate to the governance/retrieval queries (or use per-bundle databases).\n\n## The reverse direction: graph → bundle\n\n`ingest()` is a sync; `project()` is a **query with a serializer on the end**.\n\n| flag | what it selects |\n|---|---|\n| `--bundle` | which bundle in the graph (a directory path works too) |\n| `--concept ID` | just these concepts (repeatable) |\n| `--seed ID --hops N` | everything within N dependency hops of a concept |\n| `--tag`, `--type`, `--status` | frontmatter filters (repeatable) |\n| `--min-trust` | `unverified` \\| `machine-confirmed` \\| `human-reviewed` (§5.3) |\n| `--exclude-stale` | drop concepts where `today >= stale_after` (§5.5) |\n| `--include-referenced` | add one hop of link targets so links resolve |\n| `--format dir\\|tar\\|zip` | directory, `.tar.gz`, or `.zip` — all reproducible |\n| `--dry-run` | print the manifest and file list, write nothing |\n\nTwo behaviours are worth calling out because they are governance, not plumbing:\n\n* **The consumer contract is always closed over.** `skills/run-on-bq` is\n  `unverified`, so `--min-trust human-reviewed` would drop it — while leaving\n  two Attested Computations that name it as their `executor`. A bundle that\n  ships a sanctioned computation without the skill needed to run it asserts a\n  contract nobody can follow, so `EXECUTED_BY` / `ATTESTED_BY` /\n  `COMPUTATION_FILE` targets are pulled back in regardless of the filter.\n* **Nothing is quietly dropped.** Links that point at concepts the filter\n  excluded stay in the markdown — a broken cross-link is legal OKF (§6.1) and\n  it is the honest record — but every one is listed in `.okf/projection.json`,\n  along with unwritten artifacts and the full list of round-trip caveats.\n\nThe manifest lives under a dot-directory on purpose: the parser skips dotted\npaths, so it can never be mistaken for bundle content on the way back in.\n\n## Documents → an LLM-authored wiki → the graph\n\n`okf-graph wiki` is the \"agents are better wiki maintainers than we are\" claim,\nwired up. Documents go in; an OKF bundle comes out; the ordinary parser ingests\nit.\n\n```\ncorpus/acme_intranet/*.{html,md,txt}\n      │  documents.py     strip boilerplate, keep headings, capture provenance\n      ▼\n  concept drafts          heuristic | openai | anthropic\n      │  wiki.py          merge by id, resolve cross-references, mint stubs\n      ▼\n  bundles/acme_wiki/      real OKF: frontmatter, sections, sources[], footnotes\n      │  parser.py        ← the same parser that reads Google's bundles\n      ▼\n  Neo4j\n```\n\nWhat the generated bundle asserts about itself:\n\n| OKF construct | what the wiki builder writes |\n|---|---|\n| `generated.by` (§5.2) | the real producer — `okf-wiki/<model>`, or `process:okf-wiki-heuristic` |\n| `verified` (§5.2) | **absent**, always — so trust tier is `unverified` (§5.3) |\n| `status` (§5.4) | `draft` |\n| `sources[]` (§5.1) | one entry per source document, with its author and last-modified date |\n| `references/<slug>.md` (§6.3) | each source document mirrored as a first-class concept, so `sources[].resource` resolves *inside* the bundle and provenance becomes `(:Source)-[:RESOLVES_TO]->(:Concept)` — a traversal, not a string |\n| `[^sid]` footnotes (§5.1) | per section, so claims carry attribution rather than files |\n| links to unwritten concepts (§6.1) | kept — they become stub nodes, i.e. a queryable authoring backlog |\n\nThe demo point: `corpus/acme_intranet/wiki-gross-margin.html` is an intranet page\nstill describing the **pre-2026 margin formula**. The extractor faithfully\nrecords it — and because the result is `draft` / `unverified`, it lands in the\nsame graph as Finance's `human-reviewed` definition without being able to\noutrank it. That is the whole argument for putting trust in the format.\n\n`build_wiki` re-parses what it just wrote and refuses to return if any concept\ndid not survive. Emitting markdown and reading it back makes the filesystem a\nchannel, and its failure modes — reserved filenames, case-folding collisions,\nduplicate slugs — are all silent; one equality check catches the class.\n\n### Fetching the web\n\n`okf-graph fetch` and `--url` go through a deny-by-default policy: `http`/`https`\nonly, ports 80/443, robots.txt respected, credentials-in-URL refused, response\nsize capped on *decoded* bytes, and every redirect hop re-validated against\nprivate, loopback, link-local, reserved and IPv4-mapped-IPv6 address space.\n`--allow-private-hosts` opts out for intranet testing.\n\nOne residual risk, stated rather than papered over: the name is resolved for the\ncheck and then again by the HTTP client, so a DNS answer that changes between\nthe two (rebinding) is not caught. Fetch untrusted URLs from somewhere that\ncannot reach anything you care about.\n\n## Why a graph (the demo's argument)\n\n1. **Governance queries are one-hop Cypher**: trust tiers, staleness reports, impact analysis, co-citation.\n2. **Governed GraphRAG**: vector similarity alone happily retrieves a *deprecated* metric definition — it's\n   well-written text about exactly the topic. `VectorCypherRetriever` walks from the matched `:Section` to its\n   `:Concept`, reads `status`/`trust_tier`/`stale_after`, follows `LINKS_TO` to the sanctioned\n   `:AttestedComputation` and its SQL, and hands the LLM context that carries governance.\n3. **Text2Cypher** answers questions that have no similarity anchor (\"which metrics were never human-reviewed?\").\n4. **Two construction modes compose**: deterministic structural ingestion (no LLM) + optional\n   `SimpleKGPipeline` entity extraction over the prose — lexical layer and domain layer in one graph.\n5. **The format is the interchange, the graph is the selection.** Because the\n   projection is lossless, nothing is locked in: OKF goes in, OKF comes out, and\n   what the database adds is the ability to decide *which* OKF comes out.\n\n## Round-trip fidelity, precisely\n\nThe claim is **equivalence**, not byte equality, and the difference is\nenumerated rather than hand-waved:\n\n* `parse → emit → parse` reproduces the model exactly — concepts, sections,\n  links, citations, sources, trust tiers, log references and artifact contents.\n* `parse → ingest → project → emit` is **byte-identical** to `parse → emit`.\n* `emit` is idempotent from the second pass; the first pass normalizes.\n\nWhere re-emitted text differs from hand-authored text — regenerated `index.md`,\nexplicit `status: stable`, `Z` normalized to `+00:00`, unknown keys moved below\nthe known families — every case is listed in `emit.ROUNDTRIP_NOTES` and copied\ninto each projection's manifest. Known limits: unknown keys *nested* inside a\nknown family (`generated.model`, `sources[].license`) are not retained, and\nartifacts over 64 KB or non-UTF-8 are recorded by hash and path rather than\nwritten.\n\n## OpenWiki wikis, backed by Neo4j\n\n[OpenWiki](https://github.com/langchain-ai/openwiki) (LangChain's agent-written repo wiki) emits\nOKF v0.1 bundles — and computes a page graph, an impact plan, and grounding metadata that it\n**throws away on every run**. The `okf_graph.openwiki` adapter ingests any OpenWiki wiki and\npersists all of it:\n\n```bash\nuv run okf-graph ingest-wiki bundles/openwiki_self --reset --embed --embedding-provider hash\nuv run okf-graph impact --bundle openwiki_self \\\n    --changes bundles/openwiki_self/.git-changes.wide.sample.json\nuv run jupyter lab notebooks/openwiki_neo4j_demo.ipynb     # the demo\n```\n\nOn top of the standard OKF mapping, the wiki layer adds:\n\n| OpenWiki construct | Graph element |\n|---|---|\n| backticked repo path in prose | `(:Concept)-[:GROUNDED_IN {sections, mentions}]->(:SourceFile {path, kind})` |\n| `openwiki:` frontmatter extension (`source_paths`, `symbols`, `test_paths`, `invariants`) | declared `GROUNDED_IN` edges + `(:Symbol)`, `(:Invariant)`, `[:VALIDATED_BY]` |\n| `.last-update.json` (`gitHead` watermark) | `(:WikiRun {git_head, updated_at, model})-[:PRODUCED]->(:Bundle)` |\n| reserved docs (`index.md`, `INSTRUCTIONS.md`, `_plan.md`, …) | excluded, per OpenWiki's own rules |\n\nThat turns `git diff --name-only <gitHead>..HEAD` into a **persistent impact plan** (OpenWiki\nderives one into `_plan.md` per run and deletes it), gives per-page staleness verdicts instead\nof one SHA for the whole wiki, and surfaces coverage gaps and dangling groundings the flat\nfiles can't see. `bundles/openwiki_self/` vendors OpenWiki's own dogfooded wiki (MIT) with real\n`git` sidecars so everything runs offline.\n\n### The retrieval MCP they never shipped\n\nOpenWiki's eval harness references an `openwiki-retrieval-mcp` that doesn't exist upstream.\nThis repo ships it, backed by the graph — read-only tools `wiki_search`, `wiki_get_page`,\n`wiki_change_surface`, `wiki_stale_pages`, `wiki_coverage_gaps`, `wiki_list_bundles`:\n\n```bash\nuv run okf-graph-mcp                      # stdio server (env: OPENWIKI_BUNDLE, OPENWIKI_REPO_ROOT)\nuv run python scripts/smoke_mcp.py        # end-to-end smoke test\n\n# register with Claude Code:\nclaude mcp add openwiki-graph -- uv --directory /path/to/neo4j-okf run okf-graph-mcp\n```\n\nGenerate a wiki for your own repo and ingest it: `./scripts/generate-wiki.sh /path/to/repo`.\n\n## Attribution\n\n`bundles/acme_retail` and the OKF specification are from\n[GoogleCloudPlatform/knowledge-catalog](https://github.com/GoogleCloudPlatform/knowledge-catalog), Apache License 2.0.\n`bundles/openwiki_self` is a snapshot of the `openwiki/` wiki in\n[langchain-ai/openwiki](https://github.com/langchain-ai/openwiki), MIT License (upstream commit in\n`.provenance.json`).\nThis repo is a community demo and is not affiliated with Google or LangChain.\n",
  "bytes": 20680,
  "sha": "62010ef0a4a0c86f959ad637d48e35a8ee210451fc0bb728f815b3bd526c94f1",
  "repo_slug": "johnymontana/neo4j-okf",
  "fonte": "repo",
  "truncated": false,
  "api": "https://agentalog.com/api/listings/okf_johnymontana_neo4j_okf_bundles_openwiki__34740ba4/readme"
}