{
  "markdown": "---\ntype: Project\ntitle: okf-parser\ndescription: Relational inspection and validation for Open Knowledge Format bundles\n---\n\n# okf-parser\n\nRelational inspection and validation for\n[Open Knowledge Format (OKF) v0.2](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/SPEC.md)\nbundles.\n\n`okf-parser` reads an OKF bundle without imposing a domain taxonomy, preserves\nunknown frontmatter fields, and exposes concepts and links as\n[Ibis](https://ibis-project.org/) tables. This makes bundle-wide rules—identity,\nlineage, cardinality, provenance, and profile-specific constraints—expressible\nas deterministic relational checks.\n\n## Why another OKF tool?\n\nThe ecosystem already has good static linters and generators, including\n`okflint`, `okf-cli`, and `google-okf`. This project focuses on a different\nlayer:\n\n- compile a bundle into queryable relational tables;\n- project those same relations into a NetworkX graph;\n- validate OKF v0.2 conformance without rejecting extensions;\n- distinguish normative errors from advisory diagnostics;\n- let projects add cross-concept rules as Ibis expressions;\n- produce stable human-readable and JSON reports for CI and agents.\n\nThe architectural boundary between strict authored OKF and source adapters is documented in\n[`docs/architecture.md`](docs/architecture.md).\n\nThe parser and validation model are inspired by\n[`franklinbaldo/sisprev`](https://github.com/franklinbaldo/sisprev): parse\ndocuments independently from semantic validation, aggregate violations instead\nof failing at the first bad concept, preserve authored bodies, and test\nfilesystem identity explicitly. No Sisprev-specific legal types are copied into\nthe core.\n\n[`mrorigo/rust-okf`](https://github.com/mrorigo/rust-okf) inspired the stable\nlogical key, conservative metadata preservation, BOM/CRLF handling, and clean\nseparation between bundle parsing and downstream query surfaces. Its BM25,\nvector index, storage format, and HTTP server are intentionally outside this\nproject's scope.\n\n## Quick start\n\n```bash\nuv sync\nuv run okf-parser check path/to/bundle\nuv run okf-parser check path/to/bundle --relational-schema okf.schema.sql\nuv run okf-parser inventory path/to/bundle\nuv run okf-parser graph path/to/bundle\nuv run okf-parser format path/to/bundle\nuv run okf-parser format path/to/bundle --write\nuv run okf-parser duckdb path/to/bundle knowledge.duckdb\nuv run okf-parser duckdb path/to/bundle knowledge.duckdb --overwrite\nuv run okf-parser serve\nuv run okf-parser serve --allow-write\n```\n\nThe command exits with status `1` only when normative errors exist. Broken\ncross-links are warnings because OKF v0.2 explicitly says they do not make a\nbundle non-conformant.\n\nThe MCP server is commit-disabled by default. `serve --allow-write` exposes explicit\ncommit tools for formatting, relational apply, spec scaffolding, import, and DuckDB\nexport; preview tools remain available without the flag. Effect annotations describe\nmaximum tool effects but are not authorization or sandboxing.\n\n`format` is an opinionated canonicalizer, not a syntax-preserving rewriter. It\nkeeps [mdformat](https://mdformat.readthedocs.io/) as its base and replaces only\nthe numbering policy for ordered lists: markers are decided while rendering, so\nthey are consecutive (`1. 2. 3.`) and never zero-padded to an even width. A list\nstarting at `101` is written `101. 102. 103.`, appending an item does not rewrite\nthe lines above it, and a list whose markers would exceed CommonMark's nine-digit\nlimit keeps plain numbering.\n\nFormatting will not write a file whose **protected block structure** the rewrite\nwould change; such a file is reported in `skipped_paths` and left on disk.\nProtected means the sequence, nesting and tag of every block, block attributes\nsuch as an ordered list's `start`, and the content of code blocks, raw HTML and\nfrontmatter. Inline content — link targets, emphasis, text inside a paragraph or\ntable cell — is deliberately outside this check, because canonical formatting\nrewrites inline whitespace.\n\n## Excluding paths\n\nA repository that keeps OKF knowledge next to code, a README and vendored\ndependencies has no root that validates cleanly. Checking the repository root\nreports `OKF001` for every unrelated Markdown file; checking each bundle\nseparately makes every link *between* bundles unresolvable, because the target\nsits outside the checked root. Excluding subpaths is what lets one root cover\nthe whole tree, which is the only arrangement under which cross-bundle link\nvalidation runs at all.\n\nPut the patterns in an `.okfignore` beside the bundle, so the exclusions are\nversioned with the content and CI needs no extra flags:\n\n```gitignore\n# vendored dependencies\nvendor\n# the knowledge inside them is not noise\n!vendor/knowledge\n# project Markdown at the root, not knowledge\n/*.md\n```\n\nOr pass them for a single run — the option repeats, and adds to the file rather\nthan replacing it:\n\n```bash\nuv run okf-parser check . --exclude vendor --exclude '/*.md'\n```\n\nEvery command that reads a bundle accepts `--exclude`: `check`, `inventory`,\n`graph`, `format` and `duckdb`. Excluded files are never read and never\nwritten, so `format --write` on a repository root leaves vendored documents\nalone.\n\n### Pattern semantics\n\n`.okfignore` uses **`.gitignore` pattern semantics**, matched against\nPOSIX-style paths relative to the bundle root:\n\n| pattern        | matches                                      | does not match          |\n| -------------- | -------------------------------------------- | ----------------------- |\n| `vendor`       | `vendor/a.md`, `libs/vendor/a.md`            | `equipe/a.md`           |\n| `/vendor`      | `vendor/a.md`                                | `libs/vendor/a.md`      |\n| `vendor/`      | `vendor/a.md` (directory only)               | a *file* named `vendor` |\n| `*.md`         | `README.md`, `items/tarefa.md`               | `items/tarefa.markdown` |\n| `/*.md`        | `README.md`                                  | `items/tarefa.md`       |\n| `docs/**/x.md` | `docs/x.md`, `docs/a/b/x.md`                 | `other/x.md`            |\n| `!vendor/kb`   | re-includes what an earlier pattern excluded |                         |\n\n- a pattern without a separator matches its name **at any depth**; a separator\n  anchors it at the bundle root;\n- `*` and `?` stay inside one segment, `[abc]` and `[!abc]` classes work, and\n  `**` spans segments;\n- a trailing `/` matches directories only;\n- `!` re-includes, and **the last pattern that matches a path decides**;\n- `#` starts a comment, blank lines declare nothing, unescaped trailing spaces\n  are dropped, and `\\#` or `\\!` escape a literal first character.\n\nOne deviation from `.gitignore` is deliberate. Git cannot re-include a path\nwhose parent directory is excluded, because it prunes the walk and never looks\nback, so `vendor` plus `!vendor/knowledge` does nothing there. Here it works:\ndiscovery descends whenever a negation exists. A negation that silently does\nnothing is exactly the surprise this feature exists to prevent. With no\nnegation in the rules the walk still prunes, so excluding a vendored dependency\nof several hundred documents never walks it.\n\nThe same rules are shared with the TypeScript package through\n`conformance/exclusion.json`.\n\n### Migrating from the pre-0.14 semantics\n\nBefore 0.14.0 every pattern was anchored at the bundle root and `!` was a\nliteral character. Two rewrites cover it:\n\n| before   | after     | why                                               |\n| -------- | --------- | ------------------------------------------------- |\n| `*.md`   | `/*.md`   | unanchored patterns now match at any depth        |\n| `vendor` | `/vendor` | keep it root-only; leave as-is to match any depth |\n\nA pattern that begins with a literal `!` now needs `\\!`.\n\n## Requiring a specification per type\n\nOKF v0.2 only requires `type` to be non-empty, so a producer can invent a type,\nemit concepts of it and keep a green `check` while that type's frontmatter\nschema changes underneath its consumers.\n\nThe optional rule below closes that gap without inventing taxonomy: it derives a\ndocument path from each type in use and reports the types whose document is\nabsent.\n\n```bash\nuv run okf-parser check ./bundle --require-spec \".okf/specs/{slug}.md\"\nuv run okf-parser check ./bundle --require-spec \".okf/specs/{slug}.md\" --normative-spec\n```\n\nThe template must contain `{slug}`. The slug is lowercase, with accents and\ncedillas removed, whitespace and `/` turned into hyphens, and every remaining\nnon-alphanumeric character dropped:\n\n| `type`            | derived path                    |\n| ----------------- | ------------------------------- |\n| `Spec`            | `.okf/specs/spec.md`            |\n| `Revisão Ciência` | `.okf/specs/revisao-ciencia.md` |\n| `Peça Forense`    | `.okf/specs/peca-forense.md`    |\n\nThe path is **derived**, not declared. A `spec:` frontmatter field would be a\nsecond fact free to disagree with the first, and putting the path in `type`\nitself would tie identity to layout, so renaming a directory would invalidate\nevery concept of that type.\n\nMissing documents are reported as `OKF010` warnings, because a bundle mid-\nadoption legitimately has legacy types without a document and that is not an\nOKF v0.2 defect. `--normative-spec` promotes them to errors for a bundle that\nhas completed the adoption. The rule is off unless `--require-spec` is given.\n\n## GitHub Actions\n\nAdd the repository as a CI check:\n\n```yaml\nsteps:\n  - uses: actions/checkout@v7\n  - uses: franklinbaldo/okf-parser@v0.45.9\n    with:\n      path: knowledge\n```\n\n### Rust end-to-end engine\n\nThe default Python and TypeScript implementations remain portable and require no Rust\nbinary. For large bundles, build the native engine and pass its path explicitly:\n\n```bash\ncargo build --release --manifest-path rust-core/Cargo.toml\n```\n\nPython accepts `load_bundle(root, rust_core=Path(\".../okf-parser\"))`; TypeScript accepts\n`loadBundle(root, { rustCore: \".../okf-core\" })`. The native process owns discovery,\nbounded parallel reads, YAML/frontmatter, Markdown facts, validation, link resolution,\nand content digests. The packaged `okf-parser` executable is the single Python command;\nits ordinary subcommands dispatch to the Python CLI while private engine operations stay\ninside the same executable. Omitting the option keeps the portable language-native fallback.\n\nThe composite action installs a pinned uv version and executes the same\n`validate_path()` function used by the Python API, CLI, and MCP server.\n\nPin an exact version. There is no moving `@v1` ref: this repository's tags are\npackage versions, because `publish.yml` refuses to publish a release whose tag\ndoes not equal the version in `pyproject.toml`. A `v1` tag would either break\nthat check or drift away from the version it claims to be. A major-version ref\nbecomes worth introducing when the package itself reaches `1.0.0`.\n\nReleases are published to PyPI from GitHub Releases through OIDC Trusted\nPublishing. No long-lived PyPI token is stored in the repository.\n\nEvery pull request must increase the SemVer version in `pyproject.toml` and add\nexactly one matching `changelog/<version>.md` entry. CI compares both against\nthe target branch before allowing merge.\n\n## MCP\n\n```bash\nuv run okf-parser serve\nuv run fastmcp run\n```\n\nRead-only tools: `check`, `inventory`, `graph`, and `format_check`.\n\n## DuckDB\n\n`okf-parser` is a regular uv-managed Python app; the DuckDB integration is part\nof the same package, not a native C++ subproject:\n\n```python\nimport duckdb\n\nfrom okf_parser.duckdb import attach_okf\n\nconnection = duckdb.connect(\"knowledge.duckdb\")\nattach_okf(connection, \"knowledge/\")\n\nconnection.sql(\"\"\"\n    SELECT concept_type, count(*)\n    FROM okf.concepts\n    GROUP BY concept_type\n\"\"\").show()\n```\n\nThe call creates `okf.concepts`, `okf.links`, `okf.reserved`, and\n`okf.diagnostics` as ordinary DuckDB tables. Materializing twice into the same\nschema raises `BundleExportError` rather than clobbering an earlier export; pass\n`overwrite=True` (or `--overwrite` on the command line) to replace the four\ntables.\n\n## Python API\n\n```python\nfrom pathlib import Path\n\nfrom okf_parser import load_bundle, validate_path\n\nbundle = load_bundle(Path(\"knowledge\"))\nprint(bundle.concepts.execute())\nprint(bundle.links.execute())\nprint(bundle.to_networkx())\nprint(bundle.validate())\n\nreport = validate_path(Path(\"knowledge\"))\nassert report.markdown_count == report.concept_count + report.reserved_count\nassert report.is_conformant\n```\n\nDeclared RFC 0006 types can be queried directly as Ibis relations without a CLI or database-file round trip:\n\n```python\nbundle = load_bundle(Path(\"knowledge\"))\nwith bundle.compile_types(\"docs/types/{slug}.md\") as typed:\n    routines = typed[\"Rotina\"]\n    print(routines.filter(routines.custo.notnull()).execute())\n```\n\n`TypedRelations` owns an ephemeral DuckDB/Ibis backend; use it as a context manager or call `close()` explicitly. With no matching declarations, `tables` is empty rather than silently inventing an inferred physical schema.\n\n`load_bundle`, `validate_path` and `format_path` read the bundle's `.okfignore`\non their own, and take an `exclude` sequence for patterns supplied per call:\n\n```python\nreport = validate_path(Path(\".\"), exclude=[\"vendor\", \"*.md\"])\n```\n\n`validate_path` also takes the optional type-specification rule:\n\n```python\nreport = validate_path(Path(\"knowledge\"), require_spec=\".okf/specs/{slug}.md\")\n```\n\n## Current scope\n\n- UTF-8 Markdown discovery, matching `.md` case-insensitively, with\n  `.gitignore`-compatible exclusions from `.okfignore` or `--exclude`;\n- reserved `index.md` and `log.md` handling;\n- strict YAML-frontmatter parsing for concept documents, validated with Pydantic\n  at the parse boundary so one malformed document cannot abort a run;\n- required non-empty `type`, optionally requiring a specification document per\n  type in use;\n- stable concept IDs derived from paths;\n- Markdown-link extraction and resolution;\n- Ibis tables for concepts, reserved documents, and links;\n- NetworkX graph projection for traversal, cycles, components, and impact;\n- aggregated validation reports.\n\nProfiles, lifecycle/provenance family validation, external resources, and\npluggable Ibis rules are the next milestones.\n",
  "bytes": 14297,
  "sha": "9a3ae3066f36b2ceb06ba199c207b7aa0dee0072ace5a4c59170561fa85a9422",
  "repo_slug": "franklinbaldo/okf-parser",
  "fonte": "repo",
  "truncated": false,
  "api": "https://agentalog.com/api/listings/okf_franklinbaldo_okf_parser_src_okf_parser__f86634a8/readme"
}