{
  "markdown": "<p align=\"right\">\n<a href=\"README.md\">English</a> | <a href=\"README.zh-CN.md\">中文</a>\n</p>\n\n# okf — Open Knowledge Format\n\n> Project-level knowledge base system for AI Agents, with automatic Git repository scanning, specification linting, and automated updates.\n\n[![CI](https://github.com/superops-team/okf/actions/workflows/go.yml/badge.svg)](https://github.com/superops-team/okf/actions/workflows/go.yml)\n[![Latest Release](https://img.shields.io/github/v/release/superops-team/okf?label=release&logo=github&style=flat-square)](https://github.com/superops-team/okf/releases)\n[![Go Version](https://img.shields.io/github/go-mod/go-version/superops-team/okf?logo=go&style=flat-square)](go.mod)\n[![Platform](https://img.shields.io/badge/platform-Linux%20%7C%20macOS%20%7C%20Windows-blue?style=flat-square)](#installation)\n[![License](https://img.shields.io/badge/license-Apache--2.0-blue?style=flat-square)](LICENSE)\n[![GitHub Stars](https://img.shields.io/github/stars/superops-team/okf?style=flat-square)](https://github.com/superops-team/okf)\n[![GitHub Downloads](https://img.shields.io/github/downloads/superops-team/okf/total?style=flat-square)](https://github.com/superops-team/okf/releases)\n\n**okf** turns your Git repository into a living, queryable knowledge base that humans *and* AI Agents can use. Every piece of knowledge is a Markdown concept file with YAML frontmatter, generated from your code and documents automatically and kept up to date on every commit.\n\n## Table of Contents\n\n- [Features](#features)\n- [How it works](#how-it-works)\n- [Installation — Quick Start (30 seconds)](#installation--quick-start-30-seconds)\n- [Usage](#usage)\n- [Documentation](#documentation)\n- [Project Structure](#project-structure)\n- [Module Reference](#module-reference)\n- [OKF Concept Format](#okf-concept-format)\n- [API Usage](#api-usage)\n- [Lint Rules](#lint-rules)\n- [Build & Test](#build--test)\n- [OKF v0.2 Specification Support](#okf-v02-specification-support)\n- [Contributing](#contributing)\n- [License](#license)\n\n## Features\n\n- **📁 Open Knowledge Format** — Open knowledge format based on Markdown + YAML Frontmatter\n- **📄 Document Import** — Import PDF, DOCX, XLSX, PPTX, HTML, CSV, TXT directly (pure-Go conversion, no Python/CGO); `okf add report.pdf` just works\n- **🔍 Auto-Generation** — Automatically generates knowledge base by scanning Git repository source code\n- **⚡ Incremental Updates** — Incremental updates based on Git commits\n- **🛠 Git Hook** — One-click installation, automatic knowledge base updates on every commit\n- **📋 Lint Checking** — Built-in specification compliance checker (16 rules)\n- **🔎 Advanced Query** — Filter by type, tags, or full-text search\n- **🧠 Hybrid Semantic Search** — Local natural-language search: chunk-level MiniLM embeddings + BM25, fused with weighted RRF (fully offline, no CGO)\n- **🤖 Agent-facing MCP** — Standard MCP tools for repository status/init/refresh/query/context plus durable note/event/feedback capture\n- **🏗 Modular Architecture** — Clean, layered design following Go best practices\n\n## How it works\n\n```mermaid\nflowchart LR\n    A[Your Git repository] -->|\"okf init / scan\"| B[.okf/knowledge<br/>Markdown concepts]\n    C[\"PDF · DOCX · XLSX · PPTX<br/>HTML · CSV · TXT\"] -->|\"okf add\"| B\n    D[git commit] -->|\"okf hook / sync\"| B\n    B --> E[\"okf lint<br/>OKF v0.2 checks\"]\n    B --> F[\"okf search / query\"]\n    B --> G[\"MCP server<br/>status · init · refresh · query · context\"]\n    G --> H[AI Agents]\n```\n\n## Installation — Quick Start (30 seconds)\n\nPick one of these three install methods:\n\n### 1. One-click installer (recommended)\n\n**Linux / macOS:**\n\n```bash\ncurl -fsSL https://raw.githubusercontent.com/superops-team/okf/main/scripts/install.sh | bash\n```\n\n**Windows (PowerShell):**\n\n```powershell\nirm https://raw.githubusercontent.com/superops-team/okf/main/scripts/install.ps1 | iex\n```\n\n> If the one-liner fails with `Unexpected token` / `&#34;` parse errors (caused by proxies HTML-encoding the response), use the download-then-run method:\n> ```powershell\n> iwr -useb \"https://raw.githubusercontent.com/superops-team/okf/main/scripts/install.ps1\" -OutFile install.ps1; .\\install.ps1\n> ```\n\nThe installer:\n- Automatically detects your OS (Linux / macOS) and CPU architecture (amd64 / arm64)\n- Downloads the latest pre-built binary from GitHub Releases\n- Verifies SHA256 checksums\n- Installs to `/usr/local/bin/` (or `~/.local/bin/` without sudo)\n\n### 2. Install via Go\n\n```bash\ngo install github.com/superops-team/okf/cmd/okf@latest\n```\n\n### 3. Download from releases\n\nDownload pre-built binaries for your platform from the\n[Releases](https://github.com/superops-team/okf/releases) page.\n\n| OS | Architecture | Archive |\n|----|-------------|---------|\n| Linux | amd64 (x86_64) | `okf_<version>_linux_amd64.tar.gz` |\n| Linux | arm64 (aarch64) | `okf_<version>_linux_arm64.tar.gz` |\n| macOS | amd64 (Intel) | `okf_<version>_darwin_amd64.tar.gz` |\n| macOS | arm64 (Apple Silicon) | `okf_<version>_darwin_arm64.tar.gz` |\n| Windows | amd64 | `okf_<version>_windows_amd64.zip` |\n| Windows | arm64 | `okf_<version>_windows_arm64.zip` |\n\n---\n\n## Usage\n\n```bash\n# Initialize knowledge base from your repo\ncd /your/repo\nokf init\n\n# Show knowledge base information\nokf show\n\n# Search concepts\nokf search -q \"database\"\n\n# Import a real document (converts PDF/DOCX/XLSX/... to Markdown)\nokf add report.pdf\n\n# Lint check\nokf lint\n\n# Semantic (natural-language) search — build the index once, then search\nokf vector index\nokf search -q \"check my notes for errors\" -semantic\n\n# Install Git Hook (automatic updates on every commit)\nokf hook -type post-commit\n\n# Start the MCP server for a repository. Relative --dir values resolve under --repo;\n# absolute --dir values remain absolute.\nokf mcp --repo /your/repo --dir .okf/knowledge\n```\n\n### Agent-facing MCP tools\n\nThe MCP server exposes the repository knowledge service through `okf_status`, `okf_init`, `okf_refresh`, `okf_query`, and `okf_context`. Durable knowledge capture is available through `okf_note`, `okf_log`, and `okf_feedback`; `okf_ask` queries only those durable note/event/feedback concepts. Existing bundle/list/get/search/lint/document-import tools remain available.\n\nWrites require a stable `idempotency_key`, use deterministic identities, reject unknown or incorrectly typed fields, and fail closed for path escape, symlink-root, size-limit, and credential-like metadata violations. The server persists only feedback explicitly submitted by the caller; it does not inspect a host application's private event bus. See [`docs/knowledge/mcp-server.md`](docs/knowledge/mcp-server.md) and [`docs/knowledge/durable-capture.md`](docs/knowledge/durable-capture.md).\n\n## Semantic Search\n\n`okf search -semantic` performs natural-language search over concepts, using a locally embedded MiniLM model (384-dim vectors) and an HNSW index — no network, no external runtime required. Long documents are split into heading-aware chunks, and results blend a **semantic** channel with a **BM25 lexical** channel via weighted Reciprocal Rank Fusion.\n\n```bash\n# Build (or incrementally update) the vector index — one-time, per knowledge base\nokf vector index\n# Inspect index state (chunks, concepts, index format version)\nokf vector status\n# Full rebuild (after content changes, or when upgrading index format)\nokf vector rebuild\n# Search semantically (hybrid: semantic + BM25, fused with weighted RRF)\nokf search -q \"check my notes for errors\" -semantic\n# Pure semantic, no lexical channel\nokf search -q \"how do I rebuild the index\" -semantic -lexical-weight 0\n```\n\nResults are annotated with their source: `semantic`, `lexical`, or `both`. If no index exists, `-semantic` warns and falls back to lexical search. The MCP server exposes the same capability via `okf_semantic_search`.\n\n### Measuring retrieval quality\n\n`okf eval` scores retrieval against a golden query set and can compare strategies side by side:\n\n```bash\nokf eval -golden pkg/eval/testdata/golden_semantic.json -path docs/knowledge -compare\n```\n\nMeasured on this repository's own knowledge base (28 queries, 26 positive, K=5):\n\n| Strategy | Recall@5 | MRR |\n|---|---|---|\n| `lexical-substring` (pre-0.5.0 behaviour) | 0.0769 | 0.0769 |\n| `bm25-only` | 0.8077 | 0.6538 |\n| `semantic-only` | 0.9615 | 0.7096 |\n| **`hybrid-default`** | **0.9615** | **0.7256** |\n\n### How it works & constraints\n\n- **Chunked indexing**: concepts are split on `##`–`####` headings into ≤1024-character chunks (code fences and tables are never split), each carrying a `Title > Section` breadcrumb. This matters because MiniLM truncates at 256 tokens: indexing whole concepts dropped **70.6%** of this repository's knowledge-base content, so text past the truncation point was unsearchable.\n- **Hybrid retrieval**: the semantic channel (HNSW over chunk vectors) and the BM25 channel are fused with weighted RRF (`k=60`, equal weights by default). Tune with `-lexical-weight`; `0` disables the lexical channel. BM25 tokenizes identifiers into subwords (`okf_semantic_search` → `okf`/`semantic`/`search`) and CJK text into overlapping bigrams, with no dictionary dependency.\n- **Reproducibility**: indexes below 2048 chunks are searched by exact scan rather than HNSW's approximate traversal, because the approximate path does not return every node even when asked for all of them, and which nodes it misses shifts between rebuilds. Combined with a fixed RNG seed and deterministic tie-breaks, this makes results identical across rebuilds — verified by rebuilding this knowledge base repeatedly and confirming the evaluation metrics do not move.\n- **Index cost (measured, 7 concepts → 97 chunks)**: chunking increases index size ~20x (14.5 KB → 287 KB) and build time ~6x (128 ms → 800 ms). Both scale with content volume, not concept count.\n- **Index format v2 is not backward compatible**: chunk-level keys differ from the old concept-level keys. `okf vector status` reports the format version, and loading an older index fails with an explicit prompt to run `okf vector rebuild` (search falls back to lexical meanwhile) rather than silently returning wrong results.\n- **Embedded resources**: the ONNX Runtime CPU library (per-OS, ~10–15 MB) plus a quantized MiniLM model (~23 MB) are embedded into the binary via `go:embed` and extracted to the user cache directory on first use (checksum-verified). Building for each platform only embeds that platform's resources (`scripts/fetch-ort.sh` / `scripts/fetch-model.sh` fetch them at build time; the runtime never goes online).\n- **Dynamic loading (transparency)**: the ONNX Runtime shared library is loaded at runtime via `dlopen` from the extracted cache — the binary is self-contained but not statically linked. Cache location: `os.UserCacheDir()/okf/` (override with `OKF_ORT_DIR`).\n- **Limits**: MiniLM embeddings are English-centric. Chunking and BM25's CJK bigrams improve Chinese retrieval, but a purely Chinese query against English content still relies on the semantic channel alone. `Embedder` is an interface, leaving room for stronger models (e.g. BGE-M3) or remote APIs later.\n- **Licenses**: pure-onnx (MIT), coder/hnsw (CC0-1.0), ONNX Runtime (MIT), MiniLM-L6-v2 model (Apache-2.0).\n\n## Documentation\n\n- [Knowledge base index](docs/knowledge/index.md) — module overview\n- [CLI reference](docs/knowledge/cli.md)\n- [Lint rules](docs/knowledge/lint.md)\n- [MCP server](docs/knowledge/mcp-server.md)\n- [Durable knowledge capture](docs/knowledge/durable-capture.md)\n- [v0.2 example — income statement](examples/v0.2/income-statement/)\n\n## Project Structure\n\n```\n.\n├── cmd/okf/          # CLI entry point\n│   └── main.go      # Main application\n├── pkg/\n│   ├── okf/         # Core types and public API\n│   │   ├── types.go # Concept, KnowledgeBundle definitions\n│   │   ├── api.go   # LoadBundle, SaveBundle\n│   │   ├── errors.go # Error types\n│   │   ├── helpers.go # Helper functions\n│   │   └── meta/    # Version information\n│   ├── parser/      # Markdown + YAML parser\n│   │   └── parser.go\n│   ├── query/       # Query engine\n│   │   └── query.go\n│   ├── lint/        # Specification checker\n│   │   └── lint.go\n│   ├── git/         # Git integration\n│   │   ├── git.go       # Git operations\n│   │   └── generator.go # Knowledge base generation\n│   ├── convert/     # Pure-Go document conversion (PDF/DOCX/XLSX/PPTX/HTML/CSV/TXT → Markdown)\n│   ├── mcp/         # MCP server (status/init/refresh/query/context + durable capture)\n│   └── tool/        # Durable note/event/feedback capture tools\n├── go.mod\n├── README.md            # English version (default)\n└── README.zh-CN.md      # Chinese version\n```\n\n## Module Reference\n\n| Module | Path | Purpose |\n|--------|------|---------|\n| **okf** | pkg/okf/ | Core type definitions (Concept, KnowledgeBundle) and public API |\n| **parser** | pkg/parser/ | Markdown + YAML frontmatter parsing and serialization |\n| **query** | pkg/query/ | Advanced query builder and matching engine |\n| **lint** | pkg/lint/ | OKF specification compliance checking (16 rules) |\n| **git** | pkg/git/ | Git repository scanning, code analysis, knowledge base generation |\n| **convert** | pkg/convert/ | Pure-Go document import (PDF/DOCX/XLSX/PPTX/HTML/CSV/TXT/DOC → Markdown) |\n| **mcp** | pkg/mcp/ | MCP server for AI agent integration |\n| **tool** | pkg/tool/ | Durable note/event/feedback capture |\n\n## OKF Concept Format\n\n```markdown\n---\ntype: table\ntitle: users\ndescription: User accounts table\nresource: bigquery.project.dataset.users\ntags:\n  - production\n  - pii\ntimestamp: \"2024-01-15T10:30:00Z\"\n---\n\n## Users Table\nStores all user account information.\n```\n\n## API Usage\n\n```go\nimport (\n    okf \"github.com/superops-team/okf/pkg/okf\"\n    \"github.com/superops-team/okf/pkg/git\"\n    \"github.com/superops-team/okf/pkg/lint\"\n)\n\n// Load knowledge base\nbundle, err := okf.LoadBundle(\".okf/knowledge\", nil)\n\n// Search concepts\nresults := bundle.Search(\"database\")\n\n// Lint check\nresult := lint.LintBundle(concepts, lint.DefaultConfig())\n\n// Generate from Git\nbundle, err := git.GenerateBundle(cfg, false)\n```\n\n## Lint Rules\n\n| Code | Severity | Description |\n|------|----------|-------------|\n| OKF001 | ERROR | `type` field is required and must not be empty |\n| OKF002 | WARNING | `title` is recommended but missing (derived from filename in v0.2) |\n| OKF003 | WARNING | `description` is too short |\n| OKF004 | INFO | `type` uses mixed case (valid for spec-defined types such as `Attested Computation`) |\n| OKF005 | WARNING | `generated.at` is recommended but missing, or not a valid ISO 8601 timestamp |\n| OKF006 | WARNING | tags contain uppercase or spaces |\n| OKF007 | WARNING | content body is empty |\n| OKF009 | WARNING | content lines are too long |\n| OKF010 | WARNING | duplicate tags found |\n| OKF011 | WARNING | required tag is missing |\n| OKF012 | WARNING | `sources` is recommended but missing |\n| OKF013 | WARNING | duplicate title across concepts |\n| OKF014 | ERROR | Attested Computation requires `runtime` field |\n| OKF015 | WARNING | `stale_after` is not a valid YYYY-MM-DD date |\n| OKF016 | INFO | legacy `timestamp` detected; consider migrating to `generated.at` |\n| OKF017 | INFO | `verified` is recommended to elevate the trust tier |\n\n## Build & Test\n\n```bash\n# Build\ngo build ./...\n\n# Build CLI\ngo build -o okf ./cmd/okf/\n\n# Run all tests\ngo test ./...\n\n# Run benchmarks\ngo test -bench=. -benchmem ./...\n```\n\n## Evaluation\n\nokf ships a reproducible IR (information-retrieval) quality benchmark that\nquantifies search quality using canonical metrics.\n\n### Metrics\n\n| Metric | Definition |\n|--------|-----------|\n| **Recall@K** | Fraction of expected relevant docs found in top-K results |\n| **Precision@K** | Fraction of top-K results that are relevant |\n| **MRR** | Mean Reciprocal Rank — 1/rank of the first relevant result |\n| **NDCG@K** | Normalized Discounted Cumulative Gain (binary relevance) |\n\n### Running the benchmark\n\n```bash\ntools/eval.sh\n```\n\nThis runs 20 golden queries (18 positive, 2 negative) across all 7 document\nformats and prints per-case and aggregate scores.\n\n### Baseline (K=5, 20 cases)\n\n| Metric | All cases | Positive only |\n|--------|-----------|---------------|\n| Recall@5 | 1.0000 | 1.0000 |\n| Precision@5 | 0.9000 | 1.0000 |\n| MRR | 0.9000 | 1.0000 |\n| NDCG@5 | 1.0000 | 1.0000 |\n\nAll 18 positive queries return the correct top-1; both negative queries\nreturn zero results. The golden set lives in\n`pkg/eval/testdata/golden_queries.json` and metric implementations in\n`pkg/eval/`.\n\n## OKF v0.2 Specification Support\n\nThis project implements the [OKF v0.2 specification](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/SPEC.md) with full backward compatibility for v0.1.\n\n### What's New in v0.2\n\n- **Provenance** — `sources` field with material references, usage counts, and credibility signals\n- **Trust** — `generated` (by/at) and `verified` (list of verification events) fields with trust tier derivation (unverified → machine-confirmed → human-reviewed)\n- **Lifecycle** — `status` (stable/draft/deprecated) and `stale_after` (YYYY-MM-DD) fields\n- **Attested Computation** — new concept type with `runtime`, `parameters`, `computation`, `executor`, and `attester` fields\n- **Reserved filenames** — `index.md` (directory listing) and `log.md` (update history)\n- **Only `type` is required** — `title` is now optional and derived from filename if missing\n\n### Backward Compatibility\n\n- v0.1 `timestamp` field is automatically mapped to `generated.at`\n- v0.1 body `# Citations` section is automatically extracted to `sources`\n- Legacy `generated: true` (boolean) is preserved for backward compatibility\n- All v0.1 concepts parse without errors in v0.2 mode\n\n### Official Example\n\nSee [`examples/v0.2/income-statement/`](examples/v0.2/income-statement/) for the complete Appendix A income statement example from the spec. The [v0.2 core types](docs/knowledge/core-types.md) document covers the full field reference.\n\n## Contributing\n\nContributions are welcome! The project follows a strict SDD → TDD workflow:\n\n1. **SDD** — write a change proposal under `openspec/changes/<change-id>/` (`proposal.md` / `design.md` / `spec.md` / `tasks.md`)\n2. **TDD** — write tests first (red), then implement (green), then refactor\n3. **Consistency** — land a `conformance.md` mapping spec ↔ implementation ↔ tests\n4. **Gate** — every change must pass [`tools/gauntlet.sh`](tools/gauntlet.sh): build, vet, gofmt, staticcheck, tests with `-race`, coverage ≥ 60%, shuffle, and mutation testing\n\nSee [`AGENTS.md`](AGENTS.md) for the full development guide.\n\n## License\n\nApache License 2.0. See the [LICENSE](LICENSE) file for the full license text.\n\n---\n\n<p align=\"center\">\n<a href=\"#okf--open-knowledge-format\">⬆ Back to Top</a> &nbsp;•&nbsp; <a href=\"README.zh-CN.md\">🇨🇳 切换到中文</a>\n</p>\n",
  "bytes": 18800,
  "sha": "335d409fde4f91e700ddcdb1285ab1ccd8cf54ff878f8a7b9fce7f03dd13afad",
  "repo_slug": "superops-team/okf",
  "fonte": "repo",
  "truncated": false,
  "api": "https://agentalog.com/api/listings/okf_superops_team_okf_docs_knowledge_index_m_9b17c6c7/readme"
}