# Agentic Data Platform
Shared infrastructure (dlt · dbt · Dagster · DuckDB) plus a business-only project
per company. Sister companies run in parallel; Claude loads exactly one project's
context per session.
```
platform/ engines, ontology, knowledge graph, MCP, 14 skill toolkits ← you never edit
groups/<g>/ a family of sister companies: ontology instance, shared macros
projects/<p>/ one legal entity: sources, models, metrics ← you only edit here
vendor/ 25 upstreams pinned as submodules, with recorded provenance ← you never edit
data/_platform.duckdb tracking DB (agent runs, spend, monitors, impact history)
```
New here? [docs/ONBOARDING.md](https://github.com/atomz-org/data-platform/blob/HEAD/docs/ONBOARDING.md) is the whole repository as a
newcomer needs it — the tiers, one company's data end to end, and the command
sequence that brings a new company in. Generated by `pf guide build` and
checked on every pull request, so it cannot go stale.
## Quick start
```bash
uv sync
uv run pf ui # control plane → http://127.0.0.1:8787
uv run pf status # every group and project
```
## Daily commands
| Command | What it does |
|---|---|
| `pf new-group <g> --domain b2b_saas` | New company group (ontology instance, shared dbt package) |
| `pf new-project <g> <p> --plan` | Resolve a scaffold and write nothing — **run this first** ([docs](https://github.com/atomz-org/data-platform/blob/HEAD/docs/SCAFFOLDING.md)) |
| `pf new-project <g> <p>` | New entity inside a group |
| `pf new-project <g> <g>-rollup --rollup --sisters a,b` | Cross-entity roll-up project |
| `pf work <g> <p>` | Launch Claude scoped to exactly one project |
| `pf seed <g> <p>` | dlt → DuckDB → annotations → monitors → dbt → graph → card |
| `pf run-all <g>` | Every sister in parallel, then the roll-up |
| `pf impact <g> <p> <node>` | Blast radius. Exits 1 on breaking. **The merge gate.** |
| `pf atlas <g> <p>` | Publish a project's own atlas beside its graph ([docs](https://github.com/atomz-org/data-platform/blob/HEAD/docs/PROJECT-ATLAS.md)) |
| `pf arch <g> <p>` | Per-project architecture map — every feature, present or not ([docs](https://github.com/atomz-org/data-platform/blob/HEAD/docs/ARCHITECTURE-MAPS.md)) |
| `pf guide build` / `check` | The onboarding guide, generated from the repository ([docs](https://github.com/atomz-org/data-platform/blob/HEAD/docs/ONBOARDING.md)) |
| `pf harness <g> [<p>]` / `build` / `check` | The harness map of a group, a project and its report: what wraps an agent there, computed from the files that enforce it |
| `pf check` | Ontology conformance across every project |
| `pf group list/show/verify` | Tenant families: lifecycle, owner, readiness |
| `pf group set-state <g> <state>` | Move a family through its lifecycle |
| `pf offboard <g>` | What leaving costs. Enumerates; removes only with `--apply` |
| `pf tokens` | Always-on token budget; fails if a card is over |
| `pf kg build/card/search/neighbors` | Knowledge graph operations |
| `pf vendor list/sync/drift/verify/why` | Vendored upstreams and their provenance |
| `pf workflow link/list/watch/show` | Claude Code workflow runs, written into `logs/workflows/` as they happen |
| `pf housekeeping [g p] [--apply]` | Maintenance the engines skip: lakehouse compaction and snapshot expiry per project, Dagster history and stale artefacts platform-wide |
| `pf pr report` | Blast radius, conformance and drift for the current change |
| `pf ui` / `pf mcp` | Dashboard / MCP server |
## Developing on the platform
`uv run pf install-hook` on a fresh clone — git does not clone `.git/hooks`,
so both gates, `pre-commit` over what you staged and `pre-push` over what you
committed, are otherwise silently absent. What
[`platform-tests.yml`](https://github.com/atomz-org/data-platform/blob/HEAD/.github/workflows/platform-tests.yml) runs on every
`platform/**` pull request: `ruff check platform/`, `pf test check`,
`pf arch check`, `pytest platform/tests -q`. Full commands, the generated-file
table (what regenerates each committed artefact, and what to edit instead),
the traps in verifying one locally, and the commit gate's `maxFiles` and
denylists: [`docs/DEVELOPMENT.md`](https://github.com/atomz-org/data-platform/blob/HEAD/docs/DEVELOPMENT.md). What each of the
twenty-three workflows asks, which ones your change wakes, and the local command
behind every check: [`docs/CI.md`](https://github.com/atomz-org/data-platform/blob/HEAD/docs/CI.md).
## What you write vs never touch
| You write, per project | You never touch |
|---|---|
| `src/<p>/sources/*.py` — annotated dlt sources | `platform/**` (engines, factories, toolkits, MCP) |
| `transform/models/**` — staging, intermediate, marts, semantic | Dagster wiring, profiles, executors, pools |
| `CLAUDE.md` — rules the graph can't encode | `kg/*` — cards and graphs are generated |
| `groups/<g>/ontology/instance.yaml` | CI, engine pins, extension set |
| `groups/<g>/.claude/skills/` — domain skills every sister loads | `STATE.md`, `loop-ledger.json` — written by `pf loop` |
| `groups/<g>/loops.yaml` — which loops watch the family, with waivers | |
| `.memory/notes/` — one lesson per file, for the next agent of any tool (`pf memory add`) | `.memory/MEMORY.md` — the index, generated by `pf memory index` |
Every coding agent — Claude Code, Copilot, Gemini, Codex, a local model — works
to one protocol, [AGENTS.md](https://github.com/atomz-org/data-platform/blob/HEAD/AGENTS.md), whose rules go by how the agent was
invoked (session, autonomous, inline) rather than by vendor.
Adding a sister company is `pf new-project`, sources and models. Upgrading dbt for
every entity is one line in `pyproject.toml`. Plan the scaffold before applying
it — `--plan` resolves the capability set, the gate rules and the CI jobs without
writing, and refuses on the two mistakes that actually happen: a group that does
not exist, and a directory already taken. See [docs/SCAFFOLDING.md](https://github.com/atomz-org/data-platform/blob/HEAD/docs/SCAFFOLDING.md).
## The three mechanisms that make it work
**Ontology → everything.** Every dlt resource declares a concept, column roles and
links. Those annotations drive staging generation, PII policy, statistical
monitors, metric candidates, graph edges and conformance checks. `pf check` fails
on an unannotated or non-conforming source.
**Knowledge graph → retrieval, not grep.** `kg_search` / `kg_neighbors` / `kg_path`
let an agent route before it reads. The generated context card (~390 tokens) is the
always-in-context index; the full graph is queried on demand. `kg/architecture.md`
(~2,000 tokens, on demand) is the map of how one project is *built* — every
feature it has and every feature it lacks, generated from a registry so a feature
cannot be quietly left out. See [docs/ARCHITECTURE-MAPS.md](https://github.com/atomz-org/data-platform/blob/HEAD/docs/ARCHITECTURE-MAPS.md).
**Impact analysis → safety.** `pf impact` walks the graph downstream and names
every model, metric, dimension and exposure affected — plus the exposure owner.
Wire it into CI with `pf impact-gate`.
## The layer contract
dlt extracts; dbt transforms; the raw dataset is the line between them. dlt Core
lands one dataset per source with types frozen and row counts read back, and it
never transforms. dbt owns everything after: staging is generated 1:1 from the
annotations (`pf gen-staging`), currency, unit and FX conversion happen once in
`intermediate` or a mart, marts declare the grain, the semantic layer declares
the metrics, and reporting is a projection of them. The why, and commodity as
the worked example: [`docs/ENGINEERING.md`](https://github.com/atomz-org/data-platform/blob/HEAD/docs/ENGINEERING.md).
## Sister companies run genuinely in parallel
One DuckDB file per project (DuckDB's single-writer lock is per file), one Dagster
writer pool per project, and roll-ups that `ATTACH … (READ_ONLY)`. Sisters share
the group ontology, so `fct_revenue` has the same grain in both and the union is
safe by construction.
## Token budget
Constant regardless of how many groups, sisters or skills exist:
| Artefact | Budget | Actual |
|---|---:|---:|
| Project context card | 1500 | ~393 |
| Project CLAUDE.md | 600 | ~149 |
| Group card | 400 | ~136 |
| ROUTING.md | 500 | ~399 |
| **Session preamble** | | **~1000** |
`pf tokens` records these to the tracking DB and fails when one goes over.
## Adding a capability
Post-scaffold work lives in **one** ordered, idempotent list — `pf.scaffold.bootstrap.STEPS`.
`pf new-project` runs it; `pf bootstrap --all` re-runs it over every existing
project. Adding a platform capability means adding a step there, and both new and
old projects pick it up.
```bash
uv run pf bootstrap-steps # what runs, and why each step exists
uv run pf bootstrap <g> <p> # retrofit one project
uv run pf bootstrap --all # retrofit everything after a platform upgrade
```
This exists because the steps used to be inlined in `new-project`: every
capability added afterwards — the PreToolUse hook, the dbt macro-paths, the
placeholder card — had to be hand-patched across existing projects, and each was
a silent hole until someone noticed.
## Loop engineering
Building blocks adapted from [loop-engineering](https://github.com/atomz-org/data-platform/blob/HEAD/vendor/loop-engineering) (added as
a submodule): durable state, path gate, token budget, circuit breaker, ledger and
a readiness score. The loops themselves watch data-platform subjects.
```bash
uv run pf loop list # loops, autonomy levels, budgets
uv run pf loop run-all acme acme-us # every L1 loop, then rewrite STATE.md
uv run pf loop audit # Loop Readiness Score (0-100)
uv run pf loop status # ledger
```
Governance files: `LOOP.md` (definitions) · `STATE.md` (generated) ·
`loop-constraints.md` (binding) · `gate.yaml` (path policy) · `loop-budget.md`.
A group overrides the registry in `groups/<g>/loops.yaml` (cadence, budget, lower-only autonomy, waivers with a reason); `pf loop list --group <g>` shows the result.
**Three enforcement gates** make the impact check structural rather than a rule
an agent must remember:
| Gate | Fires | Effect |
|---|---|---|
| `pf check` | on demand / CI | blast radius of every working-tree change |
| pre-commit hook | `git commit` | blocks a breaking change; `--no-verify` to override with a reason |
| PreToolUse hook | before any Edit/Write | denies secrets and generated files; prints blast radius for models |
## Skill toolkits
14 plugins in `platform/toolkits`, referenced by every project and copied into
none. Ported from dltHub AI Workbench, dbt-labs/dbt-agent-skills,
duckdb/duckdb-skills and dagster-io/skills, adapted for dlt Core / dbt Core.
`ROUTING.md` resolves their overlaps: graph → metrics → ad-hoc SQL, in that order.
## Vendored upstreams
Twenty-three upstreams are pinned as branch-tracked submodules under `vendor/`,
and every borrowing is written down as a **path pair** — one file upstream, one
or more of ours — in `platform/src/pf/vendor/registry.yaml`: 24 entries in all
(one, `floci`, is registered but its submodule was never added — `pf vendor
verify` reports it `missing`), 88 adoptions, 51 explicit declines. Full
account: [`docs/VENDOR.md`](https://github.com/atomz-org/data-platform/blob/HEAD/docs/VENDOR.md); the agent-loadable index is
[`docs/VENDOR-CARD.md`](https://github.com/atomz-org/data-platform/blob/HEAD/docs/VENDOR-CARD.md) (~924 tokens). Both are generated
from the registry.
| Command | What it does |
|---|---|
| `pf vendor list [-v]` | Every upstream, licence, pinned commit, reviewed state |
| `pf vendor sync` | Fetch each tracking branch, then report what it means for us |
| `pf vendor drift` | What moved since a human last looked — local, never fetches |
| `pf vendor approve <id>` | Record the current checkout as reviewed |
| `pf vendor verify` | Declared paths still exist; schema contracts still hold |
| `pf vendor why <file>` | Where a file of ours came from, and what we changed |
| `pf vendor licences` | Licences, and the two that constrain how this ships |
**`vendor.lock.json` is not the submodule pointer.** The gitlink records which
commit is checked out; the lock records which commit a person *reviewed*, plus a
git OID per adopted path. Those diverge the moment someone runs
`--remote` without reading anything, and that gap is what makes drift observable.
An upstream that moved forty commits without touching anything we adopted is a
free fast-forward; one that touched `core/wren-mdl/mdl.schema.json` names
`projections/mdl.py` and stops.
Three of the pins are **executable contracts**, checked by `pf vendor verify`:
| Vendored file | Checks |
|---|---|
| `opentopology/schemas/otop-core-v0.2.schema.json` | `pf semantic otop` output |
| `wrenai/core/wren-mdl/mdl.schema.json` | `pf semantic mdl` output |
| `loop-engineering/patterns/registry.schema.json` | field parity with `LoopSpec` |
Sync runs weekly in CI and opens a PR; it never approves, because approval means
a human read the diff.
## Pull requests
`pf pr report` assembles what a reviewer would otherwise rebuild by hand: per
project, the blast radius with exposure owners, conformance and readiness;
repo-wide, path-gate verdicts and vendor drift; then one verdict — **block /
review / clear**. `clear` means nothing downstream breaks and no gate fired. It
is not an approval.
The report is written to `data/pr/<n>.json` and posted as a single PR comment
that is edited in place. `pf ui` reads the same JSON, so the dashboard shows the
verdict CI computed rather than a second opinion.
## License
Apache License 2.0 — see [LICENSE](https://github.com/atomz-org/data-platform/blob/HEAD/LICENSE). Attribution for adapted
third-party material is in [NOTICE](https://github.com/atomz-org/data-platform/blob/HEAD/NOTICE) and
[THIRD-PARTY-NOTICES.md](https://github.com/atomz-org/data-platform/blob/HEAD/THIRD-PARTY-NOTICES.md); the per-path provenance
record is `platform/src/pf/vendor/registry.yaml` (`pf vendor why <file>`).
Upstreams under `vendor/` are submodule pointers and keep their own licenses.
[COMMERCIAL.md](https://github.com/atomz-org/data-platform/blob/HEAD/COMMERCIAL.md) reads all of that back for someone deciding
whether they can ship something built on this: Apache 2.0 grants commercial use
already, and the care is owed to three pinned upstreams, not to this licence.
Contributions are welcome — see [CONTRIBUTING.md](https://github.com/atomz-org/data-platform/blob/HEAD/CONTRIBUTING.md). Security
reports go through [SECURITY.md](https://github.com/atomz-org/data-platform/blob/HEAD/SECURITY.md), not the issue tracker.
