LLM observability
Bundle OKF 0.1 · 0 conceitos · xkazm04/ai-registry
Open source Repository Open in the app JSON README (API)
About
# LLM observability
The operator-side craft of running production LLM traffic as a product: telemetry
that stays auditable months later, price books and cost attribution, usage
governance, per-customer unit economics, continuous judge-scoring of live traces,
statistically defensible quality verdicts, and federated benchmark sharing that
leaks no one.
## Boundary contract with `software-engineering`
The `software-engineering` bundle's `llm-agent` category owns the **builder side** —
instrumenting and operating one's *own* agent system: `tracing` (span emission),
`eval-harness` (fixed offline datasets pre-ship), `cost-metering` (metering one's
own runs), `model-routing` (the runtime model decision), `structured-output`
(getting parseable JSON out). This bundle owns the **operator side**: receiving
traffic you did not emit, on clocks you do not own, from SDK versions you cannot
pin, for customers whose money flows both directions. The seam is stated per
subject in its golden path; no s
Details
- Kind
- OKF bundles
- Topic
- AI, RAG & memory
- Publisher
- xkazm04
- Origin
- okf_github
- Category
- dados
- Version
- 0.1
- Last push
- 2026-09-10T18:08:12Z
- Repository state
- ativo
- Language
- JavaScript
- License
- CC0-1.0
- Added
- 2026-09-09 05:09:53
- Updated
- 2026-09-09 05:09:53
- Origin id
xkazm04/ai-registry:knowledge/llm-observability/index.md
README
# ai-registry
An **AI development registry**: the knowledge, skills, practices and shared memory an
organization's agents run from, kept in git, owned by the organization, and reviewed like
code.
Start with the [workflow selection guide](docs/skill-selection.md) to use the library,
[CONTRIBUTING.md](CONTRIBUTING.md) to change it, or the
[RKB profile](docs/rkb-profile.md) for the format every knowledge bundle is gated
against. [AGENTS.md](AGENTS.md) gives agents a concise entry point. Use the [installation guide](docs/installations.md)
to select a harness and a live development or pinned release installation.
The repository carries eight lanes, declared in [`registry.yaml`](registry.yaml):
| Lane | Holds | Status |
| --- | --- | --- |
| [`knowledge/`](knowledge/README.md) | **Reference Knowledge Bundles** - four-layer domain knowledge (Golden Path → Technique → Application → Evidence), one bundle per domain. | Real content. Gated by CI. |
| [`skills/`](docs/skills-lane.md) | The fleet's shared skill library, one directory per skill, published as a **plugin marketplace** for the reference harness. | Real content; current inventory in catalog.json. Gated by CI. |
| [`recipes/`](recipes/README.md) | **Craftsman knowledge** - one kind of work done well, connector- and trigger-agnostic, versioned like a skill. Everything that binds a recipe to one installation lives on the adopted charter, never here. | Real content, all ten domains. Gated by CI. Declared 2026-09-06 with one worked example; the corpus migrated after operator approval and grows by assay. |
| `practices/` | Repo-level habits plus the starter artifacts they drop. | Templates and reusable practices; inspect each adoption contract. |
| `memory/` | Organizational memory notes, one fact per file. | Examples and dated observations; inspect scope and confidence. |
| [`usage/`](docs/usage-lane.md) | Which skills actually get used - counts contributed by the installations that run them, one file per contributor. | Real, gated. First contributor reporting. |
| [`signals/`](docs/signals-lane.md) | Whether the knowledge is still TRUE where it is used - stack versions, citation-resolution verdicts, deviations and consults, one file per contributor. | Real, gated. First contributor reporting (stack only, so far). |
| [`librarian/`](librarian/index.md) | Coverage memory for the maintenance loop - what was swept when, what was dispatched, what external sources were mined, and what was declined and why. | Real. Seeded by the founding sweep. |
The practices and memory lanes include worked examples and subsequently added material.
Read each artifact's scope and provenance before treating it as an adopted policy.
Lane depth is declared, not incidental. `knowledge/` and `recipes/` are `depth: nested` and cap
every level at ten folders, because their consumers read a GENERATED index rather than walking
the tree; `skills/`, `practices/`, `memory/`, `usage/` and `signals/` are `depth: fixed`
because a consumer's indexer selects their artifacts by exact path length - a category folder
there would not error, it would make every artifact silently vanish from the index.
## Why a repository
Agent instructions are code that runs against your codebase. They deserve the same treatment as
code: version control, review, an owner, and history.
- **Git is the door for content.** Every change to a skill, a practice or a memory note arrives as
a pull request. Merging is adopting - see [`CODEOWNERS`](CODEOWNERS).
- **Reading the library needs no account.** A developer with `git` and a text editor is a first-class
citizen. Clone it, read it, copy what you need.
- **Indexing is read-only.** A tool (here, [Ascent](.ascent/registry.yaml)) reads the tree, parses
the frontmatter, and keeps an index. It is never in the write path.
## Layout
```
README.md
registry.yaml # what this repository IS: its lanes, their specs and guarantees
.ascent/registry.yaml # Ascent's overlay: how ONE consumer indexes it (mode, telemetry, policies)
.claude-plugin/marketplace.json # GENERATED: the skills lane as a plugin marketplace, one plugin per skill
.claude/skills/ # the skills that maintain THIS registry: /forge /deepen /librarian /intake
CODEOWNERS # who merges = who adopts
catalog.json # GENERATED index: skills, practices, memory, bundles, hashes, adopters, counts
docs/rkb-profile.md # the knowledge lane's format spec (an OKF profile)
docs/skills-lane.md # the skills lane's format spec: shape, sub-resources, versions, distribution, resolution
docs/recipes-lane.md # the recipes lane's format spec: the v3 object, the rendered view, versions, dual improvement
docs/usage-lane.md # the usage lane's format spec + what may never go in it
docs/signals-lane.md # the signals lane's format spec: verdicts, never pointers
docs/reconcile-brief.md # the external-reconcile lane's contract, and how it extends past repositories
scripts/gate.mjs # runs the gate chain CI enforces, in CI order: --all, or --lane <lane>
scripts/check-bundles.mjs # the knowledge lane's gate (zero dependencies)
scripts/check-skills.mjs # the skills lane's gate: shape, sub-resources, the version-bump rule
scripts/check-recipes.mjs # the recipes lane's gate: the v3 object, the rendered view's coupling, depth, the version-bump rule
scripts/build-recipes-index.mjs # GENERATES recipes/index.json (--check in CI)
scripts/apply-skill-clauses.mjs # stamps the shared Skill Reflection / Knowledge sync clauses from docs/skill-clauses/ (--check in CI)
scripts/check-usage.mjs # the usage lane's gate: shape + the counts-only privacy rule
scripts/check-signals.mjs # the signals lane's gate: shape + the same privacy rule
scripts/check-currency.mjs# REPORTS how old the knowledge is; never fails a build
scripts/librarian-scan.mjs# REPORTS the maintenance scorecard; the instrument /librarian reads
scripts/research-ingest.mjs# normalizes an external source into an auditable transcript (/intake)
scripts/research-map.mjs # maps a claim's terms onto existing subjects: prior art, and where new goes
scripts/apply-taxonomy.mjs# the ONLY thing allowed to move a subject (moves + rewrites links)
scripts/lib/taxonomy.mjs # the shared slug -> path resolver; nothing else may build a subject path
scripts/lib/skills-lane.mjs # the ONE reader of the skills lane: frontmatter, digest, lessons, sub-resources
scripts/lib/fleet.mjs # the shared bulk-model dispatcher: retry, budget and model rotation
scripts/experiments/fleet-use-when.mjs # proposes the missing use_when lines, then applies the reviewed ones
scripts/build-index.mjs # regenerates knowledge/<domain>/index.json (--check in CI)
scripts/build-catalog.mjs # regenerates catalog.json from every lane (--check in CI)
scripts/build-marketplace.mjs # regenerates .claude-plugin/marketplace.json from the skills lane (--check in CI)
scripts/check-hash-stability.mjs # asserts the bundle digest is the same from a CRLF and an LF checkout
scripts/link-registry.mjs # OPERATOR-SIDE: links each project's .claude/skills + .claude/rules at this registry
scripts/build-knowledge-rules.mjs # regenerates rules/ - the always-on knowledge context projects link (--check in CI)
scripts/fleet-audit.mjs # OPERATOR-SIDE: which installation runs which copy of which skill; writes adopters
scripts/signals-collect.mjs # OPERATOR-SIDE: folds connected projects' consult logs + stacks into signals/
scripts/usage-from-personas.mjs # OPERATOR-SIDE: bootstraps usage/ from a Personas installation's own counts
knowledge/<domain>/ # a Reference Knowledge Bundle - see knowledge/README.md
knowledge/<domain>/taxonomy.json # the authority on where every subject lives; max 10 folders/level
knowledge/<domain>/index.json # GENERATED: every subject, technique, law and application
rules/ai-registry-*.md # GENERATED: the always-on knowledge context, linked into each project's .claude/rules/
skills/<name>/SKILL.md # frontmatter: name, description, category, memory, version (+ harness keys)
skills/<name>/LESSONS.md # append-only reflection lane, beside the skill it is about
skills/<name>/references/ # material the method loads on demand; scripts/, tools/, assets/ likewise
practices/<slug>/PRACTICE.md # frontmatter: id, dimension, applies-when; body = the shape
practices/<slug>/starter/** # templatized artifacts the practice drops into a repo
memory/<kind>/<slug>.md # frontmatter: kind, confidence, namespace, source
memory/_index.md # map of content over the notes
usage/<contributor>.json # counts from ONE installation - see docs/usage-lane.md
signals/<contributor>.json# currency verdicts from ONE installation - see docs/signals-lane.md
.projects.local.json # GITIGNORED: slug -> checkout path on this machine; the bridge the operator-side scripts read
```
Two `registry.yaml` files is deliberate, not drift: the root one says what this repository is,
the `.ascent/` one says how Ascent indexes it. A second consumer adds its own overlay; neither
rewrites the other, and a reader that knows only one of them still works.
## What is in here
### Knowledge bundles
| Bundle | Covers |
| --- | --- |
| [`software-engineering`](knowledge/software-engineering/) | Building and operating software: UI surfaces (including the published surfaces a product shows people who are not users yet), client architecture, LLM/agent engineering, backend platform (including the runtime for a graph of long-lived processes joined by typed channels, the semantics of data pipelines whose values carry their own frame and history, the contracts a packaged model and its workflow expose to hosts, and the internals of an embeddable language runtime - object shapes, engine strings, a tracing collector, register bytecode, guest execution ceilings, the host contract, native interop, standards-layered runtime crates and guest-language introspection), operations, security, secret custody and issuance (a server's own key hierarchy and seals, priced authority, dynamic-secret issuance and the roles that shape an issued artifact), integration, engineering process, and engineering assessment (measuring maturity, delivery and adoption). |
| [`media-generation`](knowledge/media-generation/) | Producing factual audiovisual content with generative models: narrative craft, research grounding, image generation and prompting, frame direction, production operations. |
| [`civic-intelligence`](knowledge/civic-intelligence/) | Watching public power with data: parliamentary records, legislation, public money, and the accountability methodology for publishing about real, named people. |
| [`grant-funding`](knowledge/grant-funding/) | Finding, winning and accounting for grant money: the funding landscape, eligibility and matching, proposal craft, and grant operations from deadline to post-award. |
| [`llm-observability`](knowledge/llm-observability/) | Operating production LLM traffic as a product: telemetry and cost attribution, price books and usage governance, unit economics, judge-scoring of live traces, and federated benchmark sharing. |
| [`game-production`](knowledge/game-production/) | Producing a game's systems and content at scale with machine assistance: systems canon and balance validation, the content pipeline and its acceptance ladder, generative asset production, engine integration, machine craft judgment, and production governance. |
| [`localization`](knowledge/localization/) | Language-specific craft, terminology, typography, register, and script/direction concerns, with product voice and termbases kept in the consuming project. |
| [`recruiting`](knowledge/recruiting/) | Hiring people with machine assistance and staying defensible: role definition and intake, candidate evidence and its provenance, interviews and work samples, automated screening and its fairness gates, pipeline operations, candidate experience, governance and consent, and honest measurement of a small-sample process. |
| [`marketing`](knowledge/marketing/) | Getting a small or mid-size business found, chosen and measured: search intent and keyword-metric honesty, site architecture and local pages that survive doorway filters, visibility in AI answers, content briefs, brand voice and channel-native social, paid-search triage, budget reallocation, search-term mining and ad craft, profit-aware measurement and diagnosis, zero-budget and local visibility, conversion and lead handling, and honest proof and positioning - with every threshold labelled as measured or convention. |
A bundle's two upper layers (Golden Path, Technique) carry **no** repo paths, file extensions or
product names - enforced by [`scripts/check-bundles.mjs`](scripts/check-bundles.mjs), not left to
discipline - so they transplant to any codebase unchanged. Applications are the opposite by
design: they cite real code and name their stack in the filename.
**Evidence is not published.** The pointers proving a claim against a particular tree are noise to
everyone else, so they live in each consumer's gitignored `<subject>/.evidence.local.md` overlay.
The gate fails any published file that declares them. Format spec:
[`docs/rkb-profile.md`](docs/rkb-profile.md).
**Read a bundle without reading 965 files.** Each bundle carries a generated
`index.json` - every subject with its category, status, techniques (and the laws they cite),
and applications. That is the file an agent selecting knowledge to consult should read; the
markdown is for humans and for the agent that decided to go deeper. It excludes evidence for
the reason above, and says so in its own `meta.excludes`. Regenerate with
`node scripts/build-index.mjs` **before** `build-catalog.mjs`, whose hash covers it.
**The corpus is present, not fetched.** `rules/ai-registry-*.md` are generated from the
bundle indexes and **linked into each project's `.claude/rules/`**, where a rule with no
`paths:` frontmatter loads in *every* session at `.claude/CLAUDE.md` priority. So an agent
opens a session already holding the access contract and the subject map for the domains its
project declares — no invocation, no copy, ~2k tokens. That closes the failure a
consult-only design has: recall. A standard nobody remembers to look up is a standard that
is not there at the moment the decision is made.
**And consulted deliberately when it matters.** The lane skill [`consult`](skills/consult/SKILL.md)
is how a connected project reads the right subject before a product, architecture or domain
call: it resolves the registry (a sibling checkout or GitHub), matches the task against
`use_when` triggers, reads the golden path and the techniques that apply, and logs the consult
to the project's gitignored `.ai/consults.jsonl`, which is what reaches the `signals/` lane as
a count. A project declares the bundles it consumes in its `.ai/manifest.yaml`
(`knowledge.domains`) and its agent guide says "run `/consult` before deciding".
**Knowledge has an age, and the registry cannot check it alone.** Every application
carries `verified_on` - the date its citations were last resolved against a real tree -
and [`scripts/check-currency.mjs`](scripts/check-currency.mjs) derives an expiry from it
per stack. That answers "how old is this claim". It cannot answer "is it still true",
because the registry does not have the consuming repository's checkout. That half arrives
from the other side, through [`signals/`](docs/signals-lane.md): the installation resolves
its own evidence overlay and reports **verdicts, never pointers** - `{"gone": 2}`, not
which two files. A bundle nobody reports on reads as **unknown**, never as current, for
the same reason `invokes30d: 0` with no contributors means nobody is looking.
**Some knowledge is correct but not yet.** A technique may declare a `stage` - `solo`,
`team`, `multi-service` or `fleet` - naming the rung at which it *starts to pay*
([`docs/rkb-profile.md` §3.2](docs/rkb-profile.md)). It is a floor, not a mandate: below it
the technique is over-engineering and a consumer is right to skip it; at or above it, its
absence is a gap. The field is optional and rare on purpose, and it is carried into each
bundle's `index.json` so a consumer can filter on it.
### Skills
Use the [workflow selection guide](docs/skill-selection.md) to choose a coordinating
skill and its supporting methods. The generated [catalog](catalog.json) carries the
current names, versions, hashes, and resources; versions are not duplicated here.
Registry maintenance methods live in [.claude/skills/](.claude/skills/): forge, deepen,
librarian, intake, reconcile, harvest, and assay. They maintain the library rather than
belonging to its portable skills lane. The [skills specification](docs/skills-lane.md)
defines layout, versioning, overlays, and distribution.
### Practices
| Practice | Dimension | Starter |
| --- | --- | --- |
| [`agent-guidance`](practices/agent-guidance/PRACTICE.md) | D1 | [`AGENTS.md`](practices/agent-guidance/starter/AGENTS.md) |
| [`local-first-readme`](practices/local-first-readme/PRACTICE.md) | D5 | [`README.md`](practices/local-first-readme/starter/README.md) |
| [`prior-art-teardown`](practices/prior-art-teardown/PRACTICE.md) | D5 | [`docs/prior-art/`](practices/prior-art-teardown/starter/docs/prior-art/EXAMPLE.md) |
| [`open-contribution`](practices/open-contribution/PRACTICE.md) | D5 | [`.github/` templates](practices/open-contribution/starter/.github/PULL_REQUEST_TEMPLATE.md), [`CODEOWNERS`](practices/open-contribution/starter/CODEOWNERS), [`CODE_OF_CONDUCT.md`](practices/open-contribution/starter/CODE_OF_CONDUCT.md) |
| [`supply-chain-security`](practices/supply-chain-security/PRACTICE.md) | D9 | [`SECURITY.md`](practices/supply-chain-security/starter/SECURITY.md), [`supply-chain.yml`](practices/supply-chain-security/starter/.github/workflows/supply-chain.yml) |
| [`least-powerful-test-first`](practices/least-powerful-test-first/PRACTICE.md) | D6 | Test-tier selection and a hermetic default test command. |
| [`justified-lint-exemptions`](practices/justified-lint-exemptions/PRACTICE.md) | D6 | Explicit reasons for lint exemptions. |
| [`self-declaring-spec-debt`](practices/self-declaring-spec-debt/PRACTICE.md) | D5 | Documents identify their known implementation gaps. |
A practice describes the **shape** of what good looks like, never a repo's actual content. Its
`starter/` files are templates full of `<...>` and `TODO:` markers: they scaffold, they do not
pretend to know your architecture. The starter workflow lives under `practices/` and does not run
in this repository - copy it into a target repo's `.github/workflows/` to use it.
### Memory
Notes and worked examples, indexed in [`memory/_index.md`](memory/_index.md): `semantic` (durable
facts), `procedural` (what worked), `episodic` (what happened, dated), `summary` (a rollup).
Confidence is a 0..1 float, banded as 1.0 verified, 0.6 probable, 0.3 a hunch.
## How a project consumes this registry
Plain git is the baseline. Nothing below requires an account or a token.
```sh
# read it
git clone https://github.com/xkazm04/ai-registry.git
# skills, door 1 - LINK (the default when one owner holds the registry and the consumers)
node ai-registry/scripts/link-registry.mjs # match every project to its manifest
node ai-registry/scripts/link-registry.mjs --check # verify; nothing to sync, so nothing drifts
# skills, door 2 - the plugin marketplace (a second machine, a second person, or CI)
claude plugin marketplace add xkazm04/ai-registry # once per machine
claude plugin install uat@ai-registry --scope project # records adoption in .claude/settings.json
# skills, door 3 - copy (the oldest model; still reviewed, still pinned by its version)
cp -r ai-registry/skills/ci-gate-check <your-repo>/.claude/skills/
# check what you have against what is current
cat ai-registry/catalog.json # name, version, contentHash, adopters per skill
```
A project points at the registry from its manifest, and names the bundles it consumes:
```yaml
# .ai/manifest.yaml
registry:
remote: github:xkazm04/ai-registry
local: ../ai-registry # the checkout this project links against
knowledge:
domains: [software-engineering] # which bundles it consumes -> which rules get linked
skills: # which shared skills it uses -> what link-registry links
- perfect
- uat
```
That manifest is the whole declaration. The links themselves are machine state and
gitignored — a link committed into a repo is a dangling path on the next machine — so
`scripts/link-registry.mjs` is what makes a machine match the declaration, and `--check`
is what tells you it no longer does.
**Which copy runs, when a name exists in more than one place.** The reference harness
resolves a same-named skill *enterprise over personal over project*; plugin skills are
namespaced and never collide. So the lane's rule is **one home per name**: a name that
lives in this lane lives nowhere else on an installation - not in `~/.claude/skills`, not
as a second project copy beside an enabled plugin. A higher version here does **not**
displace any copy; a version reports staleness, a human updates. Declared in
[`registry.yaml`](registry.yaml) under `lanes.skills.resolution`, explained in
[`docs/skills-lane.md`](docs/skills-lane.md), and checked - from the operator's machine,
because the registry cannot see installations - by
[`scripts/fleet-audit.mjs`](scripts/fleet-audit.mjs), which also writes what it finds into
the catalog's per-skill `adopters`.
## How a change gets in
This is the short form; the full contributor guide - per-lane gates, the privacy rule,
issue and AI-assistance expectations - is [`CONTRIBUTING.md`](CONTRIBUTING.md).
1. Branch, edit `skills/<name>/SKILL.md` (or anything in its directory), **bump `version`**.
2. Append an entry to that skill's `LESSONS.md`: `## <version used> - <YYYY-MM-DD> - <project>`
followed by `-` bullets. Record the version the run *used*, not the bump target.
3. `node scripts/check-skills.mjs && node scripts/build-marketplace.mjs && node scripts/build-catalog.mjs`
- the marketplace and the catalog are generated views and CI fails when they are stale.
4. Open a pull request. A `CODEOWNERS` owner reviews and merges - that merge is the adoption
decision.
5. Cached or copied installations update separately. Linked installations read the
working checkout immediately, including uncommitted changes. Review protects the
published history; it does not isolate consumers linked to a development checkout.
Version discipline: **versions are the comparison currency, hashes only detect drift.** Bump
minor or major when behaviour changes, patch when it does not - but bump. A checker cannot
tell a typo from a behaviour change, so `scripts/check-skills.mjs --since <ref>` asks for the
cheapest honest signal on every pull request that edits a skill, and rejects a version that
moves backwards. Appending to `LESSONS.md` needs no bump: a lesson records a run *against* a
version. Full rule in [`docs/skills-lane.md`](docs/skills-lane.md).
## Counts, witnesses and adopters
`invokes30d` in `catalog.json` is **derived** from the [`usage/`](docs/usage-lane.md) lane -
the first contributor is a Personas installation reporting its own 30-day counts. A zero with
an empty `usageContributors` means nobody is reporting on that skill, not that nobody runs it.
`adopters` is written by the operator-side `fleet-audit` from the installations it can see
(`<project>@<version>` for copies, `<project>@plugin:<version>` for enabled plugins); the
registry carries it forward and never invents it. Bundle currency reads from
[`signals/`](docs/signals-lane.md): a bundle with no reporting installation is **unknown**,
never current.
## Conventions
- LF line endings, no trailing whitespace.
- **ASCII where it bites.** `practices/` and `memory/` frontmatter is ASCII; prose is UTF-8: terse, templated files
that get pasted into terminals and `.claude/` directories. In `skills/`, frontmatter is ASCII
and fenced code may carry no lookalike punctuation (a Unicode dash or quote that reads as
ASCII and breaks a pasted command); prose is UTF-8. **`knowledge/` is UTF-8 prose**: OKF
requires valid UTF-8, and a bundle is long-form writing where an em dash is correct
typography. The lanes differ because their readers do.
- No secrets, ever - not in a file, not in an example, not in a test fixture. A tracked
credential is a hard failure and has to be rotated, not deleted.
- One idea per skill, per practice, per memory note.
- Vendor-neutral: name the capability (`Test: npm test`), not the tool.
## License
Public domain / [CC0 1.0](LICENSE). Copy anything here into your own registry and change
it to fit.