{
  "markdown": "# Lineage Collector\n\nA locally runnable, production-shaped implementation of evidence-first lineage collection. It accepts signed repository events, runs deterministic Baseline and Incremental collection, validates optional runtime evidence, creates and reviews proposals, publishes with fencing, handles exact-artifact deployment promotion, evaluates a bounded read-only PR gate, and serves version-pinned lineage and impact queries.\n\nThe default operator path remains local: FastAPI + SQLite + a write-once object directory on the backend, and React + TypeScript + Vite on the frontend. The same repository also contains nine independently addressable Lambda handlers, an SCA Fargate worker, four generated Step Functions workflows, concrete AWS adapters, CDK stacks, deterministic OCI packaging, and guarded ephemeral-AWS verification. Local use does not require AWS credentials or an LLM key.\n\n## Start here\n\nNew to the repository, or looking for a specific file or command? **[docs/NAVIGATION.md](docs/NAVIGATION.md)** is the guided map: what each workspace owns, where a given concept lives, the full command reference, and a troubleshooting table.\n\nEach application also documents itself:\n\n| Application | What it is | README |\n|---|---|---|\n| `apps/api` | Collection engine and product API — Python, FastAPI, SQLite | [apps/api/README.md](apps/api/README.md) |\n| `apps/web` | Operator control room — React, TypeScript, Vite | [apps/web/README.md](apps/web/README.md) |\n| `infra` | CDK stacks, OCI packaging, generated Step Functions | [infra/README.md](infra/README.md) |\n\n## Prerequisites\n\n- Python 3.12 or 3.13\n- [uv](https://docs.astral.sh/uv/)\n- Node.js 20 or newer with npm\n- Docker Desktop only for `make package-aws`\n- AWS CLI v2 and an approved AWS account only for the explicit ephemeral-AWS flow\n\n## Local setup\n\n```bash\nmake setup\n```\n\nThis installs the locked Python environment from `apps/api/uv.lock` and JavaScript dependencies from `package-lock.json`.\n\n## Run locally\n\n```bash\nmake dev\n```\n\nOpen the UI at [http://127.0.0.1:5173](http://127.0.0.1:5173). The API health endpoint is [http://127.0.0.1:8000/healthz](http://127.0.0.1:8000/healthz), and interactive API documentation is at [http://127.0.0.1:8000/docs](http://127.0.0.1:8000/docs).\n\nGenerated state is written under `data/` and is ignored by Git. Stop both processes with `Ctrl-C`.\n\nIf port 8000 or 5173 is already in use, move the whole stack — the web proxy follows the API port automatically:\n\n```bash\nLINEAGE_API_PORT=8021 LINEAGE_WEB_PORT=5181 make dev\n```\n\nTo drain durable local work without the web process, run:\n\n```bash\nuv run --project apps/api python -m lineage_api.cli worker --drain --max-messages 100\n```\n\nUse `--once` instead of `--drain` to process at most one available command.\n\n## Secrets and access\n\n`make dev` and `make reset` need no configuration: both set `LINEAGE_DEV_MODE=1`, which permits the demo signing secret published in this repository.\n\nAnything else must supply its own. `LINEAGE_WEBHOOK_SECRET` authenticates **every** signed input the platform accepts — push deliveries, deployment outcomes, and, through a derived key, runtime observations. Resolution therefore fails closed: a missing secret, the published demo value, or anything shorter than 16 bytes is refused rather than silently falling back to a value that anyone who can read this repository already knows.\n\n```bash\nexport LINEAGE_WEBHOOK_SECRET=a-private-value-at-least-16-bytes\n```\n\n`LINEAGE_API_TOKEN` is unset by default, which leaves the API open — correct for a single-operator local run and what the demo walkthrough expects. Set it for any instance reachable by more than one person, and every route except `/healthz` will require `Authorization: Bearer <token>`. It gates reads as well as writes, because the lineage graph discloses the estate's schema and topology.\n\nThe React UI does not yet send a bearer token, so leave `LINEAGE_API_TOKEN` unset when using the UI, and use it for headless or deployed access.\n\n## Collect an exact Java/Spring checkout\n\n`collect-checkout` analyzes canonical committed Git blobs without running Maven, Gradle, tests,\napplication code, hooks, or repository executables. It verifies the credential-free origin, exact\nrevision, index/tree identity, bounded tracked scope and immutable scope digest before creating a\nsigned delivery. The signed source determinant also pins a digest of the analyzer's complete path\ndisposition, so retry cannot silently select a different subset. That delivery then follows the\nnormal SQLite outbox, queue, lease, I1–I10 stage ledger, evidence, consolidation and review path.\nSuccessful collection stops at `IN_REVIEW`; it does not approve or publish.\n\n```bash\nexport LINEAGE_DATA_DIR=/tmp/lineage-real-state\nexport LINEAGE_WEBHOOK_SECRET=change-me-local-only\n\nuv run --project apps/api python -m lineage_api.cli collect-checkout \\\n  --checkout /absolute/path/to/spring-service \\\n  --origin https://github.com/acme/spring-service \\\n  --revision <exact-40-or-64-character-lowercase-commit> \\\n  --repository spring-service \\\n  --environment staging \\\n  --platform postgres \\\n  --system orders \\\n  --analyzer-pack java-spring-data-jpa-v1 \\\n  --ruleset spring-data-rules-v1 \\\n  --profile postgres\n```\n\nThe v1 pack is repository-neutral. It reconciles literal root Maven and Gradle Spring Boot/Data JPA\ncells, analyzes production Java under `src/main/java`, and selects exactly one\nrepository-relative `src/main/resources/db/<profile>/schema.sql`. Conflicting or dynamic build\ncells and missing or multiple trusted schema candidates return `INTEGRATION_REQUIRED`. Alternate\nmodule, fixture, test, seed-data, user and setup paths are outside this static production scope.\nThe H2 profile is deliberately approximation-only and returns\n`INTEGRATION_REQUIRED`; use a trusted PostgreSQL or MySQL schema profile to authorize tables.\n\nCoverage keeps the full tracked snapshot in `expectedScope`. Only root build cells, production\nJava and the exact trusted profile schema appear in `completedScope`. README/test Java/data/setup/\nuser/alternate schema and nested-build paths are recorded in `skippedScope`; relevant Java or SQL\nthat cannot be safely classified is recorded in `unsupportedScope` and blocks completion. Every\ntracked path must occur in exactly one disposition, and `COMPLETE` is impossible with unsupported\nor failed paths.\n\nThe bounded JSON result contains only identifiers, digests, status, stage names and counts. Static\n`exact=true` means the source citation is exact, not that the operation ran. By default, runtime verification is off and `runtimeStatus` remains `NOT_PROVIDED` (with `runtimeReasons == [\"not-requested\"]`). Enable it per collection with `--runtime-verification` on `collect-checkout`, or `\"runtimeVerification\": true` on `POST /api/collections`; both reach `RepositoryCollectionService.collect(..., runtime_execution=True)`, which executes the Python and Java runtime stages against the SCA edges. A successful run reports `runtimeStatus == \"CORROBORATED\"` with empty `runtimeReasons`.\n\nSpring Petclinic is an acceptance example, not a special case in production code:\n\n```bash\ngit clone https://github.com/spring-projects/spring-petclinic.git /tmp/spring-petclinic\ngit -C /tmp/spring-petclinic checkout --detach 88e37c15cf6fc8490b01bc3e8e2c800cec1ac272\n\nuv run --project apps/api python -m lineage_api.cli collect-checkout \\\n  --checkout /tmp/spring-petclinic \\\n  --origin https://github.com/spring-projects/spring-petclinic \\\n  --revision 88e37c15cf6fc8490b01bc3e8e2c800cec1ac272 \\\n  --repository spring-petclinic \\\n  --environment staging \\\n  --platform postgres \\\n  --system petclinic \\\n  --analyzer-pack java-spring-data-jpa-v1 \\\n  --ruleset spring-data-rules-v1 \\\n  --profile postgres\n```\n\nAt that pinned revision the accepted oracle is 23 static edges (18 reads and 5 writes -- 15\ndataset-scope plus 8 element-scope `dataset#column` edges), zero\nunresolved invocations, proposal `IN_REVIEW`, and runtime `NOT_PROVIDED`. Repeating the identical\ncommand targets that exact durable command: queued or redrivable work resumes through the same\nstage ledger, while a completed command returns `DUPLICATE` with the same command, run and proposal\nand no extra ledger effects. A completed `INTEGRATION_REQUIRED` command remains non-successful on\nrepeat rather than being relabeled as a successful duplicate.\n\nTo drive the same pinned revision through the product API instead of the CLI, run the collection\nproduct-flow proof:\n\n```bash\nLINEAGE_REAL_REPOSITORY_CHECKOUT=/path/to/spring-petclinic \\\n  uv run --project apps/api --extra dev pytest \\\n  tests/integration/test_repository_collection_product_flow.py -q\n```\n\nIt submits the checkout through `POST /api/collections`, polls `GET /api/collections/{commandId}`,\nand requires the 23/18/5 oracle, the complete 131-path disposition, `NOT_PROVIDED` runtime status,\nand a duplicate submission with byte-identical database and evidence digests.\n\nTo run the reproducible compatibility proof rather than the one-shot operator command, use the\ndedicated opt-in acceptance runner:\n\n```bash\nexport LINEAGE_REAL_REPOSITORY_CHECKOUT=/tmp/spring-petclinic\nexport LINEAGE_ACCEPTANCE_RUN_ID=spring-petclinic-local\n./scripts/run_real_repository_acceptance.sh\n```\n\nThe runner verifies the official origin and exact revision, composes two fresh local SQLite/object\nstates, runs the actual durable `collect-checkout` flow in each, repeats one delivery to prove\nduplicate no-effect, and requires the complete 131-path disposition and exact 23-edge oracle. The\nduplicate proof opens SQLite read-only with `query_only`, holds one transaction across every read,\nand dynamically snapshots every user table (currently 34). The bounded canonical proof includes\nevery `sqlite_schema` table/index/view/trigger record, `table_xinfo`, foreign keys, index metadata,\nand every typed value; same-row-count data or schema changes cannot escape detection. The retained\nlogical database digest normalizes only the documented time/lease values and time-derived references\nneeded for fresh-run reproducibility.\n\nThe shell clears the inherited environment and invokes the already provisioned\n`apps/api/.venv/bin/python` directly in isolated mode. A standard-library supervisor enforces a\nsecret-free allowlist, a wall timeout, process-group termination, stdout/stderr caps, and the exact\nPASS JSON schema. No package manager is invoked and the runner neither resolves nor downloads\ndependencies. If the locked environment is absent or unsafe, the result is bounded\n`INTEGRATION_REQUIRED`, never an install. The runner does not build or execute Petclinic. On success\nit prints bounded JSON and publishes canonical evidence with no-follow, directory-descriptor-anchored\nwrite-once operations at\n`data/acceptance/spring-petclinic-local/java-spring/sha256-<checksum>.json`. The manifest contains no\ncheckout path, source bytes, raw query, credential, or timestamp. Missing checkout, a wrong\norigin/revision, incomplete scope, tampering, or oracle drift exits `2` and cannot report a pass.\nPetclinic constants live only in this acceptance oracle; the production analyzer remains generic.\n\n## Reset the deterministic demo\n\n```bash\nmake reset\n```\n\nReset removes only the generated local object directory, clears the local control tables, reloads the checked-in catalog snapshot, and restores the active `staging` pointer to `v1`. The same operation is available while the API is running through `POST /api/demo/reset`.\n\n## Demo walkthrough\n\n1. Open **Operations** and select **Run seeded collection**. The backend resets state, signs the seeded delivery, verifies it, and processes `payments-pipeline` to `IN_REVIEW`.\n2. Open the resulting **Run timeline**. Confirm the ordered stages `QUEUED` through `IN_REVIEW`, the shared correlation ID, and SCA/runtime evidence references.\n3. Open **Review queue**, select the payments proposal, and inspect the three added edges. Confidence (`HIGH`) and corroboration (`ELEMENT`) are deliberately separate; SCA citations and evidence checksums are visible.\n4. Enter a rationale and choose **Approve and publish**. The manifest is written immutably, `v2` is staged and verified, and the active pointer advances with fencing token `1`.\n5. Open **Lineage explorer**. Select an edge with the keyboard or pointer to inspect its mechanisms, citation, and checksum.\n6. Run a `COLUMN_DROP` impact analysis. High-confidence downstream edges produce a `BLOCK` verdict. Change direction or depth to exercise bounded traversal.\n7. Running the same signed delivery again returns `DUPLICATE` and creates no second run.\n\n## Test and build\n\n```bash\nmake test\nmake build\nmake verify\nmake synth\nmake package-aws\n```\n\n- `make test` runs the backend domain/API/walking-skeleton suite and the frontend component suite.\n- `make build` performs strict TypeScript compilation and a Vite production build.\n- `make verify` runs both commands as the local pre-handoff gate. The documented walkthrough was also exercised in a real Chromium session for the prototype handoff.\n- `make synth` creates the no-credential fixture CDK assembly under `infra/cdk.out/`.\n- `make package-aws` builds the Python wheel plus `linux/amd64` Lambda and `linux/arm64` SCA OCI archives. Verify exact image/archive digests and handler inventory in `infra/dist/runtime-build-metadata.json` and `infra/dist/package-manifest.json`.\n\n## Workflow trigger table\n\nTrigger choice is versioned policy. Receipt is acknowledged only after the canonical event and its durable command or no-impact decision exist.\n\n| Flow | Trigger it | Do not trigger it |\n|---|---|---|\n| Baseline B1–B10 | Repository/system onboarding, missing trusted base, explicit full rebaseline, or a major unsupported determinant/schema change | Every push, PR update, merge, or deployment |\n| Incremental I1–I10 | Observed-branch push, affected ruleset/resolver/policy change, accepted correction, late validated runtime evidence, or missing-package remediation | Unchanged paths/determinants with a complete no-impact proof |\n| PRGate P1–P8 | PR open/synchronize/reopen, target-environment change, manual rerun, or active-pointer refresh | Merge, deployment, Nightly, LLM completion, or runtime-session close |\n| Deployment D1–D6 | Canonical succeeded, failed, or rollback deployment outcome with exact artifact digest and authoritative provider ordering | Source merge alone |\n| Nightly N1–N6 | Reconciliation schedule, drift/rebuild sample, bounded stale derivation, or manifest/cache audit | User-facing interactive requests |\n\nPRGate is read-only except for its check/audit record. It rechecks both the PR head and environment pointer; incomplete, stale, degraded, truncated, or timed-out analysis returns `WARN`, never a false `PASS`.\n\n## Local-to-AWS mapping\n\nOne Python package is reused, but deployment boundaries are explicit and independently scalable.\n\n| Concern | Local adapter | AWS adapter/deployment |\n|---|---|---|\n| Receipt, command, lease, stage ledger | SQLite transactions and simulated clock | DynamoDB conditional writes and lease epochs |\n| Queue lanes | SQLite broker with FIFO groups and deterministic priority | Interactive/events FIFO SQS plus bounded batch SQS, DLQs, partial-batch redrive and reserved headroom |\n| Workflow | Python application definitions | Four versioned Standard Step Functions aliases; Deployment remains one D1–D6 Lambda |\n| Static analysis | Local Python SCA module | Callback-token ARM64 Fargate task with heartbeat |\n| Immutable evidence/packages | Write-once checksummed files | Versioned, encrypted S3; production adds Object Lock and cross-region replication |\n| Runtime observations | Closed-schema local session store | Kinesis partitioning plus the same OpenLineage/custom SDK/OTel validation contracts |\n| Projection and pointer | Versioned SQLite graph and conditional pointer | Neptune idempotent merge plus DynamoDB fenced pointer |\n| Product/API | Local FastAPI and React | Versioned Lambda image targets, API Gateway and canary aliases |\n\nThe local implementation proves behavior and recovery. CDK assertions and offline synth prove topology. A real AWS deployment, canary, scale window and recovery drill remain separate evidence gates.\n\n## Target AWS architecture\n\nThe canonical production target — with no local mapping shown as an architecture component — lives in\n[`docs/architecture/lineage-platform-target.md`](docs/architecture/lineage-platform-target.md). The\nMermaid block in that file is normative; the offline browser-ready rendering at\n[`docs/architecture/lineage-platform-target.html`](docs/architecture/lineage-platform-target.html) is\ngenerated from it and must never be edited by hand.\n\n```bash\nmake architecture\nmake architecture-check\n```\n\n`make architecture-check` fails when the rendering is stale. The same parity check, the layer/status\nlegend, the arrow semantics, and the evidence boundaries are asserted by\n`tests/test_documentation.py`.\n\n## Generated workflow safety\n\n`apps/api/src/lineage_api/application/workflows/definitions.py` is the workflow authority. The checked-in files under `infra/workflows/` are deterministic exports; do not edit ASL or the generated contract by hand.\n\n```bash\nmake workflow-export\nmake workflow-check\n```\n\n`make workflow-check` fails if B1–B10, I1–I10, P1–P8, N1–N6, D1–D6, timeouts, attempts, terminals, or generated ASL drift from the Python definitions. `data/`, `apps/web/dist/`, `infra/dist/`, and `infra/cdk.out/` are generated outputs and may be recreated. Do not treat them as authoritative source or commit them.\n\n## Fault injection and acceptance evidence\n\nRun the deterministic local acceptance gate:\n\n```bash\nmake acceptance-smoke\n```\n\nIt writes content-addressed, schema-valid `AcceptanceEvidenceManifest` records under a fresh gitignored `data/acceptance/<run>/` directory. The current local suite exercises named crash/redrive, checksum equivalence, lane headroom/fairness, and evidence integrity. AWS-only 100/s, 10k burst, 10k-repository/12-hour, availability and DR rows are emitted as `AWS_REQUIRED`, not `PASS`.\n\nThe default command is hermetic and reports `HERMITIC_LOCAL_PASS` or failure independently. Because\nan external repository is not implicit test input, it also reports\n`LOCAL_REAL_REPOSITORY_REQUIRED`; that status is not counted as a pass. Set\n`LINEAGE_REAL_REPOSITORY_CHECKOUT` to the exact pinned Petclinic checkout before running\n`make acceptance-smoke` to add the separate `LOCAL_REAL_REPOSITORY_PASS` proof. This static proof defaults to `RUNTIME_NOT_PROVIDED` (no runtime verification). `make acceptance-smoke` does not enable runtime verification; the CLI and the collections API both do expose it, via `--runtime-verification` and `\"runtimeVerification\": true`. Live cloud rows remain `AWS_REQUIRED`, and collection defaults to runtime off.\n\nRun only the named crash/redrive scenario with:\n\n```bash\nuv run --project apps/api --extra dev pytest -q tests/acceptance/test_replay_and_faults.py\n```\n\nThe acceptance runner returns nonzero when a scenario or recorded manifest is `FAIL`.\n\n## AWS ephemeral verification\n\nNo AWS command runs implicitly. `scripts/deploy_ephemeral_aws.sh` requires explicit account, primary/recovery regions, three availability zones, a `lineage-e2e-*` namespace, approved PrivateLink and paging values, a clean commit, and `ALLOW_LINEAGE_EPHEMERAL_AWS=1`. It packages exact OCI digests, creates repositories first, pushes those digests, then deploys the remaining stacks.\n\nSet all values from approved enterprise/account context; the angle-bracket values below are labels,\nnot defaults:\n\n```bash\nexport AWS_PROFILE=<approved-profile>\nexport AWS_ACCOUNT_ID=<approved-12-digit-account>\nexport AWS_REGION=<approved-primary-region>\nexport LINEAGE_SECONDARY_REGION=<approved-recovery-region>\nexport LINEAGE_PRIMARY_AVAILABILITY_ZONES=<az-a,az-b,az-c>\nexport LINEAGE_EPHEMERAL_PREFIX=lineage-e2e-<unique-suffix>\nexport LINEAGE_ENTERPRISE_ENDPOINT=https://<approved-private-dns-name>\nexport LINEAGE_ENTERPRISE_ENDPOINT_SERVICE_NAME=<approved-vpce-service-name>\nexport LINEAGE_PAGING_TOPIC_ARN=<approved-sns-topic-arn>\nexport ALLOW_LINEAGE_EPHEMERAL_AWS=1\n```\n\n```bash\nmake aws-deploy\nmake aws-smoke\nmake aws-cleanup\n```\n\n- `make aws-deploy` requires the deploy opt-in and environment values documented by the script.\n- `make aws-smoke` requires `ALLOW_LINEAGE_EPHEMERAL_AWS_SMOKE=1` and the generated CDK outputs file. It seeds a checksummed Incremental input and verifies I1–I10, DynamoDB/S3 evidence, CloudWatch correlation and duplicate no-effect.\n- `make aws-cleanup` requires the separate destructive acknowledgement `ALLOW_LINEAGE_EPHEMERAL_AWS_CLEANUP=DESTROY` and deletes only the validated namespace in the validated account/regions.\n\nProduction mode remains deletion-protected and Object-Locked. Only the strict ephemeral mode is disposable. Without approved credentials and opt-ins, the correct outcome is `AWS_REQUIRED`.\n\n## Architecture and delivery sources\n\n- [Repository navigation and command reference](docs/NAVIGATION.md)\n- [Normative architecture and six Mermaid views](docs/plans/2026-08-05-lineage-collection-architecture-refactor-design.md)\n- [Executable implementation plan and Tasks 1–22](docs/plans/2026-08-05-lineage-collection-architecture-refactor.md)\n- [Acceptance specification](docs/acceptance/lineage-platform-acceptance.md)\n- [Build-ready B01–B16 PRDs](docs/build-prds/README.md)\n- [Current implementation/evidence coverage](docs/prototype-coverage.md)\n- [Remaining enterprise context and resolved ambiguities](docs/prd-ambiguities.md)\n\n## Project map\n\n```text\napps/api/           domain/application core, FastAPI, local and AWS adapters/entry points\napps/web/           React control-room UI\ninfra/              CDK stacks, runtime packaging and generated ASL\nfixtures/           catalog and seeded payments-pipeline source\npackages/contracts/ strict JSON Schemas at component boundaries\nscripts/            workflow export, acceptance, AWS deploy/smoke/cleanup\ntests/              cross-component, acceptance, AWS-gated and documentation tests\ndocs/               approved design, implementation plan, coverage, ambiguities\ndata/               generated local state (ignored)\n```\n\nThe implementation coverage and deliberate external gates are listed in [docs/prototype-coverage.md](docs/prototype-coverage.md). PRD gaps, enterprise context seams, resolved conflicts, and reversible local decisions are recorded in [docs/prd-ambiguities.md](docs/prd-ambiguities.md).\n\nThe 18 source component PRDs used to design the prototype are included verbatim in [docs/component-prds](docs/component-prds).\n",
  "bytes": 22652,
  "sha": "d6d0efc8111b40ee912fd9d45e8611a282458ae550d01d95d0de31af652ee1a9",
  "repo_slug": "kart-rc/lineage-collection",
  "fonte": "repo",
  "truncated": false,
  "api": "https://agentalog.com/api/listings/okf_kart_rc_lineage_collection_second_brain__0bdb5b6f/readme"
}