{
  "markdown": "# Operator ETL\n\nAgentic data intake for FOIA and public comments — a locally proven MVP with a\ndeterministic Medallion warehouse, LangGraph orchestration, Model Context Protocol\n(MCP) allowlist, and a fail-closed PII policy plane.\n\n[![CI](https://github.com/khaosans/operator-etl/actions/workflows/ci.yml/badge.svg)](https://github.com/khaosans/operator-etl/actions/workflows/ci.yml)\n[![Security](https://github.com/khaosans/operator-etl/actions/workflows/security.yml/badge.svg)](https://github.com/khaosans/operator-etl/actions/workflows/security.yml)\n[![Secret scan](https://github.com/khaosans/operator-etl/actions/workflows/secret-scan.yml/badge.svg)](https://github.com/khaosans/operator-etl/actions/workflows/secret-scan.yml)\n[![CodeQL](https://github.com/khaosans/operator-etl/actions/workflows/codeql.yml/badge.svg)](https://github.com/khaosans/operator-etl/actions/workflows/codeql.yml)\n[![Release](https://img.shields.io/github/v/release/khaosans/operator-etl?include_prereleases)](https://github.com/khaosans/operator-etl/releases)\n[![Docs](https://img.shields.io/badge/docs-GitHub%20Pages-blue)](https://khaosans.github.io/operator-etl/)\n[![Python 3.12+](https://img.shields.io/badge/python-3.12%2B-blue.svg)](https://www.python.org/)\n[![License](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](LICENSE)\n\n> Python and SQL decide what data exists. Agents orchestrate within typed boundaries.\n> The Critic proves numeric claims.\n\n## Contents\n\n- [Status](#status)\n- [Features](#features)\n- [Architecture](#architecture)\n- [Quickstart](#quickstart)\n- [Configuration](#configuration)\n- [Repository layout](#repository-layout)\n- [Testing](#testing)\n- [Docker and packages](#docker-and-packages)\n- [Documentation](#documentation)\n- [Contributing](#contributing)\n- [Security](#security)\n- [Support](#support)\n- [License](#license)\n\n## Status\n\n| Area | State |\n|---|---|\n| Local FOIA MVP (`./scripts/verify.sh`) | **IMPLEMENTED** |\n| Medallion + LangGraph + MCP + critic | **IMPLEMENTED** |\n| Observability (sanitized OTel) + A2A task surface | **IMPLEMENTED** |\n| Multi-cloud Terraform (GCP / AWS / Azure) | Staging stacks present |\n| Live GCP / BigQuery E2E | **PARTIAL** |\n| Presidio PII engine | Optional (`--extra presidio`); default is regex |\n\nHonest inventory: [docs/FINAL-REVIEW.md](docs/FINAL-REVIEW.md) ·\n[okf/models/implementation-status.md](okf/models/implementation-status.md).\n\n**Who this is for:** agencies and regulated teams exploring agentic FOIA / public-comment\nintake with proof gates, not a turnkey production FOIA deployment.\n\n**What we do not claim:** FedRAMP / ATO, live cloud E2E as proven, or that `verify.sh`\ngreen means production-ready FOIA software. See [docs/PUBLIC-READINESS.md](docs/PUBLIC-READINESS.md).\n\n## Features\n\n- **Medallion warehouse** — bronze → silver + quarantine → gold SQL marts (DuckDB local)\n- **Fail-closed PII** — scan before insight; encrypted vault (`0600`); no vault decrypt via MCP\n- **Critic faithfulness** — insight numbers must appear in gold metrics\n- **MCP allowlist** — three tools only; no raw SQL\n- **Observability** — OpenTelemetry / OpenInference metadata without raw PII in spans\n- **A2A** — JSON-RPC task surface with bearer auth and sanitized artifacts\n- **Proof gate** — `./scripts/verify.sh` runs OKF validate, pytest, and the FOIA demo\n\n## Architecture\n\nThree planes keep generative intelligence away from raw operational data:\n\n```mermaid\nflowchart TB\n  subgraph control [Control plane]\n    LG[LangGraph state machine]\n    Critic[Critic audit]\n    HITL[HITL approval]\n  end\n  subgraph policy [Policy plane]\n    PII[PII scan]\n    Vault[AES vault]\n    MCP[MCP allowlist]\n  end\n  subgraph data [Data plane]\n    Bronze[Bronze raw]\n    Silver[Silver validated]\n    Quarantine[Quarantine]\n    Gold[Gold SQL marts]\n  end\n  LG --> MCP\n  MCP --> PII\n  PII --> Vault\n  Bronze --> Silver\n  Bronze --> Quarantine\n  Silver --> Gold\n  Gold --> Critic\n```\n\n| Layer | Stack | Invariant |\n|---|---|---|\n| Data | Python 3.12+, DuckDB, SQL, Pydantic 2 | Deterministic transforms; quarantine preserves bad rows |\n| Control | LangGraph, MCP, SQLite / Postgres checkpoints | Resumable runs; critic gate |\n| Policy | Cryptography (Fernet), regex PII (Presidio optional) | No raw PII in insights, MCP, or OTel |\n| Packaging | uv, Docker (GHCR), GitHub Actions, MkDocs | Frozen lockfile; CI SAST/SCA/secrets/IaC |\n\n## Quickstart\n\n### Prerequisites\n\n- Python 3.12+ (or [uv](https://docs.astral.sh/uv/))\n\n### Verify in one command\n\n```bash\ngit clone https://github.com/khaosans/operator-etl.git\ncd operator-etl\n./scripts/verify.sh\n```\n\nInstalls `uv` if needed, syncs frozen deps, validates the OKF bundle, runs pytest, and\nexecutes the FOIA demo on a fresh warehouse. Success ends with `OPERATOR_ETL_VERIFY=PASS`.\n\nExpected demo metrics on sample data: `status=complete`, `silver=10`, `quarantined=2`.\n\nFull guide: [docs/QUICKSTART.md](docs/QUICKSTART.md).\n\n### Run the FOIA graph\n\n```bash\nuv run etl-graph --source public_comments --pipeline public_comments\n```\n\n```text\nstatus=complete  run_id=...\nrows_in=12  silver=10  quarantined=2\npii_findings=3  critic_passed=True\n```\n\n`pii_findings=3` is scanner groups (EMAIL, PHONE, US_SSN). Dashboard **PII flagged ≥ 4**\ncounts silver comments with PII — both are expected on the synthetic sample.\n\n### Dashboard (optional)\n\n```bash\nexport OPERATOR_ETL_WAREHOUSE=\".tmp/mvp-demo/operator.duckdb\"\nexport OPERATOR_ETL_PIPELINE_NAME=public_comments\nexport OPERATOR_ETL_DOMAIN=gov\nuv run streamlit run dashboard/app.py\n```\n\nScreenshots: [docs/TOUR.md](docs/TOUR.md).\n\n## Configuration\n\nCopy [`.env.example`](.env.example) for local DuckDB runs. Common variables:\n\n| Variable | Purpose |\n|---|---|\n| `OPERATOR_ETL_WAREHOUSE` | DuckDB path (default `warehouse/operator.duckdb`) |\n| `OPERATOR_ETL_PIPELINE_NAME` | Pipeline id (e.g. `public_comments`) |\n| `OPERATOR_ETL_DOMAIN` | `gov` or commercial demo domain |\n| `OPERATOR_ETL_BACKEND` | `duckdb` locally |\n| `OPERATOR_ETL_A2A_BEARER_TOKEN` | Optional A2A auth |\n| `OTEL_*` | Optional observability export |\n\nCloud secrets (`PII_VAULT_KEY`, API keys) live in\n[`infra/env.example`](infra/env.example) / Terraform examples — never commit `.env` or\n`terraform.tfvars`.\n\n## Repository layout\n\n```text\nsrc/operator_etl/            Data plane\nsrc/operator_etl_graph/      LangGraph control plane\nsrc/operator_etl_policy/     PII + vault\nsrc/operator_etl_mcp/        MCP server\nsrc/operator_etl_{gcp,aws,azure}/  Cloud adapters\nsrc/a2a/                     A2A JSON-RPC surface\nsrc/telemetry/               Sanitized OTel\npipelines/  sql/  samples/   Registry, gold SQL, synthetic data\ninfra/{gcp,aws,azure}/       Terraform staging stacks\ntests/  harness/  scripts/   Proof gate\nokf/  skills/  docs/         Knowledge bundle, agent skills, wiki\n```\n\n## Testing\n\n```bash\nmake test        # pytest\nmake e2e         # OKF + pytest + FOIA demo\nmake lint        # ruff\nmake security    # bandit + pip-audit\n```\n\nEvery architectural invariant has automated coverage (ingest idempotency, quarantine, PII,\ncritic, MCP deny, telemetry, A2A). Map: [docs/TESTING.md](docs/TESTING.md) ·\nproof citations: [docs/FOUNDATIONS.md](docs/FOUNDATIONS.md).\n\n## Docker and packages\n\n```bash\n# Tagged release (see GitHub Releases for current version)\ndocker pull ghcr.io/khaosans/operator-etl:0.7.0\ndocker run --rm -it ghcr.io/khaosans/operator-etl:0.7.0 etl-graph --help\n\n# Or :latest for the newest non-prerelease tag\ndocker pull ghcr.io/khaosans/operator-etl:latest\n```\n\n```bash\npip install operator-etl --index-url https://pypi.pkg.github.com/khaosans/simple/\n```\n\nRelease SBOMs (CycloneDX) attach to GitHub Releases. Process: [docs/RELEASING.md](docs/RELEASING.md).\n\n## Documentation\n\n**Wiki:** [https://khaosans.github.io/operator-etl/](https://khaosans.github.io/operator-etl/)\n\n| Document | Description |\n|---|---|\n| [QUICKSTART.md](docs/QUICKSTART.md) | One-command verify |\n| [WALKTHROUGH.md](docs/WALKTHROUGH.md) | Local operational tour |\n| [SECURITY-HARDENING.md](docs/SECURITY-HARDENING.md) | HTTP guards, vault, CI SAST/SCA |\n| [HOW-IT-WORKS.md](docs/HOW-IT-WORKS.md) | Runtime and cloud architecture |\n| [A2A.md](docs/A2A.md) | Agent task API contract |\n| [Operator-ETL-White-Paper.md](docs/Operator-ETL-White-Paper.md) | Full engineering spec ([PDF](docs/Operator-ETL-White-Paper.pdf)) |\n\n## Contributing\n\nSee [CONTRIBUTING.md](CONTRIBUTING.md) and [CODE_OF_CONDUCT.md](CODE_OF_CONDUCT.md).\n\n```bash\nmake e2e && make lint && make security\nuv run pre-commit install   # optional local hooks\n```\n\nCI must be green before merge (e2e, docker, terraform, gitleaks, bandit, pip-audit, CodeQL,\nTrivy, Checkov). Ruleset setup: [docs/PUBLIC-READINESS.md](docs/PUBLIC-READINESS.md).\n\n## Security\n\nReport vulnerabilities per [SECURITY.md](SECURITY.md). Do not open public issues for\nsensitive disclosures.\n\nAgent checklist: [skills/operator-security/SKILL.md](skills/operator-security/SKILL.md).\n\n## Support\n\n- Docs wiki and [docs/index.md](docs/index.md)\n- [GitHub Issues](https://github.com/khaosans/operator-etl/issues) for bugs and questions\n- Changelog: [CHANGELOG.md](CHANGELOG.md) · Versioning: [docs/VERSIONING.md](docs/VERSIONING.md)\n\nAll sample intake records are **synthetic**.\n\n## License\n\nLicensed under the [Apache License 2.0](LICENSE).\n",
  "bytes": 9280,
  "sha": "3486abab26530fde04204b1c0c9179e018218419d705590e71c56af40aaffd45",
  "repo_slug": "khaosans/operator-etl",
  "fonte": "repo",
  "truncated": false,
  "api": "https://agentalog.com/api/listings/okf_khaosans_operator_etl_okf_index_md_384f88ce/readme"
}