{
"markdown": "\n\n
csv-quality-gate
\n\ncsv-quality-gate is a command-line data quality gate that runs CSV preflight validation, failing fast before a pipeline ingests broken, incomplete, duplicated, or junk input.\n\ncsv-quality-gate is developed by [Hermes Labs](https://hermes-labs.ai).\n\nHermes Labs is an agentic infrastructure company building the reliability layer for autonomous systems.\n\n[](https://github.com/hermes-labs-ai/csv-quality-gate/actions/workflows/ci.yml)\n[](https://pypi.org/project/csv-quality-gate/)\n[](https://pypi.org/project/csv-quality-gate/)\n[](LICENSE)\n\n\n\nIt runs batch quality checks on a CSV and returns `pass`, `warn`, or `fail` (with matching exit codes) before expensive pipeline steps burn time on bad input. It checks for missing required columns, empty files, empty critical cells, duplicate rows, and — under the `outreach` profile — suspicious company-name patterns. Teams can declare their own columns, thresholds, and patterns in a small TOML/JSON config, every issue points at the affected line numbers (never cell values), and the same gate runs as a pre-commit hook or a GitHub Action. It is stdlib-only: no third-party runtime dependencies.\n\nThe problems it is built for:\n\n- \"We keep running expensive pipeline steps on broken CSVs.\"\n- \"A batch run fails 20 minutes in because the input CSV was junk.\"\n- \"We only discover missing required columns after the job already started.\"\n- \"Duplicate rows and empty contact fields keep polluting our batch runs.\"\n- \"The gate says 12% duplicates, but which rows?\"\n- \"Our CSVs have `email` and `order_id`, not `company`.\"\n- \"I want CSV preflight validation, not a whole data platform.\"\n\n## Quickstart (60 seconds)\n\n```bash\npip install csv-quality-gate\npython -c \"from importlib.metadata import version; print(version('csv-quality-gate'))\" # confirm the installed release\ncsv-quality-gate check leads.csv --profile outreach\n```\n\nExample output for a CSV with a missing column and a borderline duplicate rate:\n\n```text\ncsv-quality-gate: FAIL\nfile: leads.csv\nprofile: outreach\nrows: 125\n ERROR: missing required column: person_name\n WARNING: duplicate rate 12% exceeds warning threshold 10%\n evidence: column=company affected=15 rows at line(s) 4, 9, 15, 22, 31 (+10 more)\n```\n\nThe process exits `0` on pass, `1` on warnings only, and `2` on fail, so you can wire it directly into a shell script, a pre-commit hook, or a CI step.\n\n\n\n### Try it with the included fixtures\n\nYou can verify the gate before connecting it to your own pipeline. From a\ncheckout, install the package and run the included clean and broken inputs:\n\n```bash\ngit clone https://github.com/hermes-labs-ai/csv-quality-gate.git\ncd csv-quality-gate\npython -m pip install .\ncsv-quality-gate check examples/clean.csv\ncsv-quality-gate check examples/promptfoo-dataset/tests-broken.csv \\\n --config examples/promptfoo-dataset/csv-quality-gate.toml \\\n --profile promptfoo\n```\n\nThe first command exits `0` with `PASS`; the second exits `2` with bounded\nline-number evidence for the broken rows. This gives a first-use check without\ncreating a sample CSV or guessing which profile and config to choose.\n\n## Development installation\n\nFor everyday use, follow the [Quickstart](#quickstart-60-seconds) above. For a\nsource checkout, editable install, and lint/test commands, see\n[Development](#development) below.\n\n### As an agent skill (Claude Code, Codex CLI, Gemini CLI)\n\nThe repository root is a portable [Agent Plugin](https://agent-plugins.org)\n(`plugin.json`) that ships one skill, `skills/csv-quality-gate/SKILL.md`. It\nteaches the agent to run this gate on the CSV files you name and report the\nstatus and line-number evidence without overclaiming.\n\n| Host | Install | Read back |\n| --- | --- | --- |\n| Claude Code | `claude plugin marketplace add hermes-labs-ai/csv-quality-gate`
`claude plugin install csv-quality-gate@csv-quality-gate` | `claude plugin list` |\n| OpenAI Codex CLI | `codex plugin marketplace add hermes-labs-ai/csv-quality-gate`
`codex plugin add csv-quality-gate@csv-quality-gate` | `codex plugin list` |\n| Gemini CLI | `gemini extensions install https://github.com/hermes-labs-ai/csv-quality-gate --ref main` | `gemini skills list` |\n| skills.sh | `npx skills add https://github.com/hermes-labs-ai/csv-quality-gate --skill csv-quality-gate` | `npx skills list` |\n\nWhat each host reads:\n\n- Claude Code reads `.claude-plugin/marketplace.json` (source `.`) and\n `.claude-plugin/plugin.json`.\n- Codex reads the repo marketplace `.agents/plugins/marketplace.json` (source\n `./`, the root) and the portable `plugin.json`.\n- Gemini CLI reads `gemini-extension.json` and discovers the skill under\n `skills/`. Keep `--ref main`: without a ref, Gemini CLI installs the latest\n GitHub release archive, and releases up to v0.3.0 predate\n `gemini-extension.json`.\n\nInstalling the skill does not install the Python package. The skill uses an\ninstalled `csv-quality-gate` or runs the pinned release with\n`uvx csv-quality-gate==0.3.1`.\n\n## Usage\n\n```bash\ncsv-quality-gate check leads.csv\ncsv-quality-gate check leads.csv --profile outreach\ncsv-quality-gate check leads.csv --profile generic --json\ncsv-quality-gate check leads.csv --config csv-quality-gate.toml --profile leads\ncsv-quality-gate check leads.csv --max-examples 20\ncsv-quality-gate check data/*.csv\n```\n\nOptions for `check`:\n\n- `--profile NAME` — a built-in profile (`generic`, `outreach`) or one declared in `--config`.\n- `--config FILE` — TOML or JSON file with project-specific profiles (see [Custom profiles](#custom-profiles)).\n- `--max-examples N` — how many affected line numbers each issue lists (default `5`; `0` keeps counts only).\n- `--json` — machine-readable output.\n\nSeveral paths can be checked in one call. Text mode prints one block per file,\nJSON mode emits an array (a single path keeps the plain object), and the exit\ncode is the worst status across files.\n\nExit codes:\n\n- `0` pass\n- `1` warnings only\n- `2` fail (or `2` when the file does not exist, is not UTF-8, the profile is unknown, or the config is invalid)\n\n## Profiles\n\nBuilt-in profiles:\n\n- `generic`\n - checks for a required `company` column, empty `company` cells, duplicate `company` values, and empty files\n- `outreach`\n - requires `company` and `person_name`, with higher empty-rate tolerances and an added suspicious company-name heuristic for GTM/contact pipelines\n\nThe thresholds for each profile are defined in `src/csv_quality_gate/profiles.py`.\n\n### Custom profiles\n\nDeclare your own profiles in a TOML (Python 3.11+) or JSON file and pass it\nwith `--config`. A copyable example lives in\n[`examples/csv-quality-gate.toml`](examples/csv-quality-gate.toml).\n\n```toml\n[profiles.leads]\nextends = \"outreach\" # optional: inherit a built-in's columns, patterns, and rates\nrequired_columns = [\"email\", \"company\", \"person_name\"]\ncritical_columns = [\"email\", \"person_name\"]\nduplicate_column = \"email\"\nempty_warning_rate = 0.05\nempty_fail_rate = 0.20\nduplicate_warning_rate = 0.02\nduplicate_fail_rate = 0.10\nsuspicious_column = \"email\"\nsuspicious_patterns = ['^[^@]+$'] # case-insensitive regular expressions\nsuspicious_warning_rate = 0.20\nsuspicious_fail_rate = 0.50\n```\n\n```bash\ncsv-quality-gate check data/leads.csv --config csv-quality-gate.toml --profile leads\n```\n\nRules:\n\n- Every key is optional. Without `extends`, a profile starts with no columns,\n no patterns, and the `generic` thresholds.\n- A rate of `0.0` means \"any occurrence\": a check with no affected rows never\n raises an issue, so a clean file still passes.\n- Config is data only: column names, rates between `0` and `1`, and regex\n strings. Unknown keys, out-of-range rates, a warning rate above its fail rate,\n and invalid regexes are rejected with a precise message, and the CLI returns a\n normal `fail` receipt (exit `2`) instead of a traceback.\n- Built-ins stay available next to your profiles. A custom profile with the same\n name as a built-in replaces it for that run only.\n- Regexes come from your own repository; keep them simple, since the gate does\n not guard against pathological patterns.\n\n## Output\n\nText mode (default):\n\n```text\ncsv-quality-gate: FAIL\nfile: leads.csv\nprofile: outreach\nrows: 125\n ERROR: missing required column: person_name\n WARNING: duplicate rate 12% exceeds warning threshold 10%\n evidence: column=company affected=15 rows at line(s) 4, 9, 15, 22, 31 (+10 more)\n```\n\nEvery empty-rate, duplicate-rate, and suspicious-value issue carries bounded\nevidence: the affected column, the total number of affected rows, and up to\n`--max-examples` physical line numbers (the header is line 1; a quoted record\nthat spans several lines reports its last line). Cell values are never printed,\nso receipts stay safe to attach to CI logs or tickets. Duplicate evidence points\nat the second and later occurrences, so those are the lines to remove.\n\nJSON mode (`--json`) emits an object with `path`, `profile`, `rows`, `status`,\n`issues[]`, and (when `--config` is used) `config`. Each issue has `severity`\nand `message`; issues backed by rows add an `evidence` object:\n\n```bash\ncsv-quality-gate check leads.csv --json\n```\n\n```json\n{\n \"path\": \"leads.csv\",\n \"profile\": \"outreach\",\n \"rows\": 125,\n \"status\": \"fail\",\n \"issues\": [\n {\"severity\": \"error\", \"message\": \"missing required column: person_name\"},\n {\n \"severity\": \"warning\",\n \"message\": \"duplicate rate 12% exceeds warning threshold 10%\",\n \"evidence\": {\"column\": \"company\", \"total\": 15, \"rows\": [4, 9, 15, 22, 31]}\n }\n ]\n}\n```\n\n## Limitations / What it does not do\n\n- Heuristics are intentionally simple: empty-rate, duplicate-rate, and regex-based name patterns. They do not learn from your data.\n- It validates shape and obvious noise, not semantic correctness — it cannot tell whether `company` values are real, only whether they are present, unique, and not obviously junk.\n- The `outreach` profile is opinionated. Its suspicious-name patterns and thresholds were chosen for GTM contact lists and should not be treated as universal truth.\n- Duplicate and empty checks operate on the columns a profile names; it does not auto-detect which columns matter. Built-ins use `company` and `person_name`; use a config file for anything else.\n- Evidence is line numbers and counts only. It never quotes cell values, so it cannot tell you *what* a bad value was, only where it is.\n- It validates each CSV file independently and assumes UTF-8 (BOM-tolerant), comma-separated input.\n- It is not a data quality platform: no lineage, no profiling reports, no schema inference, no row-level remediation.\n\n## When to use it\n\n- Before enrichment, outreach, ETL, or batch scoring runs\n- As a pre-commit hook or CI step for checked-in CSV inputs\n- As a preflight gate before expensive pipeline work\n\n## When not to use it\n\n- When you need semantic validation of the data itself\n- When your input is not CSV\n- When you need a full data quality framework with lineage and profiling\n\n## pre-commit\n\nRun the same gate on staged CSV files before they are committed. Add this to\nyour `.pre-commit-config.yaml` (full example in\n[`examples/pre-commit-config.yaml`](examples/pre-commit-config.yaml)):\n\n```yaml\nrepos:\n - repo: https://github.com/hermes-labs-ai/csv-quality-gate\n rev: v0.3.1\n hooks:\n - id: csv-quality-gate\n args: [--profile, outreach]\n # or: args: [--config, csv-quality-gate.toml, --profile, leads]\n```\n\nThe hook receives every staged `.csv` file, prints one report block per file,\nand blocks the commit on `warn` or `fail`, matching the CLI exit codes. Narrow\nit with pre-commit's `files:` pattern if only some CSVs should be gated, and\ntune thresholds through a config file rather than skipping the hook.\n\nTo try the hook from a local checkout before pinning a release, point a\nconfig at the checkout path and a commit:\n\n```yaml\nrepos:\n - repo: /path/to/csv-quality-gate\n rev: \n hooks:\n - id: csv-quality-gate\n args: [--profile, outreach]\n```\n\n```bash\npre-commit run --config that-file.yaml csv-quality-gate --files data/leads.csv\n```\n\n`pre-commit try-repo . csv-quality-gate --files data/leads.csv` also works for\nthe default `generic` profile (try-repo cannot pass hook `args`).\n\n## CI / GitHub Actions\n\nUse this repository directly as a composite Action. It installs the packaged CLI,\nruns the selected profile, and always writes a JSON receipt at\n`$GITHUB_WORKSPACE/csv-quality-gate-receipt.json`. The `status` and `receipt`\noutputs remain available even when the Action exits with a warning or failure.\n\n```yaml\n- id: csv_gate\n uses: hermes-labs-ai/csv-quality-gate@0b7bf4635b2db468620855e577cd0d9f09f09ec7 # hardened current main; annotations + batch\n with:\n csv-path: data/leads.csv\n profile: leads\n config: csv-quality-gate.toml # optional; omit to use built-in profiles\n annotate: true # optional; defaults to false\n\n- run: echo \"${{ steps.csv_gate.outputs.status }}\"\n```\n\nFor a batch, use `csv-paths` with one path per line instead. Set exactly one of\n`csv-path` and `csv-paths`; the Action deliberately accepts no free-form command\nor shell arguments.\n\n```yaml\n- id: csv_gate_batch\n uses: hermes-labs-ai/csv-quality-gate@0b7bf4635b2db468620855e577cd0d9f09f09ec7 # hardened current main; batch support\n with:\n csv-paths: |\n data/leads.csv\n data/customers.csv\n profile: generic\n```\n\nThe single-file receipt remains a JSON object. A batch receipt is a JSON array\nin the same order as `csv-paths`, and its status and exit code reflect the worst\nfile. The Action\nreturns the same exit codes as the CLI: `0` for pass, `1` for warn, and `2` for\nfail. The receipt contains the same bounded evidence as `--json`, so it is safe\nto upload as a workflow artifact.\nUse `continue-on-error: true` on a calling step if your workflow needs to inspect\nwarning or failure outputs before deciding how to proceed.\n\nSet `summary: true` to write a compact result to the GitHub Actions job summary.\nIt contains only status, profile, row count, and issue counts; it never includes\nthe CSV path or cell values. Evidence column labels are omitted unless you also\nset `summary-columns: true`; enable that only when those labels are suitable for\nthe job summary. The additive `tool-version` and `summary-written` outputs\nidentify the installed package version and whether this invocation rendered the\nsummary; the JSON receipt is unchanged.\n\nThe receipt path is fixed per workspace, so do not run more than one instance in\nparallel in the same workspace. The Action validates the package's existing CSV\nheuristics only; it does not add schema inference, semantic verification, or\narbitrary CLI options.\n\nSet `annotate: true` to add up to 50 GitHub Actions warnings or errors at the\nreceipt-backed CSV file and physical row. The annotations use the gate's existing\nbounded evidence only and omit cell values, issue messages, and column names.\nPaths outside the checked-out workspace and issues without row evidence are not\nannotated. Receipts larger than 1 MiB are skipped without changing the gate's\nstatus. This is an advisory location aid, not a security or SARIF report.\n\nA ready-to-copy, commit-pinned composite Action workflow also lives in\n[`examples/github-action.yml`](examples/github-action.yml).\n\n### GitHub Marketplace\n\n`action.yml` carries the metadata a Marketplace listing needs (name,\ndescription, author, branding). Listing is a maintainer step taken on the GitHub\nrelease form, not something the repository does on its own; the steps are in\n[Releasing](CONTRIBUTING.md#releasing). Whether you reach the Action through\nMarketplace or this repository, use a full commit SHA in the `uses:` line.\n\n## Recipes\n\nShort, copyable setups that put the gate in front of a specific tool. Each uses\na custom profile and ships with one passing and one failing fixture you can run\nfrom a checkout:\n\n- [dbt seeds](examples/dbt-seed.md) — gate `seeds/*.csv` before `dbt seed`.\n- [promptfoo datasets](examples/promptfoo-dataset.md) — gate a CSV test set\n before `promptfoo eval`.\n\n## Development\n\n```bash\npip install -e \".[dev]\"\nruff check .\npython3 -m pytest -q\npre-commit try-repo . csv-quality-gate --files tests/fixtures/clean.csv # optional hook smoke test\n```\n\n## Where it fits\n\ncsv-quality-gate guards the data that goes into a pipeline; it is complementary to, not a replacement for, agent- and prompt-level reliability tools.\n\n## More from Hermes Labs\n\nBrowse the [open-source catalog](https://hermes-labs.ai/open-source) or contact [roli@hermes-labs.ai](mailto:roli@hermes-labs.ai).\n\n## License\n\nApache-2.0 — see [LICENSE](LICENSE).\n\n## Citation\n\nIf you use this software, please cite it using the metadata in [`CITATION.cff`](CITATION.cff).\n",
"bytes": 17117,
"sha": "3cc46bcf7d1c2cf1f3eb808762677d5291805332d266394874f5c937eff6320d",
"repo_slug": "roli-lpci/csv-quality-gate",
"fonte": "repo",
"truncated": false,
"api": "https://api.agentalog.com/api/listings/plg_roli_lpci_csv_quality_gate_a1b21bf4/readme"
}