{
  "markdown": "<!-- mcp-name: io.github.gauthierpiarrette/datadiffer -->\n\n# datadiffer\n\n**Diff two tables. See what changed, and which slice of your data it's concentrated in.**\n\n```\n98.6% of modified rows have country = 'DE' (8.87x over-represented)\n```\n\n[![PyPI](https://img.shields.io/pypi/v/datadiffer)](https://pypi.org/project/datadiffer/)\n[![CI](https://github.com/gauthierpiarrette/datadiffer/actions/workflows/ci.yml/badge.svg)](https://github.com/gauthierpiarrette/datadiffer/actions/workflows/ci.yml)\n[![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)\n[![Python 3.10+](https://img.shields.io/badge/python-3.10%2B-blue)](https://pypi.org/project/datadiffer/)\n\n`datadiffer` compares two tables (dev vs. prod, PR vs. main, source vs.\ndestination) and tells you **what changed and where it's concentrated**: rows\nadded / removed / modified, per-column change rates, and segment attribution.\n\nBuilt for the era when agents write your pipelines and someone has to check\ntheir work. A maintained, MIT-licensed successor to the archived\n[data-diff](https://github.com/datafold/data-diff).\n\n## Try it in 60 seconds\n\nNo credentials, no config, no account. It generates its own data:\n\n```bash\nuvx datadiffer demo\n```\n\n![datadiffer demo](docs/demo.gif)\n\n```\ndatadiffer-demo/orders_prod.parquet vs datadiffer-demo/orders_dev.parquet\nkey: order_id (inferred from *_id naming)\nrows: +800 added  -250 removed  ~432 modified  49318 unchanged\n┏━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┓\n┃column ┃ changed rows ┃ % of matched┃\n┡━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━━━━┩\n│amount │          432 │        0.87%│\n└───────┴──────────────┴─────────────┘\n-> 98.6% of modified rows have country = 'DE' (8.87x over-represented)\n   93.2% of added rows have plan = NULL (63.19x over-represented)\nsample changes:\n  [order_id=0]  amount: 5.0 -> 7.1\n  [order_id=1]  amount: 6.37 -> 6.38\n  [order_id=117]  amount: 32.4 -> 34.5\n  [order_id=234]  amount: 59.8 -> 61.9\n  [order_id=351]  amount: 87.2 -> 89.3\nDIFF FOUND\n```\n\nThat last block is the point. A diff tells you 432 rows changed; datadiffer\ntells you they're almost all German orders, which is the difference between\n\"something moved\" and \"the VAT change landed.\"\n\n## Install\n\n```bash\npip install datadiffer                      # local files: Parquet, CSV, DuckDB\npip install 'datadiffer[snowflake]'         # + Snowflake\npip install 'datadiffer[postgres]'          # + Postgres\npip install 'datadiffer[mcp]'               # + MCP server for AI agents\n```\n\n## Use it\n\n```bash\n# local files\ndatadiffer diff prod.parquet dev.parquet\n\n# DuckDB\ndatadiffer diff prod.duckdb:orders dev.duckdb:orders\n\n# Postgres (attached read-only)\ndatadiffer diff orders orders_v2 --source \"postgresql://user@host:5432/db\"\n\n# Snowflake, or anything in datadiffer.toml (run `datadiffer init` first)\ndatadiffer diff wh::analytics.orders wh::analytics_dev.orders\n\n# cross-source: warehouse vs. a local file\ndatadiffer diff wh::orders orders_snapshot.parquet\n\n# machine-readable, for scripts and CI\ndatadiffer diff a.parquet b.parquet --format json\n```\n\nExit codes follow the GNU diff convention: **0** no differences, **1**\ndifferences found, **2** operational error.\n\nUseful flags: `--key` (inferred when omitted), `--where`, `--columns` /\n`--exclude-columns`, `--sample-limit`, `--no-attribution`, `-q`.\n\n## In your pull requests\n\n[datadiffer-action](https://github.com/gauthierpiarrette/datadiffer-action)\nposts the report as a sticky PR comment:\n\n> ### datadiffer: `ANALYTICS.ORDERS` vs `ANALYTICS_PR_482.ORDERS`: differences found ❌\n>\n> **+800 added · −250 removed · ~432 modified · 49,318 unchanged** (2.96% of base rows affected)\n> Policy: max-changed-rows-pct exceeded: 2.96% > 0.5%\n>\n> | Column | Changed rows | % of matched |\n> |---|---:|---:|\n> | `amount` | 432 | 0.87% |\n>\n> > **Where it's concentrated:** 98.6% of modified rows have `country = 'DE'` (8.87× over-represented)\n> > 93.2% of added rows have `plan = NULL` (63.19× over-represented)\n\n```yaml\n- uses: gauthierpiarrette/datadiffer-action@v1\n  with:\n    config: datadiffer.toml          # holds the warehouse connection\n    table-a: wh::ANALYTICS.ORDERS\n    schema-map: \"ANALYTICS=ANALYTICS_PR_${PR_NUMBER}\"\n    max-changed-rows-pct: \"0.5\"\n  env:\n    SNOWFLAKE_PRIVATE_KEY: ${{ secrets.SNOWFLAKE_PRIVATE_KEY }}\n```\n\nRunning dbt? [docs/dbt-ci.md](docs/dbt-ci.md) turns `dbt ls --select\nstate:modified` into one diff per changed model, each with its own comment.\n\n## For AI agents (MCP)\n\nLet a coding agent verify its own data changes:\n\n```bash\ndatadiffer init          # scaffold datadiffer.toml, then add the server to your client\n```\n\n```json\n{ \"mcpServers\": { \"datadiffer\": { \"command\": \"datadiffer\", \"args\": [\"mcp\"] } } }\n```\n\nAny MCP client works. `datadiffer init` prints the server entry to paste,\nincluding a `uvx` form for clients that prefer no global install.\n\nFour read-only tools: `list_connections`, `schema_diff`, `diff_summary`\n(cheap preflight), `diff_tables` (full report). **Credentials never pass\nthrough the model.** Tools take connection *names* from your local\n`datadiffer.toml`, never DSNs. Over-cap requests come back as structured\nrefusals with a remedy, so the agent can self-correct instead of failing.\n\n## What makes it different\n\nMost diff tools answer *whether* two tables match. datadiffer answers *what\nchanged and where*: which columns moved, and which slice of the data the\nchange is concentrated in.\n\n| Tool | Use it instead when |\n|---|---|\n| [Datafold](https://www.datafold.com/) | You need billion-row cross-database reconciliation, a collaboration UI, lineage/impact analysis, and a support contract. |\n| [SQLMesh `table_diff`](https://sqlmesh.readthedocs.io/) | You already run SQLMesh and diff within a single gateway (cross-database diffing is a Tobiko Cloud feature). |\n| [Recce](https://github.com/DataRecce/recce) | You want a dbt-native PR review UI and your whole workflow is dbt. |\n| [Google DVT](https://github.com/GoogleCloudPlatform/professional-services-data-validator) | You're validating a migration across 17 warehouse types and want pass/fail validation with YAML configs. |\n| [datacompy](https://github.com/capitalone/datacompy) | You're comparing two pandas/Spark/Polars DataFrames inside one process. |\n\ndatadiffer is for the case in between: one command, no platform, works from\nyour terminal, your CI, or your agent, and it explains itself.\n\n## Migrating from data-diff / reladiff\n\nSame job, different flags:\n\n| data-diff / reladiff | datadiffer |\n|---|---|\n| `data-diff DB1 table1 DB2 table2` | `datadiffer diff <a> <b>` (locators or `--source`/`--target`) |\n| `-k`, `--key-columns` | `--key` (repeatable, or comma-separated) |\n| `-c`, `--columns` | `--columns` |\n| `-w`, `--where` | `--where` (validated: single boolean expression, no subqueries) |\n| `--json` | `--format json` (stable [schema v1](src/datadiffer/report/json_schema.py)) |\n| `-l`, `--limit` | `--sample-limit` |\n\nDifferences worth knowing: keys are inferred from declared constraints or\n`*_id` conventions when you omit `--key`; unknown column names in filters are\nan **error**, never silently ignored; and exit code 1 means \"diff found\"\n(2 is reserved for operational errors), so CI can tell them apart.\n\n**Not yet ported:** checksum-bisection hashdiff for very large cross-database\ndiffs, which is reladiff's headline capability, plus MySQL, Oracle,\nClickHouse, Trino and the rest of its connector list. If you need those today, use\n[reladiff](https://github.com/erezsh/reladiff); the bisection port is the top\nitem on our v0.2 roadmap and will credit its author.\n\nFull command mapping and behavior differences:\n**[docs/migrating-from-data-diff.md](docs/migrating-from-data-diff.md)**.\n\n## Scope and limits (v0.1)\n\n- **Warehouses:** Snowflake, Postgres, DuckDB, Parquet, CSV. BigQuery is next (v0.1.1).\n- **Every diff runs in local DuckDB.** Postgres is attached read-only and\n  scanned in place; Snowflake is pulled once over Arrow with only the needed\n  columns. Both are capped at 50M rows and 10 GiB per side, so narrow the\n  comparison with `--where`. Pushing the comparison into the warehouse, and lifting that cap\n  with checksum bisection, are the next two pieces of work.\n- Postgres reads are unsnapshotted (the report says so in `execution.snapshot`);\n  Snowflake pulls are a single consistent SELECT.\n- Segment attribution is single-column, categorical, and descriptive. It says\n  \"over-represented\", never causal.\n\n## Project\n\nMIT, no CLA, no telemetry. The [report schema](src/datadiffer/report/json_schema.py)\nis frozen at v1 and contract-tested, so pipelines and agents that parse it keep\nworking across releases. Dependencies are deliberately few: DuckDB, PyArrow,\nsqlglot, click, rich.\n\n## What's next\n\nBigQuery, then checksum-bisection hashdiff so large cross-database diffs stop\nneeding a row cap (ported with credit to reladiff), then more connectors.\nTracked in [issues](https://github.com/gauthierpiarrette/datadiffer/issues).\nComment on the one you need and it moves up.\n\n## Contributing\n\n```bash\ngit clone https://github.com/gauthierpiarrette/datadiffer && cd datadiffer\nuv sync --group dev\nuv run pytest -q            # warehouse tests skip unless credentials are set\nuv run ruff check .\n```\n\nPrior art gratefully acknowledged: [data-diff](https://github.com/datafold/data-diff)\nand [reladiff](https://github.com/erezsh/reladiff) (Erez Shinan), whose\nchecksum-bisection design v0.2 will port; Adtributor (Microsoft Research) for\nthe attribution scoring; and DuckDB, which makes all of this fast enough to be\nboring.\n\n## License\n\nMIT\n",
  "bytes": 9573,
  "sha": "d7432a25821e855ccc1d474bfd9d23f6a85487911163d94c8c7b323368986453",
  "repo_slug": "gauthierpiarrette/datadiffer",
  "fonte": "repo",
  "truncated": false,
  "api": "https://agentalog.com/api/listings/mcp_io_github_gauthierpiarrette_datadiffer_a046cf36/readme"
}