{
  "markdown": "<!-- mcp-name: io.github.Jackxiaozhiren/datasentry -->\n\n<p align=\"center\">\n  <img src=\"docs/assets/logo.svg\" alt=\"DataSentry\" width=\"128\">\n</p>\n\n<h1 align=\"center\">DataSentry</h1>\n\n<p align=\"center\">\n  <strong>Find bad data before your users do.</strong><br>\n  Automatic data-quality discovery, evidence-backed explanations, and safe reversible repair.<br>\n  <strong>Local-first. Deterministic by default. AI optional.</strong>\n</p>\n\n<p align=\"center\">\n  <a href=\"https://jackxiaozhiren.github.io/datasentry/\">Live demo</a> ·\n  <a href=\"#try-it-in-30-seconds\">30-second start</a> ·\n  <a href=\"#start-with-your-problem\">Use cases</a> ·\n  <a href=\"docs/FAQ.md\">FAQ</a> ·\n  <a href=\"https://pypi.org/project/datasentry-ai/\">PyPI</a> ·\n  <a href=\"examples/\">Examples</a> ·\n  <a href=\"docs/MCP.md\">MCP setup</a> ·\n  <a href=\"CONTRIBUTING.md\">Contribute</a>\n</p>\n\n<p align=\"center\">\n  <img alt=\"Release\" src=\"https://img.shields.io/github/v/release/Jackxiaozhiren/datasentry\">\n  <img alt=\"PyPI\" src=\"https://img.shields.io/pypi/v/datasentry-ai\">\n  <img alt=\"Python\" src=\"https://img.shields.io/badge/python-3.12%2B-blue\">\n  <img alt=\"CI\" src=\"https://img.shields.io/github/actions/workflow/status/Jackxiaozhiren/datasentry/ci.yml?label=CI\">\n  <img alt=\"License\" src=\"https://img.shields.io/github/license/Jackxiaozhiren/datasentry\">\n</p>\n\n<p align=\"center\">\n  <img src=\"docs/demo/quickstart.gif\" alt=\"DataSentry scans dirty data, explains evidence, and starts a safe repair loop\" width=\"780\">\n</p>\n\n> **中文导读**：DataSentry 会先自动发现数据质量问题，再给出样本、比例、置信度等证据。修复采用 `propose → preview → apply to a copy → verify → rollback` 的保守流程。检测与评分不依赖 LLM，AI 只作为可选辅助，数据可以完全留在本机。\n\n## Try it in 30 seconds\n\nInstall the current PyPI release and run the zero-config product tour:\n\n```bash\npip install --upgrade datasentry-ai\ndatasentry demo\n```\n\n`datasentry-demo` is also available as a direct console alias.\n\nThe demo generates synthetic dirty data, runs the built-in detectors, exports JSON + HTML reports, applies one safe repair to a copy, re-scans the repaired copy, and prints a rollback command. It needs no dataset, cloud service, API key, or LLM.\n\n```text\nsynthetic dirty CSV\n        ↓\n39 deterministic detectors\n        ↓\nevidence-backed issues + quality score\n        ↓\npreview → repaired copy\n        ↓\nre-scan → verify new/persistent issues\n```\n\nPrefer scanning your own data immediately?\n\n```bash\ncurl -L https://raw.githubusercontent.com/Jackxiaozhiren/datasentry/main/demo-data/orders.csv -o orders.csv\ndatasentry scan orders.csv\ndatasentry issues list --severity high\n```\n\nOr launch the interactive interfaces:\n\n```bash\ndatasentry          # terminal UI\ndatasentry-server   # Web UI + REST API at http://localhost:8000/ui/\n```\n\n## Start with your problem\n\n| If you need to... | Start here |\n|---|---|\n| Find suspicious data before you know every rule | `datasentry scan data.csv` |\n| See the full discovery → repair → verify loop | `datasentry demo` |\n| Block severe data issues in CI | [`docs/GITHUB_ACTIONS.md`](docs/GITHUB_ACTIONS.md) |\n| Add quality gates to dbt / Airflow | [`examples/integrations/`](examples/integrations/) |\n| Inspect and repair issues without overwriting the source | [`Safe repair`](#safe-repair-not-blind-mutation) |\n| Give AI agents deterministic data-quality tools | [`docs/MCP.md`](docs/MCP.md) |\n\nThe default path is intentionally low-commitment: **scan first, inspect evidence, then decide whether a finding deserves a permanent rule or a repair.**\n\n## Why DataSentry exists\n\nMost data-quality tools are excellent once you already know the expectations, checks, or contracts you want to enforce. Real incidents often start one step earlier: **you do not yet know what is wrong.**\n\nDataSentry is built around the complete remediation loop:\n\n```text\nFind → Explain → Fix safely → Verify\n```\n\n- **Find** — discover common quality problems without writing every rule first.\n- **Explain** — attach samples, affected counts/ratios, detector evidence, and confidence.\n- **Fix safely** — preview changes and apply repairs to a copy instead of mutating the source.\n- **Verify** — re-scan the repaired copy and surface persistent or newly introduced issues.\n\nDataSentry fits best when you are looking at data you do not fully trust yet, want evidence before codifying checks, or need a conservative remediation path. It complements expectation- and contract-driven tools rather than requiring you to replace them.\n\n## What it catches automatically\n\nDataSentry ships with **39 deterministic detectors** covering common failure modes such as:\n\n- missing and placeholder values;\n- invalid emails, URLs, dates, and encodings;\n- duplicate identifiers and uniqueness violations;\n- inconsistent categories and cross-field contradictions;\n- foreign-key and referential-integrity problems;\n- numeric and statistical outliers;\n- schema, row-count, score, and issue-distribution drift.\n\nEvery scan produces an evidence-backed issue list and a six-dimension quality score across completeness, validity, uniqueness, consistency, integrity, and timeliness.\n\n## Safe repair, not blind mutation\n\n```bash\n# inspect the highest-severity findings\ndatasentry issues list --severity high\n\n# propose a repair without changing data\ndatasentry repair propose <issue_id> --file orders.csv\n\n# preview the exact effect\ndatasentry repair preview <issue_id> --file orders.csv\n\n# apply to a repaired copy; the original is not overwritten\ndatasentry repair apply <issue_id> --file orders.csv\n\n# re-scan the repaired copy and detect regressions\ndatasentry repair verify <run_id>\n\n# inspect or undo the repair\ndatasentry repair diff <run_id>\ndatasentry repair rollback <run_id>\n```\n\nRepairs are fingerprinted, auditable, and reversible. AI-generated repair proposals remain human-approved state changes.\n\n### Quality gates for CI\n\n```bash\ndatasentry scan orders.csv --fail-on high\n```\n\nFor GitHub repositories, reuse DataSentry's maintained `workflow_call` gate instead of duplicating installation and exit-code handling:\n\n```yaml\njobs:\n  datasentry:\n    uses: Jackxiaozhiren/datasentry/.github/workflows/datasentry-quality-gate.yml@main\n    with:\n      path: data/orders.csv\n      fail_on: high\n```\n\nSee [`docs/GITHUB_ACTIONS.md`](docs/GITHUB_ACTIONS.md) for inputs, artifacts, security boundaries, and version-pinning guidance.\n\nReports can be exported as JSON, Markdown, HTML, JUnit, and SARIF. The GitHub Actions example fails the workflow on severe findings while still uploading an HTML report for review.\n\n## Give AI agents deterministic data-quality tools\n\nDataSentry includes an MCP stdio server:\n\n```bash\ndatasentry mcp --project /path/to/project\n```\n\nMCP-capable clients can scan files, inspect evidence-backed issues, read quality scores and trends, compare drift, validate contracts, manage scheduled jobs, and call DataSentry tools without bypassing the same underlying safety rules used by the CLI and REST API.\n\nCopy-paste setup recipes for **VS Code** and **Claude Desktop** are in [`docs/MCP.md`](docs/MCP.md).\n\n> **Boundary:** AI may propose; humans approve state-changing repairs.\n\n## Where it fits next to popular data-quality projects\n\nThis is a positioning guide, not a winner/loser feature scorecard. These projects solve overlapping but different jobs; check their upstream documentation for current capabilities.\n\n| Project | Core mental model | A strong fit when you want... |\n|---|---|---|\n| **DataSentry** | discover → explain → repair → verify | automatic issue discovery plus a controlled, reversible remediation loop |\n| [Great Expectations](https://github.com/great-expectations/great_expectations) | Expectations / expressive data tests | explicit validation rules, validation results, and generated data-quality documentation |\n| [Soda Core](https://github.com/sodadata/soda-core) | data contracts and quality checks | YAML contracts and verification across a broad data stack |\n| [Deequ](https://github.com/awslabs/deequ) | “unit tests for data” on Spark | large-scale data verification in Spark-centric environments |\n| [ydata-profiling](https://github.com/ydataai/ydata-profiling) | one-line profiling / EDA | fast exploratory profiling and shareable analysis reports |\n\nDataSentry is intentionally not trying to replace a metadata catalog, lineage platform, or every validator. Its focus is narrower: **find bad data, show why it was flagged, and close the repair loop without gambling on the source.**\n\n## Local-first by design\n\n- deterministic detection and scoring run locally;\n- DuckDB powers core local execution;\n- OpenAI/Ollama assistance is optional;\n- PII redaction, encrypted mappings, and LLM audit records are available when AI is enabled;\n- the original source file is not overwritten by repair workflows.\n\n## Data sources\n\n- CSV, Parquet, JSONL, XLSX\n- DuckDB and SQLite\n- PostgreSQL and MySQL\n- `s3://`, `gs://`, and `az://` objects\n- single files, batches, and globs\n\n## History and drift\n\nPersist scans and compare data over time:\n\n```bash\ndatasentry drift latest orders\ndatasentry score\n```\n\nTracked signals include schema changes, row-count movement, quality-score changes, and issue-distribution drift.\n\n## Architecture\n\n```mermaid\nflowchart LR\n    Sources[Files / DBs / cloud objects] --> DuckDB[Local execution]\n    DuckDB --> Detect[39 detectors]\n    Detect --> Evidence[Evidence fusion]\n    Evidence --> Score[6-dimension score]\n    Score --> Reports[Reports / history / gates]\n    Reports --> CLI[CLI / TUI]\n    Reports --> Web[Web / REST]\n    Reports --> MCP[MCP]\n    Evidence --> Proposal[Repair proposal]\n    Proposal --> Preview[Preview]\n    Preview --> Apply[Apply to copy]\n    Apply --> Verify[Verify by re-scan]\n    Verify --> Rollback[Rollback artifact]\n    LLM[Optional OpenAI / Ollama] -. proposes .-> Proposal\n```\n\n## Reproducible benchmark\n\n```bash\nuv sync\nuv run python benchmarks/bench_scan.py 1000000 42\n```\n\nThe benchmark generates synthetic dirty data and measures profiling, detection/fusion/scoring, numeric-outlier detection, JSONL reading, sampling, score drift, and memory high-water marks. See [`docs/BENCHMARKS.md`](docs/BENCHMARKS.md).\n\n## Development\n\n```bash\nuv sync\nmake check          # lint + mypy --strict + tests/coverage\nmake demo           # exercise the public datasentry demo path\nmake bench          # benchmark\nmake build          # distributions\n```\n\n## Contributing\n\nUseful contributions include new detectors, connectors, integration examples, reproducible benchmark cases, documentation/translations, minimal bug reproductions, and CLI/TUI/Web usability improvements.\n\nWant a small first contribution? Start with [`good first issue #14`](https://github.com/Jackxiaozhiren/datasentry/issues/14) or [`good first issue #15`](https://github.com/Jackxiaozhiren/datasentry/issues/15).\n\nSee [`CONTRIBUTING.md`](CONTRIBUTING.md), [`CODE_OF_CONDUCT.md`](CODE_OF_CONDUCT.md), [`SECURITY.md`](SECURITY.md), and [`ROADMAP.md`](ROADMAP.md). Small contributions are tracked with **`good first issue`** and **`help wanted`** labels.\n\n## Documentation\n\n- [Project site and live report](https://jackxiaozhiren.github.io/datasentry/)\n- [`docs/FAQ.md`](docs/FAQ.md) — evaluation, safety boundaries, and how DataSentry fits alongside other data-quality tools\n- [`examples/`](examples/) — scenario-first runnable examples\n- [`docs/MCP.md`](docs/MCP.md) — VS Code and Claude Desktop MCP setup\n- [`docs/GITHUB_ACTIONS.md`](docs/GITHUB_ACTIONS.md) — reusable GitHub quality gate\n- [`examples/integrations/github-actions/`](examples/integrations/github-actions/) — copy-paste CI gate\n- [`docs/BENCHMARKS.md`](docs/BENCHMARKS.md) — benchmark policy\n- [`docs/DEVELOPMENT.md`](docs/DEVELOPMENT.md) — engineering notes\n- [Detect → fix → verify](.growth/blog-3-repair-loop-en.md) / [中文版](.growth/blog-3-repair-loop-zh.md)\n\n## License\n\nApache-2.0 — see [`LICENSE`](LICENSE).\n\n---\n\n<p align=\"center\">\n  If DataSentry helps you catch bad data before it reaches production, consider giving the repository a ⭐.<br>\n  It helps other data engineers discover the project.\n</p>\n",
  "bytes": 12017,
  "sha": "f90fa3bc7a2527b304379475a5061a735f48547a98ea827ef9e115af5b90bc41",
  "repo_slug": "jackxiaozhiren/datasentry",
  "fonte": "repo",
  "truncated": false,
  "api": "https://agentalog.com/api/listings/mcp_io_github_jackxiaozhiren_datasentry_d2d637c6/readme"
}