{ "markdown": "\n
\n\n\"Python\n\"PyPI\"\n\"PyPI\n\"MIT\n\"CI\"\n\"Glama\n\"MCP\n\n

\n\n\"Scherlok\"\n\n

Scherlok

\n\n

Your data broke in production. Again.
\nScherlok makes sure it doesn't happen next time.

\n\n
\n\n
\n\n\"Scherlok\n\n**Zero config. Zero YAML. Zero rules to write.**
\nScherlok learns what \"normal\" looks like, then tells you when something changes.\n\n
\n\n---\n\n## The Problem\n\nEvery data team has the same nightmare:\n\n> A source API silently changes from **dollars to cents**. Revenue dashboards show wrong numbers for **3 weeks** before anyone notices.\n>\n> A column starts returning **NULLs**. A table stops updating. Row counts drop **40% on a Tuesday**. Nobody knows until the CEO asks why the report looks weird.\n\nCurrent tools (Great Expectations, Soda, dbt tests) require you to **define what \"correct\" looks like** before you can detect what's wrong. Hundreds of rules. Dozens of YAML files. And you still miss things — because you can't write rules for problems you haven't imagined yet.\n\n## The Solution\n\nScherlok takes the opposite approach: **learn first, then detect.**\n\n```bash\nscherlok connect postgres://user:pass@host/db # connect once\nscherlok investigate # learn your data\nscherlok watch # detect anomalies\n```\n\nThree commands. Five minutes. Done.\n\nAfter five valid profiles, Scherlok learns per-metric variability from the\nlatest 30 profiles using robust historical baselines for volume, numeric mean\nshifts, NULL rates, and distinct counts. During cold start or when history is\nnot usable, it keeps the conservative fixed defaults.\n\n## What It Catches\n\n| Anomaly | What Happened | Severity |\n|---------|---------------|----------|\n| **Volume drop** | Row count dropped 40% overnight | CRITICAL |\n| **Volume spike** | 3x more rows than normal | WARNING |\n| **Freshness alert** | Table hasn't updated in 12h (normally every 2h) | CRITICAL |\n| **Schema drift** | Column removed or type changed | CRITICAL |\n| **NULL surge** | NULL rate jumped from 2% to 45% | WARNING |\n| **Distribution shift** | Column mean shifted 3+ standard deviations (Shewhart-style control limit) | INFO, WARNING above 5σ |\n| **Cardinality explosion** | Status column went from 5 values to 500 | CRITICAL |\n\nEvery anomaly is auto-scored: **INFO**, **WARNING**, or **CRITICAL**. No thresholds to configure.\n\n## Works with dbt\n\nAlready running dbt? Scherlok complements `dbt test` with **automatic** anomaly detection — no rules to write.\n\n```bash\npip install scherlok[dbt]\n\n# After `dbt run`, point Scherlok at your project\nscherlok dbt --project-dir ./my_dbt_project\n```\n\nScherlok reads `target/manifest.json`, discovers every materialized model (`table`, `incremental`, `view`), auto-resolves the connection from your `profiles.yml`, and profiles each model:\n\n```\nInvestigating 4 dbt models in ./my_dbt_project (postgres)\n ✓ stg_customers (12,345 rows)\n ✓ stg_orders (98,765 rows)\n ✗ fct_orders CRITICAL: Row count dropped 42% (98,765 → 57,283)\n ✓ dim_customers_inc (12,300 rows)\n\nSummary: 4 profiled, 1 anomalies (1 critical, 0 warning)\n```\n\nUse it as a CI gate after `dbt run`:\n\n```yaml\n- run: dbt run --target prod\n- run: scherlok dbt --project-dir . --target prod --fail-on critical\n```\n\nOr collapse both steps into one with the wrapper:\n\n```yaml\n- run: scherlok dbt-run-and-watch --project-dir . --target prod --fail-on critical\n```\n\nThe wrapper runs `dbt run` by default and uses the successful model nodes recorded in\n`target/run_results.json`, so partial runs profile only what dbt actually built. Use\n`--build` to run `dbt build`; successful models are still profiled when a test failure\ncauses downstream models to be skipped on dbt's handled failure path (exit 1), while\nthe wrapper preserves `dbt build`'s exit code. Unhandled failures fail fast without\nreading the artifact.\n\nBoth `dbt` and `dbt-run-and-watch` accept `--output json` for CI parsers — a single JSON document on stdout, nothing else.\n\n**Supported adapters:** `postgres`, `bigquery`, `snowflake`, `mysql`, `duckdb`. For others, pass `--connection-string` explicitly.\n\n📖 Full docs: [dbt integration guide →](src/scherlok/dbt/README.md)\n\n## dbt Package — native tests\n\nPrefer staying inside dbt? Install Scherlok as a dbt package for native data quality tests — no Python CLI needed.\n\n```yaml\n# packages.yml\npackages:\n - package: rbmuller/scherlok\n version: [\">=1.0.0\", \"<2.0.0\"]\n```\n\n```yaml\n# schema.yml\nmodels:\n - name: fct_orders\n tests:\n - scherlok.volume_anomaly:\n sensitivity: 3.0\n - scherlok.row_count_between:\n min_value: 100\n columns:\n - name: email\n tests:\n - scherlok.not_null_proportion:\n max_rate: 0.01\n - name: updated_at\n tests:\n - scherlok.recency:\n days: 2\n```\n\n**Tier 1 — Instant (no setup):** `not_null_proportion`, `row_count_between`, `recency`, `unique_proportion`\n\n**Tier 2 — Auto-learning (Shewhart control limits):** `volume_anomaly`, `null_anomaly` — require the `scherlok_metrics` model to build baseline history.\n\n📖 Full docs: [dbt package README →](models/_models.yml)\n\n## HTML dashboard\n\n![scherlok dashboard](assets/dashboard-screenshot.png)\n\n```bash\nscherlok dashboard --out report.html\n```\n\nOne self-contained HTML file (~28 KB): KPIs, per-table incidents grouped with first-seen timestamps, `+`/`−`/`~` schema-drift diff, sparklines, and full anomaly history. Auto dark/light theme via `prefers-color-scheme`.\n\n📖 Full docs: [dashboard guide →](src/scherlok/dashboard/README.md)\n\n## Use it from an AI agent (MCP)\n\nLet Claude Code / Claude Desktop run data-quality checks directly:\n\n```bash\npip install scherlok # scherlok-mcp ships built-in since v0.7.0\n```\n\n```json\n{\n \"mcpServers\": {\n \"scherlok\": {\n \"command\": \"scherlok-mcp\",\n \"env\": { \"SCHERLOK_CONNECTION\": \"postgresql://user:pass@host/db\" }\n }\n }\n}\n```\n\nThe agent gets `list_tables`, `investigate`, `watch`, `status`, `history`, and `check` as tools. Credentials are resolved server-side (never passed by the model), every operation is read-only on the warehouse, and there's no arbitrary-SQL tool.\n\n📖 Full docs: [MCP server guide →](src/scherlok/mcp/README.md)\n\n## AI-explained alerts (`--explain`)\n\nYour alert says **what** broke. `--explain` adds **why** — and what to check next.\n\n```bash\npip install 'scherlok[explain]'\nexport ANTHROPIC_API_KEY=sk-ant-...\n\nscherlok watch --webhook https://hooks.slack.com/... --explain\n```\n\nWhen anomalies fire, Scherlok makes **one** Claude call for the whole batch and injects a short root-cause hypothesis into the same Slack/Discord/Teams/email/JSON alert:\n\n
\n\"scherlok\n
\n\nWorks on `watch`, `ci`, `check`, `dbt`, and `dbt-run-and-watch`. On dbt projects the hypothesis is **lineage-aware**: upstream parents from `manifest.json` go into the prompt, so cascading failures get traced to the source model instead of alerting on every downstream symptom.\n\n- **What it costs** — one call per fired run (not per anomaly), Claude Haiku 4.5 by default: well under a cent per run (~$0.003). Override the model with `SCHERLOK_EXPLAIN_MODEL`. Runs with zero anomalies make no API call.\n- **What it sends** — aggregates only: the anomaly type/severity/message strings already in your alert, dbt model names, detection timestamps. **Never** warehouse rows, cell values, or credentials — the test suite pins this as a contract.\n- **How to turn it off** — it's opt-in; don't pass `--explain`. If the API call fails (no key, timeout, rate limit), the original alert is delivered unchanged with a one-line note. Alerting never blocks on the LLM.\n\n📖 Full docs: [explainer guide →](src/scherlok/explainer/README.md)\n\n## How It Works\n\n### 1. `investigate` — Learn the patterns\n\n```bash\n$ scherlok investigate\n\n Profiling 12 tables...\n ✓ users — 45,231 rows, 8 columns\n ✓ orders — 1,203,847 rows, 15 columns\n ✓ products — 892 rows, 12 columns\n ...\n Done. Profiles saved.\n```\n\nScherlok profiles every table: row counts, column types, NULL rates, value distributions, freshness cadence, cardinality. Stores everything locally in SQLite.\n\n### 2. `watch` — Detect anomalies\n\n```bash\n$ scherlok watch\n\n Checking 12 tables against learned profiles...\n\n 🔴 CRITICAL orders volume_drop Row count dropped 52% (1,203,847 → 578,412)\n 🟡 WARNING users null_increase Column \"email\": NULL rate 2.1% → 18.7%\n 🔵 INFO products distribution Column \"price\": mean shifted 3.2σ\n\n 3 anomalies detected. Exit code: 1\n```\n\n### 3. Alert — Slack, CI/CD, or both\n\n```bash\n# Slack\nscherlok watch --webhook https://hooks.slack.com/services/...\n\n# Discord\nscherlok watch --webhook https://discord.com/api/webhooks/...\n\n# Microsoft Teams\nscherlok watch --webhook https://outlook.office.com/webhook/...\n\n# Any endpoint (generic JSON payload)\nscherlok watch --webhook https://my-api.com/alerts\n\n# CI/CD gate (fails pipeline on CRITICAL)\nscherlok watch --exit-code --fail-on critical\n```\n\nAuto-detects Slack, Discord, and Teams from the URL and formats the payload accordingly. Any other URL receives a generic JSON payload.\n\n## CI/CD Integration\n\nUse Scherlok as a data quality gate. The `ci` command does it in one line:\n\n```yaml\n# GitHub Actions\n- name: Data quality check\n run: |\n pip install scherlok\n scherlok config --store s3://my-bucket/scherlok/profiles.db\n scherlok ci ${{ secrets.DATABASE_URL }} \\\n --webhook ${{ secrets.SLACK_WEBHOOK }} \\\n --fail-on critical\n```\n\nIf Scherlok detects a critical anomaly, the pipeline fails. Bad data never reaches production.\n\n## Email alerts\n\n```bash\nexport SCHERLOK_SMTP_HOST=smtp.gmail.com\nexport SCHERLOK_SMTP_USER=alerts@company.com\nexport SCHERLOK_SMTP_PASSWORD=app-specific-password\n\nscherlok watch --email team@company.com --email cto@company.com\n```\n\n## Connectors\n\n```bash\n# PostgreSQL\nscherlok connect postgres://user:pass@host:5432/db\n\n# BigQuery — see src/scherlok/connectors/bigquery.md for auth, billing, CI patterns\npip install scherlok[bigquery]\nscherlok connect bigquery://project-id/dataset-name\n\n# Snowflake\npip install scherlok[snowflake]\nexport SNOWFLAKE_USER=...\nexport SNOWFLAKE_PASSWORD=...\nexport SNOWFLAKE_WAREHOUSE=...\nscherlok connect snowflake://account/database/schema\n\n# MySQL\npip install scherlok[mysql]\nscherlok connect mysql://user:pass@host:3306/dbname\n\n# DuckDB\npip install scherlok[duckdb]\nscherlok connect duckdb:///path/to/file.db\n```\n\n| Database | Status |\n|----------|--------|\n| PostgreSQL | Available |\n| BigQuery | Available |\n| Snowflake | Available |\n| MySQL | Available |\n| DuckDB | Available |\n\n## Remote Storage\n\nShare profiles across CI runs and team members:\n\n```bash\n# AWS S3\nscherlok config --store s3://my-bucket/scherlok/profiles.db\n\n# Google Cloud Storage\nscherlok config --store gs://my-bucket/scherlok/profiles.db\n\n# Azure Blob Storage\nscherlok config --store az://my-container/scherlok/profiles.db\n```\n\n## Why Not [Other Tool]?\n\n| | Great Expectations | Soda | Monte Carlo | **Scherlok** |\n|---|---|---|---|---|\n| Setup time | Hours | 30 min | Weeks | **5 minutes** |\n| Config required | Hundreds of rules | YAML checks | Dashboard setup | **None** |\n| Anomaly detection | Manual thresholds | Paid feature | Yes | **Yes, free** |\n| Self-hosted | Yes | Limited | No (SaaS) | **Yes** |\n| CI/CD gate | Yes | Yes | No | **Yes** |\n| Price | Free | Freemium | $50-200K/yr | **Free, forever** |\n\n## CLI Reference\n\n```\nscherlok connect Connect to a database\nscherlok investigate Profile all tables (learn patterns)\nscherlok watch [-w ] [-e ] Detect anomalies and alert\nscherlok ci [opts] All-in-one CI/CD command (connect + watch + exit code)\nscherlok dbt [--project-dir .] [--output json] Profile dbt models from manifest\nscherlok dbt-run-and-watch [--build] [--output json] Run dbt + profile in one step\nscherlok status [--output json] Quick health dashboard\nscherlok history [--days N] [--output json] Timeline of past anomalies\nscherlok report Detailed profile summary\nscherlok dashboard [--out .html] Generate self-contained HTML report\nscherlok config --store Set remote storage\nscherlok version Show version\n```\n\n## Install\n\n```bash\npip install scherlok\n\n# With BigQuery support\npip install scherlok[bigquery]\n```\n\nRequires Python 3.10+.\n\n### Run via Docker\n\nA pre-built image with every warehouse extra (`dbt`, `bigquery`, `snowflake`) is published to GitHub Container Registry on every release tag:\n\n```bash\ndocker run --rm ghcr.io/rbmuller/scherlok:latest version\n```\n\nMount your project directory and inject connection details the same way your CI does it; the entrypoint is the `scherlok` CLI:\n\n```bash\ndocker run --rm \\\n -v \"$PWD:/work\" -w /work \\\n -e SCHERLOK_CONNECTION=postgres://... \\\n ghcr.io/rbmuller/scherlok:latest watch\n```\n\nThe image is built from `python:3.12-slim` and runs unprivileged (`USER scherlok`).\n\n## Contributing\n\nContributions welcome! See [CONTRIBUTING.md](CONTRIBUTING.md).\n\nWe're especially looking for:\n- New database connectors (e.g. Databricks — see [#37](https://github.com/rbmuller/scherlok/issues/37))\n- Anomaly detection improvements\n- Documentation and examples\n\n## License\n\n[MIT](LICENSE) — Developed by [Robson Bayer Müller](https://github.com/rbmuller)\n", "bytes": 14660, "sha": "23729d2cc6fde27029689367c4b9a7616af16e1a9ff0c333abaee90ab0d24ca2", "repo_slug": "rbmuller/scherlok", "fonte": "repo", "truncated": false, "api": "https://agentalog.com/api/listings/mcp_io_github_rbmuller_scherlok_e6e79fc5/readme" }