{
  "markdown": "# obsify\n\n[![CI](https://github.com/Formative-Sum41/obsify/actions/workflows/ci.yml/badge.svg)](https://github.com/Formative-Sum41/obsify/actions/workflows/ci.yml)\n[![PyPI](https://img.shields.io/pypi/v/obsify.svg)](https://pypi.org/project/obsify/)\n![Python](https://img.shields.io/badge/python-3.11%2B-blue.svg)\n[![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)\n\n<!-- mcp-name: io.github.Formative-Sum41/obsify -->\n\n**Let an AI assistant work on sensitive files without their raw values ever entering the model's context.**\n\nobsify is a local, deterministic [MCP](https://modelcontextprotocol.io) server. The frontier\nmodel reasons over **shape** — schemas, synthetic twins, masked feedback — while deterministic\nlocal code touches the **substance** and returns only masked, aggregated results. No LLM calls,\nno network at runtime: detection is regex + checksums + dictionaries + [Presidio](https://github.com/microsoft/presidio)'s\nlocal NER.\n\nIt ships with Australian entity support (ABN / ACN / TFN, checksum-validated),\n**credential/secret detection** (cloud keys, API tokens, private keys, DB connection\nstrings), and a label-driven **routing layer** that makes \"when should the assistant avoid\nraw data\" a deterministic, enforced decision rather than a judgement call.\n\n> **Honest scope:** `run_on_real` executes model-written code in a *best-effort* local sandbox\n> and masks its output *best-effort*. It is not a jail. Read [`SECURITY.md`](SECURITY.md) before\n> pointing it at anything you cannot afford to leak. Return aggregates.\n\n## Why\n\nFeeding confidential documents to a hosted LLM means the substance leaves your perimeter. The\nusual answers are \"don't use the LLM\" or \"trust the provider.\" obsify takes a third path —\n**compute-to-data**: bring the code to the data, not the data to the model.\n\n- The model sees the **schema** of a spreadsheet, not its rows.\n- The model develops against a **synthetic twin** (faked values, real structure).\n- The model's analysis code runs **locally**; only masked, aggregated output returns.\n\nThe frontier model's reasoning is preserved. Only its *eyes on raw values* are removed.\n\n## Tools\n\n| Tool | What it does | Returns |\n|---|---|---|\n| `scan_pii(path)` | Scan a file/folder for PII | Types, locations, counts — **never values** |\n| `make_synthetic_twin(path, out)` | Faithful fake of an Excel workbook | Schema summary; twin written to `out` (values faked, leak-verified) |\n| `run_on_real(code, data_path)` | **Compute-to-data**: run your code locally against the real file (bound to `DATA_PATH`) | Only PII-masked, size-capped stdout/stderr — **return aggregates** |\n| `redact_text(text)` | Mask PII in a string to `<TYPE>` tokens | The redacted string |\n| `verify_value_free(text, terms)` | Fail-closed check that `text` leaks none of `terms` (or their variants) | `{\"value_free\": bool}` |\n\n**Supported documents:** PDF (text + tables; complex-table fallback via `obsify[tables]`),\nExcel `.xlsx`/`.xlsm`, and Word `.docx` (paragraphs + tables). Unreadable or unsupported files\nare surfaced as explicit notes/blind spots, never silently dropped. (No OCR yet — scanned/image\npages are flagged as low-coverage, not transcribed.)\n\n**Known-entity masking (optional).** Supply a local `.obsify.entities` list of names to hide;\n`scan_pii` / `redact_text` deterministically catch them — and the suffix/abbreviation variants\nNER misses (`BRIGHTWATER HLDGS P/L` for *Brightwater Holdings Pty Ltd*) — as `KNOWN_ENTITY`. The\nlist stays local and never enters the model's context. See [`docs/known_entities.md`](docs/known_entities.md).\n\n## Demo\n\nPoke at all five tools live against synthetic data with the official\n[MCP Inspector](https://github.com/modelcontextprotocol/inspector):\n\n```bash\npython -m obsify.make_corpus --out ./corpus_demo\nnpx @modelcontextprotocol/inspector obsify-mcp\n```\n\nCall `scan_pii` on `./corpus_demo/ledger.xlsx` and confirm it returns types / counts /\nlocations only — never values. See [`docs/verifying.md`](docs/verifying.md).\n\n## Try it — synthetic corpus\n\nGenerate a fake-but-realistic corpus (all synthetic; ABN/ACN/TFN are checksum-valid) spanning\nall three formats, then point a tool at it:\n\n```bash\npip install \"obsify[demo]\"                 # reportlab, for the sample PDFs\npython -m obsify.make_corpus --out ./corpus_demo\n```\n\nIt writes a multi-sheet Excel ledger (a numeric false-positive minefield), a PDF engagement\nletter (prose + trial-balance table), and a DOCX audit memo (paragraphs + vendor table). Great\nfor kicking the tyres on `scan_pii` / `make_synthetic_twin` without touching real data.\n\n## Install & run as an MCP server\n\nRequires Python 3.11+. obsify speaks MCP over **stdio** — the client launches it as a local\nsubprocess; nothing is hosted remotely. Register it with any MCP-capable client (Claude\nDesktop, Claude Code, Cursor, VS Code, …) by adding one block to that client's config.\n\n**Recommended — zero-install via [uvx](https://docs.astral.sh/uv/):**\n\n```json\n{ \"mcpServers\": { \"obsify\": { \"command\": \"uvx\", \"args\": [\"--from\", \"obsify\", \"obsify-mcp\"] } } }\n```\n\n`uvx` fetches obsify from PyPI and runs it on demand — no permanent install. On **first run**,\nobsify downloads the spaCy NER model (`en_core_web_lg`, ~560 MB) once and caches it; this\nfetches a public model and sends no user data (set `OBSIFY_AUTO_DOWNLOAD=0` to forbid it and\ninstall the model yourself). Later runs are instant and fully offline.\n\n**Or install it (pip / pipx):**\n\n```bash\npipx install obsify        # isolated, on PATH  (or: pip install obsify)\n```\n\nThen point the client at the installed command:\n\n```json\n{ \"mcpServers\": { \"obsify\": { \"command\": \"obsify-mcp\" } } }\n```\n\nRestart the client and the tools appear. Optional extras: `obsify[tables]` (complex-table PDF\nfallback via camelot + Ghostscript), `obsify[compute]` (pandas, handy inside `run_on_real` code).\n\n> **PATH gotcha (the #1 cause of \"server won't connect\"):** the `command` must resolve on the\n> PATH the *client* sees. A GUI client may not share your venv's PATH. Fixes: use `uvx`/`pipx`\n> (globally resolvable), or give an absolute path — `\"/path/to/.venv/bin/obsify-mcp\"` (macOS/Linux)\n> or `\"C:\\\\path\\\\to\\\\.venv\\\\Scripts\\\\obsify-mcp.exe\"` (Windows).\n\n**From this repo (before it's on PyPI):**\n\n```bash\npip install \"git+https://github.com/Formative-Sum41/obsify.git\"   # gets `obsify-mcp` + `obsify`\n```\n\n## The routing layer — deterministic, not a judgement call\n\nThe hard part of \"help me, but don't read the confidential file\" is *deciding when to protect*.\nobsify moves that decision out of the model and into the environment:\n\n1. **`.obsify.json`** — a label manifest classifying paths (`public` / `confidential` / `restricted`).\n2. **`obsify.guard`** (run as `python -m obsify.guard`) — a PreToolUse guard that blocks a direct read of a labelled file\n   (exit 2) and redirects the assistant to `scan_pii` / `make_synthetic_twin` / `run_on_real`.\n3. **A convention** (in `CLAUDE.md`) so the assistant *prefers* obsify before it even hits the guard.\n\nSet it up with one command:\n\n```bash\nobsify init [--dir PATH] [--with-claude-md]\n```\n\n`obsify init` is **non-destructive by design** — it owns exactly one file and hands you snippets\nfor the rest:\n\n- **`.obsify.json`** — obsify owns this; init writes it (never overwritten without `--force`).\n- **`.claude/settings.json`** — *your* file: init **prints** the PreToolUse hook block to paste,\n  never edits it (it runs code, so registering it is your call).\n- **`CLAUDE.md`** — *your* file: the convention is **opt-in**. Default prints it; `--with-claude-md`\n  appends a marker-wrapped, idempotent block that never clobbers your content.\n\nFull convention: [`docs/obsify_routing.md`](docs/obsify_routing.md).\n\n## How detection stays precise\n\n- **Checksum-validated identifiers.** ABN/ACN/TFN candidates are proposed by regex and confirmed\n  by their official checksums, so a random number is never reported as an identifier.\n- **Context-required IDs.** A bare number is only accepted as an ABN/ACN/TFN when a label word\n  (\"TFN\", \"ABN\", \"BSB\", …) is nearby — this kills the sequential-journal-ID false-positive flood\n  on numeric ledgers.\n- **Letterless / NER-with-digit suppression.** Pure numbers, amounts, dates and alnum codes are\n  not flagged as names/orgs; real names, emails and addresses (which carry letters) are unaffected.\n  Validated letterless PII stays exempt: checksum IDs (ABN/ACN/TFN/Medicare), Luhn cards, valid\n  IPs, BSB-adjacent accounts, and phones (via context or phone shape) — while a decimal point\n  still marks an amount, not a phone.\n- **Credentials, not just PII.** Cloud keys (AWS/GitHub/Google/Slack/Stripe), JWTs, private-key\n  blocks and DB connection strings are flagged as `CREDENTIAL` by *anchored* patterns — vendor\n  prefixes (`AKIA…`, `ghp_…`) or a keyword-gated `secret = <value>`, never entropy heuristics\n  (which would flood on hex/base64 ledger columns). The whole `BEGIN…END` private-key block is\n  masked, not just its header, so no key body is left behind.\n\n## Measured accuracy\n\nobsify ships a scored evaluation harness (`eval/` — labelled synthetic corpus + answer key +\nscorer against the *shipping* detector, plus an independent third-party cross-check). Headline\non the synthetic corpus: **100% recall** on expected-detect items, **0 false positives** on a\nnumeric FP-torture sheet (with a grouped-number guard), bare context-gated IDs correctly\nsuppressed. Independent cross-check vs Microsoft `presidio-research`: EMAIL/IBAN 100%, PERSON 94%.\n\n**The harness earned its keep — it found real defects, which were then fixed:** credit cards and\nphone numbers were being silently suppressed by the numeric-noise filter (now exempt via checksum\nvalidation / phone shape), and Medicare, IP, date-of-birth, AU passport and driver-licence had no\nrecognizer (now added, checksum- or context-gated). Full method, numbers, and remaining documented\ngaps (SWIFT/BIC, non-DOB dates): [`eval/README.md`](eval/README.md).\n\n## Tests\n\n```bash\npip install -e \".[dev]\"\npytest tests/            # or run any file directly: python tests/test_obsify.py\n```\n\nThirteen suites (88 tests), run in CI on Linux + Windows / Python 3.11 + 3.12:\n\n- **mcp-protocol** — launches the real server over stdio and speaks MCP to it (the same path a\n  client like Claude uses): confirms all five tools register with valid schemas and that calls\n  round-trip through JSON-RPC — including `scan_pii` returning **shape only, end to end**.\n- **checksums** — anchored to externally-published ABN/ACN/TFN worked examples (valid and\n  corrupted), which breaks the generator↔validator circularity.\n- **obsify / twin / redaction** — the privacy invariants: shape-only output, leak-free twins,\n  and a fail-closed self-check.\n- **precision** — the false-positive suppressors kill numeric-ledger noise while keeping real names.\n- **credentials** — the anchored secret patterns catch cloud keys / tokens / JWTs / private-key\n  blocks / connection strings, while keyword-anchored generics stay precise on prose (no entropy).\n- **routing** — the guard's block/allow classification and `obsify init`'s non-destructive contract.\n- **corpus** — the synthetic PDF+Excel+DOCX corpus end to end: per-format detection, DOCX\n  paragraph+table extraction, and shape-only output across every format.\n- **evaluation** — the scored harness as a regression gate (recall, suppression, FP-torture, gaps).\n- **robustness** — graceful degradation: corrupt/oversized/empty/nested/unsupported inputs never\n  crash and are always surfaced as notes.\n- **model / variants** — first-run model auto-download logic; variant normalization behind `verify_value_free`.\n\nFor interactive verification (MCP Inspector) and the live-client last-mile check, see\n[`docs/verifying.md`](docs/verifying.md).\n\n## Related work\n\nobsify is one of several MCP servers tackling \"let an AI touch sensitive data safely\" — they're\nmostly **complementary**, solving the same problem from different ends. Worth knowing where each fits:\n\n| Tool | Approach | Best for |\n|---|---|---|\n| **obsify** | Detection + **shape-isolation**: the model sees only shape, synthetic twins and masked aggregates — never the values (real *or* faked) | Messy, unstructured docs (PDF/Excel/DOCX) where you can't enumerate PII up front; strict value isolation; enforcement of *when* to protect |\n| [cloakbox](https://glama.ai/mcp/servers/mbufkin/cloakbox) | Policy-driven **pre-sanitization**: tokenize a database into a de-identified copy the model queries freely | Known, structured schemas where you want rich analytics (joins/aggregations) on a referentially-intact clean copy |\n| [redact-mcp](https://glama.ai/mcp/servers/r3352/redact-mcp) | **Reversible obfuscation** proxy: the model works on consistent fakes; a proxy tool round-trips real API calls | Pentest workflows and **secrets**, where the model must operate on realistic data and you restore reals later |\n| [cms-ai](https://glama.ai/mcp/servers/aammasa/cms-ai-pii-redaction-service) | Enterprise redaction **service**: Presidio + spaCy behind REST/MCP, multi-language, scalable | A hosted, multi-language redaction API with a UI and horizontal scale |\n\n**Where obsify is distinct:** it's the only one of these where the model gets *neither* raw values *nor*\na full mirror to operate on — just **shape + masked aggregates** — combined with checksum-validated\nidentifiers, credential detection, a deterministic **routing guard**, and a hard no-network / no-LLM\nguarantee. That's the strictest-isolation end of the spectrum, tuned for confidential financial documents.\n\n**Honest trade:** obsify optimizes *isolation of the values* over *utility on the data*. If you need\nreferentially-intact analytics on a clean copy (cloakbox), reversible round-tripping (redact-mcp), or a\nmulti-language hosted service (cms-ai), those are the better fit — and pair well with obsify rather than\ncompeting with it.\n\n## Contributing\n\nPRs welcome — see [`CONTRIBUTING.md`](CONTRIBUTING.md) for setup, the merge bar, and the\nnon-negotiable invariants (no LLM calls in the library, no runtime network, no real data,\nshape-not-substance). Security issues: [`SECURITY.md`](SECURITY.md) (report privately).\n\n## License\n\nMIT — see [`LICENSE`](LICENSE).\n",
  "bytes": 14262,
  "sha": "8172dc7df25483e78fdbc6ce0bfe803a6c6c3feab8189f214fa759cb87ed70d3",
  "repo_slug": "formative-sum41/obsify",
  "fonte": "repo",
  "truncated": false,
  "api": "https://agentalog.com/api/listings/mcp_io_github_formative_sum41_obsify_b1f44995/readme"
}