{
  "markdown": "# defluff\n\n> *The deterministic slop check for AI-generated prose. Point it at a changelog, a doc, or an agent's own output and get back the filler phrases to cut — plus a CI exit code and a pinnable score, identical on every run. No model, no API key.*\n\n[![CI](https://github.com/ahmedak/defluff/actions/workflows/ci.yml/badge.svg)](https://github.com/ahmedak/defluff/actions/workflows/ci.yml)\n[![PyPI](https://img.shields.io/pypi/v/defluff)](https://pypi.org/project/defluff/)\n[![Python 3.10+](https://img.shields.io/badge/python-3.10%2B-blue)](https://pypi.org/project/defluff/)\n[![License: MIT](https://img.shields.io/badge/license-MIT-green)](LICENSE)\n[![Last commit](https://img.shields.io/github/last-commit/ahmedak/defluff)](https://github.com/ahmedak/defluff/commits/main)\n\n<img src=\"https://raw.githubusercontent.com/ahmedak/defluff/main/demo/demo-fluffy.gif\" width=\"700\" alt=\"Fluffy text gets flagged: defluff lints a sentence full of buzzwords and clichés, returning the slop spans and an over-threshold score.\">\n\nEvery flagged span carries no information, so cutting them loses nothing. Clean text, same tool, passes straight through:\n\n<img src=\"https://raw.githubusercontent.com/ahmedak/defluff/main/demo/demo-clean.gif\" width=\"700\" alt=\"Clean text passes: defluff lints a plain, concrete sentence and returns a 0% slop score.\">\n\nWhat makes defluff worth installing over a one-off `grep` is the [engine around the list](#the-list-isnt-what-this-is-about--the-engine-is): bring your own phrases, per-project overlays, and an MCP server your agents pick up with no wiring.\n\n---\n\n## Install\n\n```bash\npip install defluff\n```\n\nOr on macOS/Linux via Homebrew:\n\n```bash\nbrew install ahmedak/defluff/defluff\n```\n\nThat's it. No model download. No API key. Runs anywhere Python does.\n\n---\n\n## Quick start\n\n```bash\n# Lint a file — exit 1 on slop, 0 when clean\ndefluff lint essay.md\n\n# Pipe text\ncat draft.md | defluff lint\n\n# Get a bare score for scripts (0.0 – 1.0)\ndefluff score essay.md\n\n# Machine-readable JSON for downstream tooling\ndefluff lint essay.md --json\n```\n\n---\n\n## MCP server\n\nExposes three tools so any MCP-aware agent can self-check prose without bespoke wiring — including its own draft, before returning it.\n\nZero-install via `uvx` (recommended) — pulls the package and the `mcp` extra on first run:\n\n```json\n{\n  \"mcpServers\": {\n    \"defluff\": {\n      \"command\": \"uvx\",\n      \"args\": [\"--from\", \"defluff[mcp]\", \"defluff-mcp\"]\n    }\n  }\n}\n```\n\nOr install it and run the entry point directly:\n\n```bash\npip install \"defluff[mcp]\"\ndefluff-mcp\n```\n\n```json\n{\n  \"mcpServers\": {\n    \"defluff\": { \"command\": \"defluff-mcp\" }\n  }\n}\n```\n\nPublished to the [MCP Registry](https://registry.modelcontextprotocol.io) as `io.github.ahmedak/defluff` (see `server.json`).\n\n<!-- ownership marker for the MCP Registry; do not remove -->\nmcp-name: io.github.ahmedak/defluff\n\n| Tool | Args | Returns |\n|------|------|---------|\n| `slop_detect` | `text: str` | `slop_score`, `spans` (text, category, weight, offsets), `categories`, `lexicon_version` |\n| `slop_add` | `pattern: str, category: str, scope: \"user\"\\|\"project\"` | adds a phrase to the lexicon overlay |\n| `slop_ignore` | `pattern: str, scope: \"user\"\\|\"project\"` | suppresses a phrase (e.g. domain jargon) |\n\n---\n\n## Common use cases\n\n- **Agent self-correction** — call `slop_detect` on a draft and revise the flagged phrases before returning it. Zero wiring, one session, no second model in the loop.\n- **CI gate on generated content** — fail the build when an AI-drafted changelog or doc ships full of \"furthermore\" and \"robust.\" Deterministic + exit codes + a [pinnable lexicon](#lexicon-overlays-and-versioning) is what an LLM-judge gate can't give you.\n- **Writing assistant feedback** — highlight the exact phrases an editor would cut, instead of a vague \"this sounds AI.\"\n- **A reward *component* for fine-tuning** — see [reward loops](#use-in-reward-loops-experimental) for caveats; on its own it's gameable.\n\n---\n\n## Why defluff?\n\nEvery \"AI detector\" tries to classify *whether* text was AI-generated — a hard, unreliable problem. defluff asks a different question: **does this text contain removable filler?** That's deterministic, and it's true whether a human or an LLM wrote \"at the end of the day.\"\n\n`proselint` is the closest prior art — deterministic, no model — but emits yes/no warnings rather than a tunable density score, and isn't built around a list you swap, overlay, or pin.\n\nA `grep` over a word list gives you raw hits. defluff gives you what you'd otherwise have to build around that list:\n\n- **An MCP server** — agents self-check with zero wiring.\n- **Markdown- and code-aware** — strips code fences, inline code, and URLs first.\n- **Whole-word matching** — `\"foster\"` won't fire inside `\"fostering\"`.\n- **A normalized score** instead of a hit count — filler *density*, so one threshold works on a tweet or a 5,000-word doc.\n- **Overlap handling** — \"at the end of the day\" overlapping \"end of the day\" counts each word once (longest-match-wins).\n- **Exit codes, JSON, and char-offset spans** — drop into CI, pre-commit, and editor tooling.\n- **A pinnable lexicon hash** — prove the ruler didn't move between runs.\n\n### Bring your own slop\n\n```bash\ndefluff lint draft.md --lexicon team-slop.md\n```\n\n```\n# team-slop.md\n- circle back\n- low-hanging fruit\n- boil the ocean\nparadigm shift\n```\n\nOne phrase per line (`#` comments and `-`/`*` markers ignored). These layer on top of the built-in defaults and report under a neutral `custom` category. For real categories and per-phrase weights, use a `.json` list (see [Lexicon overlays](#lexicon-overlays-and-versioning)).\n\n### Ready-made domain packs\n\n```bash\ndefluff lint post.md --pack marketing-growth\ndefluff lint post.md --pack marketing-growth,ai-llm   # stack several\n```\n\n| Pack | Catches | Pack | Catches |\n|------|---------|------|---------|\n| `corporate-linkedin` | office jargon | `crypto-web3` | crypto hype |\n| `startup-vc` | pitch-deck speak | `pr-press-release` | press-release boilerplate |\n| `marketing-growth` | hype copy | `academic` | research hedging |\n| `ai-llm` | LLM tells | `wellness-selfhelp` | influencer-speak |\n| `social-media` | X/Twitter engagement-bait | | |\n\nList them with `defluff packs`. High-false-positive terms (e.g. `pivot`, `detox`) ship commented-out so they're inert until you opt in. See the [packs README](src/defluff/data/packs/README.md).\n\n### Batteries-included defaults\n\n~130 curated patterns across five weighted categories, case-insensitive, whole-word matched:\n\n| Category | What it catches | Examples |\n|----------|----------------|---------|\n| `ai-vocab` | Words disproportionately overused by LLMs | *delve, tapestry, nuanced, pivotal, robust, showcase* |\n| `cliche` | Hollow idioms that add no information | *at the end of the day, move the needle, circle back, game changer* |\n| `hedge` | Empty qualifiers | *it should be noted that, needless to say, basically, essentially* |\n| `corporate` | Buzzword inflation | *leverage, synergy, actionable insights, cutting-edge, scalable* |\n| `transition` | Filler connectives LLMs reach for by default | *furthermore, moreover, in conclusion, first and foremost* |\n\n### Rhetorical patterns (beyond the list)\n\nSome AI tells are sentence *shapes*, not fixed phrases — the **antithesis**: `it's not X, it's Y`, `not just a list but a runtime`. A regex pattern layer catches these under a `rhetoric` category:\n\n| Mode | What it looks for | Default | Why |\n|------|-------------------|---------|-----|\n| **Compound** *(confident)* | full shape: `it's not X, it's Y` · `not just X but Y` | **on** | the second clause proves the rhetorical move — rarely a false alarm |\n| **Fragment** *(guessing)* | bare `X, not Y` — e.g. `a signal, not a verdict` | **off**, `--pack rhetoric` | also fires on plain corrections (`shipped Tuesday, not Wednesday`) |\n\nEach match counts as one unit toward the score regardless of length. Turning on fragment mode changes the lexicon hash.\n\n```bash\ndefluff lint draft.md                  # compound antithesis caught by default\ndefluff lint draft.md --pack rhetoric  # also catch the punchy \"X, not Y\" fragment\ndefluff lint draft.md --category rhetoric   # gate CI on antithesis alone\n```\n\n<img src=\"https://raw.githubusercontent.com/ahmedak/defluff/main/demo/demo-rhetoric.gif\" width=\"700\" alt=\"Same sentence run twice: the default catches the compound antithesis; --pack rhetoric also catches the short punchy fragment form.\">\n\n---\n\n## Python API\n\n```python\nimport defluff\n\nreport = defluff.detect(\"It is worth noting that we should leverage synergies.\")\nprint(report.slop_score)   # 0.0 – 1.0\nprint(report.spans)        # flagged phrase locations + categories\n\nscore = defluff.score(text)       # bare float\nclean = defluff.is_slop(text)     # bool at default threshold\n\nlex = defluff.load_lexicon()                # pin for reproducible runs\nscore = defluff.score(text, lexicon=lex)\n```\n\n`SlopReport` fields:\n\n| Field | Type | Notes |\n|-------|------|-------|\n| `slop_score` | `float` | Clamped `[0, 1]` — use for thresholds |\n| `slop_density` | `float` | Raw unclamped — better gradient for reward loops |\n| `spans` | `list[Span]` | Per hit: `text`, category, weight, char offsets *into the cleaned text* (see [Limitations](#limitations)) |\n| `categories` | `dict[str, float]` | Per-category density |\n| `n_words` | `int` | Token count |\n| `low_confidence` | `bool` | `True` when `n_words < 20` |\n| `lexicon_version` | `str` | SHA prefix of resolved entry set |\n\n---\n\n## Lexicon overlays and versioning\n\nThe bundled lexicon is the baseline; layer on top without editing the package:\n\n```bash\n# \"synergy\" is slop on this machine\ndefluff lexicon add \"synergy\" --category corporate --scope user\n\n# \"leverage\" is fine in this repo (finance context) — commit this with the repo\ndefluff lexicon rm \"leverage\" --scope project\ngit add .defluff/ignore.json && git commit -m \"allow 'leverage' in finance context\"\n```\n\n- **User overlay** (`~/.config/defluff/`) — machine-wide, not committed\n- **Project overlay** (`.defluff/` at git root) — per-repo, commit it for your whole team\n\nWrites are atomic and cross-process locked; a corrupt overlay is warned and skipped — `detect()` never crashes.\n\nEvery resolved lexicon (bundled + overlays + packs) carries a short content hash, printed on every run (`lexicon: 2cc05ba84457`) and exposed as `SlopReport.lexicon_version`. Pass `lexicon=defluff.load_lexicon()` once and every call scores against the same ruler — pinnable for CI baselines and RL rewards, and auditable since the hash changes if and only if the resolved entry set changes. Each release ships a dated lexicon with a changelog ([`CHANGELOG.md`](CHANGELOG.md)); `ai-vocab` is expected to turn over release to release, `cliche`/`hedge`/`corporate`/`transition` are near-stable.\n\n---\n\n## Reading the output and setting the threshold\n\n1. **The spans** — the exact filler phrases, with category. Deterministic, reliable. Act on these.\n2. **The score** — `slop_score` = flagged words ÷ total, weighted by category (`ai-vocab` counts a little more, `transition` a little less). `0.20` ≈ \"a fifth of this text is listed filler.\" Usually `0.0–1.0`; can edge slightly above on text that's almost nothing but slop. Instead of being a quality quantfier, its just a **tripwire** that drives the CI exit code.\n\nPick a threshold based on how filler-dense the text is *allowed* to be:\n\n| `--threshold` | Meaning | Good for |\n|---------------|---------|----------|\n| `0.05` | ~5% filler — strict | marketing copy, landing pages, customer-facing text |\n| `0.08` *(default)* | ~8% filler — a tripwire for triage | general prose, blog drafts |\n| `0.12–0.20` | only flag heavy padding | technical docs that legitimately use `robust`, `scalable`, `in order to` |\n\nThe default `0.08` is **provisional** — hand-chosen, not yet calibrated on a labeled corpus (hence the `[threshold provisional]` tag). For a hard CI gate, set your own threshold and suppress your domain's vocabulary first (below).\n\n---\n\n## Use in CI\n\n> Technical writing — API docs, ADRs, RFCs — legitimately uses words like `robust`, `scalable`, `in order to`. Suppress your project's domain vocabulary first:\n>\n> ```bash\n> defluff lexicon rm \"scalable\" --scope project\n> defluff lexicon rm \"in order to\" --scope project\n> git add .defluff/ignore.json && git commit -m \"defluff: allow domain vocabulary\"\n> ```\n>\n> Then gate only on the categories you trust, not all five:\n\n```yaml\n- name: Check AI-generated content for slop\n  # --category ai-vocab,hedge gates only on the highest-precision categories\n  run: cat generated_output.md | defluff lint --category ai-vocab,hedge --threshold 0.1\n```\n\nExit code `1` fails the step, `0` passes.\n\n---\n\n## Use with pre-commit\n\n```yaml\n# .pre-commit-config.yaml\nrepos:\n  - repo: local\n    hooks:\n      - id: defluff\n        name: defluff slop check\n        entry: defluff lint\n        language: system\n        types: [markdown]\n```\n\n---\n\n## Use in reward loops (experimental)\n\nA deterministic, non-differentiable scalar for filler density can be a small *component* of a reward mix — but it's gameable alone: a model optimized purely against a fixed phrase list learns to paraphrase the filler rather than remove it. Pair it with a real quality signal (human or LLM judge); we don't yet have a published training run showing it helps.\n\n```python\nlex = defluff.load_lexicon()  # pin once\nreward = lambda text: -defluff.detect(text, lexicon=lex).slop_density  # unclamped, better gradient\n\ndelta = defluff.compare(draft_v1, draft_v2, lexicon=lex)\n# {\"score_a\": 0.31, \"score_b\": 0.18, \"delta\": -0.13,\n#  \"improved\": [...], \"regressed\": [...]}  # set diff of flagged phrases, not a semantic diff\n```\n\n---\n\n## CLI reference\n\n```bash\ndefluff lint [FILE] [--json] [--threshold FLOAT] [--category CATS] [--lexicon PATH] [--pack NAMES] [--no-project-overlay]\ndefluff score [FILE] [--pack NAMES] [--no-project-overlay]\ndefluff packs          # list bundled domain packs\n\ndefluff lexicon list   [--category CATEGORY] [--scope SCOPE] [--json]\ndefluff lexicon add    PATTERN --category CATEGORY [--scope SCOPE] [--weight FLOAT]\ndefluff lexicon rm     PATTERN [--scope SCOPE]\n```\n\n- `--category` — comma-separated; only spans in those categories count toward the exit-code decision (still reports all hits). Valid: `ai-vocab`, `cliche`, `hedge`, `corporate`, `transition`, `custom`, `rhetoric`.\n- `--lexicon PATH` — layers your phrases on top of the defaults. `.txt`/`.md` is one phrase per line (lands in `custom`); `.json` carries explicit categories and weights.\n- `--pack NAMES` — comma-separated [domain packs](#ready-made-domain-packs). `rhetoric` is reserved for the pattern pack, enabling the opt-in `X, not Y` antithesis fragment.\n\nExit codes for `defluff lint`: `0` = clean · `1` = slop · `2` = bad input.\n\n---\n\n## Accuracy\n\ndefluff is a deterministic matcher, not a trained classifier, so the metric that matters is **precision** — when it flags something, is it actually removable filler? On a 50-example hand-labeled set ([`eval/validation.jsonl`](eval/validation.jsonl)) spanning clear slop, clean prose, and *jargon-as-content traps* (e.g. \"the **robust** standard errors\", \"**pivotal** trials\"), at the default threshold:\n\n| Metric | Score | Reading |\n|--------|------:|---------|\n| **Precision** | **1.00** | 0 false positives — clean prose and legitimate jargon were not flagged |\n| **Recall** | **0.65** | bounded by lexicon coverage |\n\nReproduce: `python eval/score.py eval/validation.jsonl`\n\nMisses are novel buzzwords the lexicon hasn't seen yet (e.g. \"operationalize the ideation funnel\") — the known limit of a list-based matcher, not noise. Recall on listed filler is 1.00 and will rise as the lexicon grows, but won't reach 1.00 against open-ended novel jargon without a semantic layer.\n\nCaveats: the set is small and labeled by the author — a sanity check on precision, not an independently adjudicated benchmark.\n\n---\n\n## Limitations\n\n- **It matches a known list by design — it doesn't *understand* text.** Novel buzzwords are missed; this is the trade for being deterministic, local, and reproducible (no model, no API key, [pinnable hash](#lexicon-overlays-and-versioning)). Pair with an LLM judge if you need semantic detection of novel filler.\n- **Domain jargon is contextual.** `\"leverage\"` in a finance document is real content. Read the flagged spans; suppress false positives with `defluff lexicon rm` (adds to an ignore list, doesn't delete from the bundled lexicon).\n- **Span offsets are into the cleaned text.** defluff strips code fences, inline code, URLs, and markdown markup before matching, so offsets won't line up with your original document — match on `span.text`, not raw offsets, against marked-up source.\n- **`custom` is read-only-via-file.** Phrases from a `--lexicon` file land in `custom`, but `defluff lexicon add --category custom` is rejected — `add` only takes the five curated categories.\n- **English only** in v0.\n- **Short texts** (< 20 words) get `low_confidence: true` — the denominator is floored at 20 so one phrase can't read as 100% slop on a two-sentence input.\n\n---\n\n## Contributing\n\nThe easiest contribution is adding a missed filler phrase:\n\n1. Add it to `src/defluff/data/lexicon-v1.json` with the right category\n2. `pytest` — smoke tests catch boundary errors\n3. PR with one or two examples of the phrase in the wild\n\nSee [CONTRIBUTING.md](CONTRIBUTING.md) for code setup and guidelines.\n\n---\n\n## License\n\nMIT\n",
  "bytes": 17506,
  "sha": "a56fc3d7f5d71e6b4cb98395f0065f237dcddeb584b938cd7f6a018ba3495f4d",
  "repo_slug": "ahmedak/defluff",
  "fonte": "repo",
  "truncated": false,
  "api": "https://agentalog.com/api/listings/mcp_io_github_ahmedak_defluff_0649061c/readme"
}