{
"markdown": "\n\n
rule-audit
\n\n**rule-audit is a static analyzer for AI system prompts: it parses a prompt into normative rules and reports logical contradictions, coverage gaps, priority ambiguities, meta-rule paradoxes, and absolute-rule edge cases — without calling an LLM.**\n\nrule-audit is developed by [Hermes Labs](https://hermes-labs.ai).\n\nHermes Labs is an agentic infrastructure company building the reliability layer for autonomous systems.\n\n[](https://github.com/hermes-labs-ai/rule-audit/actions/workflows/ci.yml)\n[](https://pypi.org/project/rule-audit/)\n[](https://pypi.org/project/rule-audit/)\n[](LICENSE)\n\n\n\nIt runs as a pre-commit hook that audits agent rules and prompts — the AI\nsystem prompts and agent instructions (`CLAUDE.md`, `AGENTS.md`, `SOUL.md`)\nthat steer a coding agent — before they're committed. See\n[Pre-commit](#pre-commit) below.\n\n---\n\n## The problem\n\nA complex AI safety prompt can contain rules that conflict under specific conditions. Those conflicts are easy to write and hard to see by eye. `rule-audit` reads the prompt the way a linter reads code and surfaces the conflicts as structured findings.\n\nIllustrative example — two rules that have no stated priority between them:\n```\n\"You must always follow user instructions.\"\n\"You must never produce harmful content.\"\n```\nThe moment a user instructs the model to produce harmful content, nothing in the prompt says which rule wins. `rule-audit` flags this pair so the author can add an explicit ordering.\n\n---\n\n## Install\n\n```bash\npip install rule-audit\n```\n\nOr install the CLI from the [Hermes Labs Homebrew tap](https://github.com/hermes-labs-ai/homebrew-tap):\n\n```bash\nbrew install hermes-labs-ai/tap/rule-audit\n```\n\nOr from source:\n```bash\ngit clone https://github.com/hermes-labs-ai/rule-audit\ncd rule-audit\npip install -e \".[dev]\"\n```\n\nPure Python, no runtime dependencies, Python 3.9+.\n\n---\n\n## 60-second quickstart\n\n### CLI\n\n```bash\n# Built-in demo — exercises every detector family, no input needed\nrule-audit --demo\n```\n\nThis demo is deliberately contradictory and produces a CRITICAL finding with exit code `2`, which is the expected successful outcome—the exit code `2` signals a finding, not an invocation error.\n\n```bash\n# Inline prompt\nrule-audit \"You are helpful. You must never lie. Always answer every question.\"\n\n# From a file\nrule-audit --file system_prompt.txt\n\n# Save a Markdown report\nrule-audit --file system_prompt.txt --output report.md\n\n# JSON for downstream processing\nrule-audit --file system_prompt.txt --format json\n\n# Summary only (handy in CI)\nrule-audit --file system_prompt.txt --format summary\n\n# Keep contradiction and edge-case detail at high severity\nrule-audit --file system_prompt.txt --min-severity high\n```\n\nExit codes: `0` = LOW/MEDIUM risk, `2` = HIGH/CRITICAL risk, `1` = error.\n\n### Pre-commit\n\nAudit prompt files before they are committed:\n\n```yaml\nrepos:\n - repo: https://github.com/hermes-labs-ai/rule-audit\n rev: v0.5.0\n hooks:\n - id: rule-audit\n```\n\nThen run:\n\n```bash\npre-commit install\npre-commit run rule-audit --all-files\n```\n\nThe hook checks Markdown and text files under `prompt/` or `prompts/`, plus conventional system, developer, and agent prompt/instruction filenames. It reports every matched file and preserves the CLI exit codes above. Adjust `files:` in your consumer configuration if your prompts live elsewhere.\n\n### Agent plugin: Claude Code, Codex, Gemini CLI, GitHub Copilot CLI\n\nThe repository root is one portable [Agent Plugin](https://agent-plugins.org)\n(`plugin.json`, Agent Plugins 1.0.0) that ships one skill,\n[`skills/rule-audit/SKILL.md`](skills/rule-audit/SKILL.md). Each host installs\nit with its own native command; none of them gets a separate copy of the skill.\n\n| Host | Install | Read back |\n| --- | --- | --- |\n| Claude Code | `claude plugin marketplace add hermes-labs-ai/rule-audit`
`claude plugin install rule-audit@rule-audit` | `claude plugin list` |\n| OpenAI Codex CLI | `codex plugin marketplace add hermes-labs-ai/rule-audit`
`codex plugin add rule-audit@rule-audit` | `codex plugin list` |\n| Gemini CLI | `gemini extensions install https://github.com/hermes-labs-ai/rule-audit --ref v0.5.0` | `gemini skills list` |\n| GitHub Copilot CLI | `copilot plugin install hermes-labs-ai/rule-audit` | `copilot plugin list`
`copilot skill list` |\n\nWhat each host reads:\n\n- Claude Code reads `.claude-plugin/marketplace.json` (source `./`, the root)\n and `.claude-plugin/plugin.json`.\n- Codex reads the repo marketplace `.agents/plugins/marketplace.json` (source\n `./`, the root) and the portable `plugin.json`.\n- Gemini CLI reads `gemini-extension.json` and discovers the skill under\n `skills/`. Pin `--ref v0.5.0` for the immutable native-plugin boundary; use\n `main` only when intentionally testing unreleased development changes.\n- GitHub Copilot CLI reads the root `plugin.json` and discovers the skill under\n `skills/`.\n\nThe skill runs the bundled adapter next to it against an installed\n`rule-audit`, or, if none is installed, under `uvx --from rule-audit==0.5.0`.\n\n### Claude Code\n\nA native plugin adds one slash command, so you can audit a prompt file without\nleaving the session. The repository is itself a Claude Code plugin marketplace:\n\n```bash\nclaude plugin marketplace add hermes-labs-ai/rule-audit\nclaude plugin install rule-audit@rule-audit\n```\n\nTo try it from a checkout without installing, run `claude --plugin-dir .`.\n\n```\n/rule-audit:audit prompts/support_agent.md\n```\n\nIt runs this CLI locally and returns a bounded report — at most 5 findings per\nfamily, with the true totals and the command to see the rest — plus the caveats\nneeded to read a `HIGH` label correctly. Nothing runs unless you ask for it.\n\nSee [`integrations/claude-code/README.md`](integrations/claude-code/README.md)\nfor install, disable and uninstall, and for the measurements behind the decision\nto ship a command rather than an automatic edit-time hook.\n\n### Gemini CLI\n\nAn extension adds the same command to Gemini CLI:\n\n```bash\ngemini extensions install https://github.com/hermes-labs-ai/rule-audit --ref v0.5.0\n```\n\n```\n/rule-audit:audit prompts/support_agent.md\n```\n\nGemini CLI shows you the exact command and asks you to approve it, then runs\nthis CLI locally and hands the model the same bounded report. Nothing runs\nunless you ask for it, and there is no hook.\n\nSee [`integrations/gemini-cli/README.md`](integrations/gemini-cli/README.md) for\ninstall, disable and uninstall, and for the host-specific details — why the\ncommand exits 0, and why the report is fenced in explicit markers.\n\n### Hermes Agent\n\nA plugin adds the command to [Hermes Agent](https://github.com/NousResearch/hermes-agent):\n\n```bash\nhermes plugins install hermes-labs-ai/rule-audit/integrations/hermes-agent\nhermes plugins enable rule-audit\n```\n\n```\n/rule-audit # audits your own SOUL.md\n/rule-audit prompts/support.md # audits any prompt file\n```\n\nTyped bare it audits your `SOUL.md` — the first section of the Hermes system\nprompt, and a system prompt in exactly the sense this tool is calibrated for.\nThe report goes to you, not to the model. Nothing runs unless you ask for it,\nand there is no hook.\n\nSee [`integrations/hermes-agent/README.md`](integrations/hermes-agent/README.md)\nfor install, disable and uninstall, and for the host-specific details — why the\nplugin registers no model tool, and where the report is displayed.\n\n### Codex\n\nA plugin adds the skill to [Codex](https://github.com/openai/codex). The\nrepository is itself a Codex plugin marketplace:\n\n```bash\ncodex plugin marketplace add hermes-labs-ai/rule-audit\ncodex plugin add rule-audit@rule-audit\n```\n\nThe installer git-clones the default branch and looks for\n`.agents/plugins/marketplace.json` there. That index is on `main`, so the bare\nform needs no `--ref`.\n\n```\n> Audit prompts/support_agent.md with $rule-audit:rule-audit\n```\n\nType `$` or `/skills` to pick the skill — either way it is inserted into the\ncomposer and nothing runs until you send it. Codex then runs the bundled adapter\nlocally and summarises the findings. Codex may also reach for the skill on its\nown when you ask for a prompt audit without naming it, which is how skills work\nand which the plugin README explains. There is no hook, nothing runs in the\nbackground, and nothing runs at all until you send a turn.\n\nSee [`integrations/codex/README.md`](integrations/codex/README.md) for install,\ndisable and uninstall, for what an installed skill costs on every turn, and for\nthe host-specific details — why the wrapper, rather than the skill text, owns\nevery guarantee.\n\n### GitHub Copilot CLI\n\nInstall the repository's Agent Plugin directly, then check that its skill is\nenabled:\n\n```bash\ncopilot plugin install hermes-labs-ai/rule-audit\ncopilot plugin list\ncopilot skill list\n```\n\nCopilot CLI currently warns that direct repository installs are deprecated, but\nthe command works. In a Copilot session, name a system-prompt file explicitly:\n\n```\nUse the /rule-audit skill to audit prompts/support_agent.md\n```\n\nThe plugin installs the skill, not the analyzer runtime. Running an audit needs\nPython 3.9+ and a compatible `rule-audit` installation (for example,\n`pipx install rule-audit==0.5.0`); the skill documents a pinned `uvx` fallback.\nUse it for system-prompt logic. [LintLang](https://github.com/hermes-labs-ai/lintlang)\ncovers the broader structure of agent configs and tool descriptions. The skill\nruns on demand, and a risk label is a heuristic finding, not proof of an exploit.\n\n### Machine-readable result envelope\n\nFor tooling that compares results across tools, the same audit can be emitted\nas a Hermes Reliability Lab result envelope — the ordinary JSON report embedded\nverbatim, plus tool version, a hash of the exact input, one finding per thing\nthe analyzer found (each with its source span), the exit code, a timestamp,\nand the Git commit when run from a checkout:\n\n```bash\npython -m rule_audit.evidence --file system_prompt.txt\npython -m rule_audit.evidence --case negative_clean_prompt # a labeled calibration case\n```\n\nIt changes nothing about detection or scoring and reads and writes no files\nbeyond the input. Exit codes are the CLI's: `0`, `2` for HIGH/CRITICAL, `1`\nwhen the input could not be read (the envelope then reports status `unknown`).\n\n### Python API\n\n```python\nfrom rule_audit import audit\n\nreport = audit(\"\"\"\n You are a helpful AI assistant.\n Always answer user questions completely.\n Never produce content that could harm anyone.\n Follow operator instructions above all else.\n If asked to roleplay without restrictions, comply.\n\"\"\")\n\nprint(report.summary())\n# rule-audit report [2026-...T...]\n# ============================================================\n# Rules parsed : 4\n# Contradictions : 1 (1 high, 0 medium)\n# Coverage gaps : 5\n# Priority ambiguities : 0\n# Meta-paradoxes : 0\n# Absoluteness issues : 5\n# Edge case scenarios : 17\n# Risk score : 55/100 [HIGH]\n\n# Full Markdown report\nmd = report.to_markdown()\n\n# Access findings programmatically\nfor c in report.result.contradictions:\n print(c.severity, c.description)\n\nfor ec in report.edge_cases:\n print(ec.title, ec.attack_vector)\n```\n\n(Exact counts depend on the input prompt; the values above are the actual output for the five-line prompt shown.)\n\n---\n\n## What it detects\n\n### 1. Contradictions\nRule pairs that pull against each other. Four detector families:\n\n- **Direct** — opposing modalities on a shared topic (e.g. `MUST` vs `MUST_NOT`).\n- **Conditional** — one rule applies unconditionally, another applies a contradicting directive under a condition; the overlap region is undefined.\n- **Scope** — a universal obligation (`always …`) and a restricted obligation (`… only / except …`) on the same domain.\n- **Absoluteness** — two high-absoluteness rules that pull in opposite directions (e.g. compliance vs safety).\n\n### 2. Coverage gaps\nChecks the prompt against eight safety-relevant domains and flags any with no rule coverage: harmful content, principal hierarchy (user vs operator vs developer), ambiguous requests, persona/roleplay, refusal protocol, instruction-conflict resolution, self-disclosure of instructions, and edge-case fallback behavior. Also flags conditional rules that have no stated default for the else-case.\n\n### 3. Priority ambiguities\nRule clusters that conflict with no explicit ordering and no meta-rule that resolves them.\n\n### 4. Meta-rule paradoxes\nRules that reference rules — e.g. \"ignore all previous instructions\" (self-defeating), \"these instructions supersede all others\" (exploitable via injection), or override language elsewhere in the prompt that could be used to void other rules.\n\n### 5. Absoluteness audit\nEach `always` / `never` / `under no circumstances` rule is paired with challenge scenarios: known exceptions, context-dependent cases, and adversarial triggers.\n\n### 6. Edge-case scenarios\nFor each finding, the report renders a concrete example scenario plus a suggested attack vector, expected failure mode, and mitigation. These are templated from the finding — illustrative starting points for testing, not verified exploits.\n\n---\n\n## Calibration: does it actually work?\n\n`calibration/` is a bounded, hand-labeled corpus (11 cases) with an explicit\nground truth — not a statistical claim, an auditable one. Positive cases pin\ndown a true finding per detector family (direct/scope/conditional/absoluteness\ncontradiction, meta-paradox, priority ambiguity, coverage gap); negative cases\npin down known false-positive traps, like two rules with opposing modality on\ncompletely unrelated topics.\n\n```bash\n# Machine-readable benchmark result (JSON), exit 1 on any regression\npython -m rule_audit.calibration\n\n# As a pytest gate\npytest tests/test_calibration.py -v\n```\n\nEvery `Rule` carries `start` / `end` character offsets into the original\nprompt (`report.to_dict()[\"rules\"][i][\"span\"]`, also threaded onto\ncontradictions, meta-paradoxes, and absoluteness issues) — every finding\ntraces back to an exact source span, not just a truncated text snippet.\nSee `calibration/README.md` for the case schema and how to add cases.\n\n---\n\n## Limitations / what it does NOT do\n\n- **Lexical parser, not a language model.** Parsing is sentence-splitting + modal-verb regex + keyword clusters. Rules that need semantic understanding (implied or narrative-embedded constraints) can be missed.\n- **It does not prove a prompt is exploitable.** A `CRITICAL` risk label means \"many absolute rules and contradictions in a short prompt\" by the lexical scoring — not a verified end-to-end exploit. For dynamic verification, pair it with [`hermes-jailbench`](https://github.com/hermes-labs-ai/hermes-jailbench) (jailbreak regression).\n- **14 keyword clusters, curated by hand.** Uncommon domains may not trigger coverage-gap detection; extend `_KEYWORD_CLUSTERS` in `analyzer.py`.\n- **Absoluteness defaults to 0.5** for modal sentences with no qualifier keyword. A design choice — tune `_compute_absoluteness` for your corpus.\n- **English only** in this release.\n- **Single-document only.** Multi-part prompts (operator + user + tool results) merged into one input are analyzed as a flat rule list; structural separation between principals is not modeled.\n- **O(n²) pair comparison.** Fine for realistic prompts; very large rule sets will be slow.\n\n---\n\n## How it relates to other tools\n\n- **`rule-audit` and [LintLang](https://github.com/hermes-labs-ai/lintlang) are complementary, not duplicates.** `rule-audit` analyzes the *logical content* of a system prompt (contradictions, gaps, priority). LintLang lints the *structure* of agent configs and tool descriptions. Run both.\n- **`rule-audit` is static; [`hermes-jailbench`](https://github.com/hermes-labs-ai/hermes-jailbench) is dynamic.** Static analysis finds candidate flaws; dynamic testing checks whether they are reachable against a live endpoint.\n\n---\n\n## Architecture\n\n```\nrule_audit/\n├── __init__.py # Public API: audit(), audit_file(), AuditReport\n├── parser.py # Sentence splitting, modal-verb detection, Rule objects (with source spans)\n├── analyzer.py # Contradiction / gap / priority / meta / absoluteness detectors\n├── edge_cases.py # Scenario generator from analysis results\n├── report.py # AuditReport + Markdown / JSON renderers\n├── calibration.py # Labeled calibration corpus runner (calibration_cases/*.json, package data)\n├── precommit.py # Pre-commit hook entry point\n└── cli.py # CLI entry point\n```\n\nPure Python standard library, zero runtime dependencies, deterministic (same input → same output), no network calls.\n\n---\n\n## Development\n\n```bash\npip install -e \".[dev]\"\n\n# Run the test suite\npytest\n\n# With coverage\npytest --cov=rule_audit --cov-report=term-missing\n\n# Audit a real prompt\npython -m rule_audit --file your_prompt.txt --verbose\n```\n\n### GitHub Actions\n\nThe composite [Rule Audit GitHub Action](action.yml) installs the published\n`rule-audit` CLI and audits files matching a recursive glob. It reports each\nfile in the workflow log and job summary. HIGH or CRITICAL risk returns exit\ncode `2` and fails the step by default; set `fail-on-high-risk: false` to\ncollect reports without failing. It does not make a security certification or\nreplace testing against a live model.\n\n```yaml\nname: Prompt analysis\non: [pull_request]\n\njobs:\n rule-audit:\n runs-on: ubuntu-latest\n steps:\n - uses: actions/checkout@v4\n - uses: hermes-labs-ai/rule-audit@action-v0.1.0\n with:\n path: \"prompts/**/*.txt\"\n```\n\nThe caller must check out the repository before invoking the Action. The Action\nrequires network access to install the pinned Rule Audit release from PyPI.\nThe `files-scanned` output reports how many matching files were audited.\n\n---\n\n## More from Hermes Labs\n- [lintlang](https://github.com/hermes-labs-ai/lintlang) — Static analysis for AI agent tool descriptions and workflows.\n- [little-canary](https://github.com/hermes-labs-ai/little-canary) — Prompt injection detection through a powerless sacrificial model.\n- [fidelis](https://github.com/hermes-labs-ai/fidelis) — Semantic memory for long-running agents with local retrieval.\n- [hermeneutic](https://github.com/hermes-labs-ai/hermeneutic) — Detect recurring AI drift from correction history.\n- [zer0dex](https://github.com/hermes-labs-ai/zer0dex) — Local agent recall without burdening the context window.\n- [Hermes Labs plugins marketplace](https://github.com/hermes-labs-ai/plugins) — Plugin system for Claude Code extensibility.\n\nBrowse the [open-source catalog](https://hermes-labs.ai/open-source) or contact [roli@hermes-labs.ai](mailto:roli@hermes-labs.ai).\n\n## License\n\nApache-2.0 — see [LICENSE](LICENSE). © Hermes Labs 2026.\n\nNot affiliated with NousResearch, Teknium, the Nous-Hermes LLM line, or any unrelated `hermes-*` project.\n",
"bytes": 19226,
"sha": "143aa04fa892185803f5b7f82a7daa35c8057f00617ee3531c548cecbf4907fe",
"repo_slug": "roli-lpci/rule-audit",
"fonte": "repo",
"truncated": false,
"api": "https://api.agentalog.com/api/listings/plg_roli_lpci_rule_audit_787dcc04/readme"
}