{
  "markdown": "# dark-pages\n\n**Make your research agent confess what it couldn't read.**\n\nEvery deep-research report has a shadow bibliography: the sources the agent\ntried to access and couldn't. Bot walls, Cloudflare challenges, paywalls,\nrobots.txt exclusions, rate limits, and plain 404s silently remove sources\nfrom consideration, and the final report reads as if they never existed.\nThese are the dark pages.\n\ndark-pages captures every blocked or failed web access during a research\nsession and renders it as a mandatory section of the deliverable:\n\n```markdown\n## Appendix: Blocked and Inaccessible Sources\n\n| Search Topic | URL | Result | Why Blocked | Recovery Outcome |\n|---|---|---|---|---|\n| community oncology closures | https://example.org/report.pdf | http-403 | Bot traffic forbidden | Wayback copy from 2026-05 used |\n```\n\nBlocked accesses also trigger a recovery ladder (Wayback Machine,\narchive.today, surviving caches, mirrors/syndicated copies) before a gap is\nconceded, and the appendix records what each rescue found.\n\n## How it works\n\nTwo surfaces, two guarantee levels:\n\n**Claude Code (guarantee).** A global PostToolUse hook\n(`claude-code/hook/dark-pages-monitor.py`, Python 3 stdlib only) watches every\nweb access path: WebFetch, WebSearch, `curl`/`wget` in Bash, MCP fetchers, and\nbrowser-automation tools. It classifies each failure:\n\n| Class | Meaning |\n|---|---|\n| `http-<code>` | 403, 404, 429, 451, 5xx returned outright |\n| `soft-block-<type>` | 200 response that is actually an anti-bot challenge, captcha, login wall, or paywall |\n| `net-<type>` | DNS failure, timeout, connection refused, TLS error |\n| `tool-refusal-<type>` | robots.txt exclusion, disallowed domain, redirect refusal |\n\nEach failure is appended to `<project>/.claude/dark-pages/<date>-<session>.jsonl`\n(the directory self-gitignores) with a topic guess taken from the most recent\nsearch query, and the hook injects a one-line nudge telling the model to run\nthe recovery ladder now. The model cannot forget what it never had to\nremember. The companion skill (`claude-code/skill/`) carries the recovery\nladder and the appendix contract.\n\n**claude.ai (promise).** No hooks exist there, so `claude-web/` ships an\ninstruction-only skill plus a paste-in Project instructions snippet: the model\nkeeps an in-context ledger of its own failed fetches and renders the same\nappendix. Best-effort by nature, and the skill says so; the built-in Research\nfeature's internal sub-fetches are not observable from a skill.\n\n## Install\n\nClaude Code (installs hook + skill user-globally, registers the hook in\n`~/.claude/settings.json` with a timestamped backup; idempotent):\n\n```bash\n./install.sh\npython3 -m pytest tests/ -q   # 13 tests\n```\n\nNew sessions pick up the hook; already-running sessions do not (hooks snapshot\nat session start).\n\nclaude.ai: upload `dist/dark-pages-web.zip` under Settings > Capabilities, and\npaste `claude-web/project-instructions.md` into any research Project.\n\nUninstall: remove `~/.claude/hooks/dark-pages-monitor.py`,\n`~/.claude/skills/dark-pages/`, and the PostToolUse entry referencing\n`dark-pages-monitor.py` in `~/.claude/settings.json`.\n\n## Repo map\n\n```\nclaude-code/hook/dark-pages-monitor.py   deterministic capture (the guarantee)\nclaude-code/skill/                       recovery ladder + appendix contract\nclaude-web/                              claude.ai skill + Project snippet\ndist/dark-pages-web.zip                  ready-to-upload web package\ninstall.sh                               global Claude Code install\ntests/                                   canned-payload tests for the hook\nokf/                                     knowledge bundle (Open Knowledge Format)\ndocs/                                    design spec + EU AI Act determination\n```\n\n## Design notes\n\n- The hook must never break a session: any internal error exits 0 silently.\n- Soft-block detection is heuristic (signature list); expect to extend it as\n  you meet new walls. PRs welcome.\n- Search-result text is exempt from soft-block sniffing, since result snippets\n  legitimately mention words like \"captcha\".\n- Failures against archive hosts are logged as recovery attempts and fold into\n  their primary row in the appendix; they never nudge (loop prevention).\n- An empty log still produces the appendix (\"No blocked or failed accesses\n  were recorded\"), so its absence is never ambiguous.\n\n## License\n\nMIT\n",
  "bytes": 4398,
  "sha": "4c6b04aa74797e6b6071671a90d2dc8003e9a64beb95119087711b0ccae239ab",
  "repo_slug": "edrod-oncology/dark-pages-skill",
  "fonte": "repo",
  "truncated": false,
  "api": "https://agentalog.com/api/listings/okf_edrod_oncology_dark_pages_skill_okf_inde_76ca2dcc/readme"
}