dark-pages knowledge bundle
Bundle OKF 0.2 · 2 conceitos · edrod-oncology/dark-pages-skill
Open source Repository Open in the app JSON README (API)
About
# dark-pages knowledge bundle
Capture of blocked/failed web accesses during deep research, rendered as a
mandatory appendix in research deliverables. Claude Code (hook + skill) and
claude.ai (instruction-only skill package).
## Contents
* [Datasets](datasets/index.md) - the JSONL access log written by the hook
* [Runbooks](runbooks/index.md) - install, test, and packaging procedures
Change history: [log](log.md)
Details
- Kind
- OKF bundles
- Topic
- Government & public data
- Publisher
- edrod-oncology
- Origin
- okf_github
- Category
- dados
- Version
- 0.2
- Last push
- 2026-09-04T12:53:05Z
- Repository state
- ativo
- Language
- Python
- License
- MIT
- Added
- 2026-09-08 09:01:12
- Updated
- 2026-09-08 09:01:12
- Origin id
edrod-oncology/dark-pages-skill:okf/index.md
README
# dark-pages
**Make your research agent confess what it couldn't read.**
Every deep-research report has a shadow bibliography: the sources the agent
tried to access and couldn't. Bot walls, Cloudflare challenges, paywalls,
robots.txt exclusions, rate limits, and plain 404s silently remove sources
from consideration, and the final report reads as if they never existed.
These are the dark pages.
dark-pages captures every blocked or failed web access during a research
session and renders it as a mandatory section of the deliverable:
```markdown
## Appendix: Blocked and Inaccessible Sources
| Search Topic | URL | Result | Why Blocked | Recovery Outcome |
|---|---|---|---|---|
| community oncology closures | https://example.org/report.pdf | http-403 | Bot traffic forbidden | Wayback copy from 2026-05 used |
```
Blocked accesses also trigger a recovery ladder (Wayback Machine,
archive.today, surviving caches, mirrors/syndicated copies) before a gap is
conceded, and the appendix records what each rescue found.
## How it works
Two surfaces, two guarantee levels:
**Claude Code (guarantee).** A global PostToolUse hook
(`claude-code/hook/dark-pages-monitor.py`, Python 3 stdlib only) watches every
web access path: WebFetch, WebSearch, `curl`/`wget` in Bash, MCP fetchers, and
browser-automation tools. It classifies each failure:
| Class | Meaning |
|---|---|
| `http-<code>` | 403, 404, 429, 451, 5xx returned outright |
| `soft-block-<type>` | 200 response that is actually an anti-bot challenge, captcha, login wall, or paywall |
| `net-<type>` | DNS failure, timeout, connection refused, TLS error |
| `tool-refusal-<type>` | robots.txt exclusion, disallowed domain, redirect refusal |
Each failure is appended to `<project>/.claude/dark-pages/<date>-<session>.jsonl`
(the directory self-gitignores) with a topic guess taken from the most recent
search query, and the hook injects a one-line nudge telling the model to run
the recovery ladder now. The model cannot forget what it never had to
remember. The companion skill (`claude-code/skill/`) carries the recovery
ladder and the appendix contract.
**claude.ai (promise).** No hooks exist there, so `claude-web/` ships an
instruction-only skill plus a paste-in Project instructions snippet: the model
keeps an in-context ledger of its own failed fetches and renders the same
appendix. Best-effort by nature, and the skill says so; the built-in Research
feature's internal sub-fetches are not observable from a skill.
## Install
Claude Code (installs hook + skill user-globally, registers the hook in
`~/.claude/settings.json` with a timestamped backup; idempotent):
```bash
./install.sh
python3 -m pytest tests/ -q # 13 tests
```
New sessions pick up the hook; already-running sessions do not (hooks snapshot
at session start).
claude.ai: upload `dist/dark-pages-web.zip` under Settings > Capabilities, and
paste `claude-web/project-instructions.md` into any research Project.
Uninstall: remove `~/.claude/hooks/dark-pages-monitor.py`,
`~/.claude/skills/dark-pages/`, and the PostToolUse entry referencing
`dark-pages-monitor.py` in `~/.claude/settings.json`.
## Repo map
```
claude-code/hook/dark-pages-monitor.py deterministic capture (the guarantee)
claude-code/skill/ recovery ladder + appendix contract
claude-web/ claude.ai skill + Project snippet
dist/dark-pages-web.zip ready-to-upload web package
install.sh global Claude Code install
tests/ canned-payload tests for the hook
okf/ knowledge bundle (Open Knowledge Format)
docs/ design spec + EU AI Act determination
```
## Design notes
- The hook must never break a session: any internal error exits 0 silently.
- Soft-block detection is heuristic (signature list); expect to extend it as
you meet new walls. PRs welcome.
- Search-result text is exempt from soft-block sniffing, since result snippets
legitimately mention words like "captcha".
- Failures against archive hosts are logged as recovery attempts and fold into
their primary row in the appendix; they never nudge (loop prevention).
- An empty log still produces the appendix ("No blocked or failed accesses
were recorded"), so its absence is never ambiguous.
## License
MIT