publishing-assurance-toolkit
Creative provenance and generative publishing evaluation tools
Open source Open in the app JSON README (API)
About
Creative provenance and generative publishing evaluation tools
Details
- Kind
- Plugins
- Topic
- No topic detected
- Publisher
- fredzannarbor
- Origin
- gemini
- Category
- ferramentas
- Version
- 0.1.0
- Last push
- 2026-08-08T01:42:08Z
- Repository state
- ativo
- Language
- Python
- License
- NOASSERTION
- Added
- 2026-08-30 14:13:39
- Updated
- 2026-08-30 14:13:39
- Origin id
fredzannarbor/publishing-assurance-toolkit
README
# Publishing Assurance Toolkit
> Open infrastructure for the book publishing community to observe, document,
> and evaluate generative-AI workflows—without surrendering manuscripts,
> judgment, or craft.
Publishing Assurance Toolkit (`pubassure`) gives authors, publishers,
researchers, and software builders a common set of building blocks for two jobs:
1. **Provenance assurance** — record inspectable evidence about human and machine
contributions to an expressive work.
2. **Generative-system evaluation** — test publishing systems against portable,
reproducible challenges and report what happened under declared conditions.
The aim is not to pronounce whether a book is “AI-generated.” That framing is
too blunt for real editorial work. The aim is to help the publishing community
build better instruments: tools that can preserve human choices, monitor system
behavior, compare declared performance, and support more informed editorial
judgment.
**Project status:** alpha reference implementation. Schemas and protocols may
change before 1.0.
## Why this exists
Generative systems are entering workflows that already contain many forms of
human expression: commissioning, research, prompting, selection, revision,
fact-checking, design, approval, and accountability. A useful assurance layer
must represent that work without reducing it to a single “human percentage.”
At the same time, publishers need practical ways to ask whether a system can do
a particular publishing job—and to see the basis for the answer. A score without
the challenge, inputs, execution conditions, checks, and disclosures is hard to
interpret and harder to reproduce.
This toolkit is a practical response to the gaps documented in Fred Zimmerman's
trade-coverage study,
[*Copyright Is the Headline; Capability Is the Blind Spot: AI Technology in the
Book-Publishing Trade Press, November 2025–August 2026*](https://arxiv.org/abs/2608.00964).
The paper calls task-specific evaluation “publishing's missing instrument panel”
and argues that provenance should move upstream into publishing workflows. The
evaluation suite and provenance ledger implement those two recommendations as
open, reusable infrastructure.
`pubassure` brings those needs into one vendor-neutral toolkit:
- a local, append-only provenance ledger;
- first-class records for human instructions, selections, edits, and approvals;
- privacy-conscious public certificates;
- portable challenge and submission formats;
- deterministic checks plus separately disclosed human ratings;
- JSON and standalone HTML reports;
- adapters for publishing systems; and
- hosted MCP, portable agent-skill, Python, and CLI interfaces.
## What it looks like
### Discover public challenges through MCP
The hosted, stateless MCP service exposes evaluator-safe challenges and tools at
`https://bigfivekiller.online/mcp/pubassure`.
### Produce reports that disclose their basis
Reports identify the challenge, system, execution mode, scoring method,
disclosures, and interpretation limits. Deterministic measurements and human
observations remain distinguishable.
### Preserve human expressive contributions
The provenance protocol records model interactions alongside human
instructions, selections, edits, and approvals. Public certificates can
summarize a chain without publishing raw prompts or manuscripts.
Inspect the [full de-attributed example provenance log](https://bigfivekiller.online/data/pubassure/community-demo/events.jsonl).
It contains five canonical, hash-chained events with synthetic content and
pseudonymous actor identifiers; it contains no personal or business data.
The screenshots are generated from the bundled examples and documentation
views in [`docs/demo`](docs/demo). They use only synthetic or public sample data.
## Components
| Component | What it does | What it produces |
|---|---|---|
| **Provenance ledger** | Stores project events in an append-only SHA-256 hash chain and keeps artifacts in a content-addressed local store. | Canonical JSONL events and artifact descriptors. |
| **Human contribution records** | Represents instructions, selections, edits, and approvals as distinct, attributable events with exact contribution artifacts and stated effects. | Inspectable evidence of editorial intervention and decision-making. |
| **Public certificates** | Summarizes event counts, selected claims, and the verified chain head while keeping private details local. | Portable JSON certificate with explicit limitations. |
| **Challenge registry** | Loads and validates portable generation, classification, and repair tasks. | Public challenge catalog and evaluator-safe task inputs. |
| **Evaluation runner** | Invokes an adapter or scores an existing submission against declared checks. | Reproducible result object with per-check observations. |
| **Reporting** | Separates deterministic checks from disclosed human ratings and adds interpretation limits. | Machine-readable JSON and dependency-free HTML reports. |
| **Adapter boundary** | Connects an existing publishing system without requiring it to adopt this package internally. | A standard `submission.json` plus referenced artifacts. |
| **Release audit** | Searches a candidate release for common secrets, personal paths, private business terms, and unsafe artifacts. | A blocking privacy/release report. |
## Three ways to use it
### 1. Hosted MCP server
Use the public service for challenge discovery, public inline evaluation, and
verification of public or redacted provenance chains:
```text
https://bigfivekiller.online/mcp/pubassure
```
Example Claude Code configuration:
```bash
claude mcp add --transport http publishing-assurance https://bigfivekiller.online/mcp/pubassure
```
The repository also includes `.mcp.json` and Gemini extension metadata. The
hosted service is stateless and retains no submitted artifacts, but network
transmission is still network transmission: **keep confidential manuscripts on
the local code path**.
Available MCP tools:
- `publishing_assurance_info`
- `list_publishing_challenges`
- `get_publishing_challenge`
- `evaluate_publishing_submission`
- `verify_provenance_chain`
See [`docs/hosted-mcp.md`](docs/hosted-mcp.md).
### 2. Portable Codex, Claude, or Gemini skill
Install one canonical skill into the agent environment you already use:
```bash
uv run python scripts/install_agent_skill.py codex
uv run python scripts/install_agent_skill.py claude
uv run python scripts/install_agent_skill.py gemini --scope project
```
The skill teaches the agent when to use hosted public tools and when to keep
work local. Gemini CLI users can also install the repository as an extension,
bundling the skill and hosted MCP configuration.
### 3. Python or CLI
Use the package directly for confidential projects, automated pipelines, custom
challenges, or product integration:
```python
from pubassure import ChallengeRegistry, ProvenanceLedger
registry = ChallengeRegistry("challenges")
for challenge in registry.list_public():
print(challenge["id"], challenge["title"])
ledger = ProvenanceLedger("private-ledger", "project:example")
```
The CLI exposes validation, scoring, adapter execution, provenance operations,
and release auditing. See
[`docs/integration-surfaces.md`](docs/integration-surfaces.md) for complete
examples.
## How the pieces fit
```mermaid
flowchart LR
A[Author or editorial team] --> P[Publishing workflow]
G[Generative system] --> P
P --> L[Local provenance ledger]
P --> S[Portable submission]
C[Challenge registry] --> E[Evaluation runner]
S --> E
E --> R[JSON + HTML report]
L --> V[Public or redacted certificate]
R --> D[Editorial review and disclosure]
V --> D
```
The ledger and evaluation suite are useful independently. Together they can
show both **how a work moved through a declared process** and **how a system
performed on a declared task**.
## Quick start
The core package has no runtime dependencies. Development uses `uv` and the
locked environment in this repository.
```bash
git clone <repository-url>
cd publishing-assurance-toolkit
uv sync
uv run pubassure challenge validate challenges/generation/book-on-a-topic.json
uv run pubassure challenge list
uv run pubassure score \
--challenge challenges/classification/genre-mini.json \
--submission examples/submissions/genre-mini.json \
--output runs/genre-mini
```
Open `runs/genre-mini/report.html` to inspect the standalone report.
## Connect any publishing system
An adapter is simply a command that receives a request path and output
directory:
```text
your-adapter --request /path/to/request.json --output /path/to/output-directory
```
It writes `submission.json` and any referenced artifacts. Then the runner scores
the result:
```bash
uv run pubassure run \
--challenge challenges/generation/book-on-a-topic.json \
--adapter "your-adapter" \
--output runs/my-system
```
The adapter may wrap a desktop application, command-line tool, hosted API,
workflow engine, or supervised manual process. See
[`docs/adapter-contract.md`](docs/adapter-contract.md).
## Bundled challenge families
### Generation
Generate a short nonfiction book from a topic brief. The reference challenge
checks artifact presence, minimum substance, required structure, and obvious
placeholders. It also names separate human-rating dimensions such as factual
support, structural coherence, editorial usefulness, and voice consistency.
### Classification
Classify short descriptions using declared labels. The compact sample is useful
for testing system adapters and reproducibility end to end.
### Repair
Repair a deliberately damaged OCR passage. The challenge asserts specified
repairs and checks that known defects are absent without inviting unbounded
rewriting.
These are seeds, not a claim of complete coverage. Community contributions can
add tasks for metadata quality, accessibility, citation integrity, indexing,
layout repair, image-description quality, editorial consistency, and other
publishing concerns.
## Trust model and limits
This project favors inspectable evidence over expansive claims.
- A hash can reveal later alteration; it does not prove who controlled a key or
establish legal identity.
- A recorded event can document a declared contribution; it does not decide
authorship, copyright ownership, or originality.
- A challenge report measures only the checks named by that challenge under the
stated conditions.
- A human rating is an attributed observation, not hidden ground truth.
- A self-maintained certificate is not third-party certification.
- No module is an “AI detector.”
For the complete boundary, read [`docs/limitations.md`](docs/limitations.md).
## Privacy and release safety
Privacy is the default architectural posture:
- unpublished artifacts stay local unless a user deliberately transmits them;
- actors may use pseudonymous identifiers;
- public certificates need not contain raw prompts or manuscripts;
- the hosted MCP accepts only public or intentionally redacted material; and
- the release audit checks for common credentials, personal paths, private terms,
and unsafe files before publication.
Run the audit before publishing a fork or release:
```bash
uv run pubassure audit-release .
```
See [`docs/privacy-and-release.md`](docs/privacy-and-release.md) and
[`SECURITY.md`](SECURITY.md).
## For the publishing community
This repository is intended as shared infrastructure, not a finished policy
answer. Useful contributions include:
- new challenges grounded in real publishing tasks;
- scorer implementations with clear measurement limits;
- adapters for publishing and editorial systems;
- better reporting and accessible output formats;
- threat-model and privacy review;
- interoperability work with provenance and content-credential standards; and
- documentation from authors, editors, designers, production specialists,
researchers, and publishers.
Start with [`CONTRIBUTING.md`](CONTRIBUTING.md), the
[`architecture`](docs/architecture.md), and the [`roadmap`](docs/roadmap.md).
## Development
```bash
uv sync
uv run python -m unittest discover -s tests -v
uv run pubassure audit-release .
```
The project deliberately keeps the core small and dependency-free. New
dependencies should earn their place and remain pinned.
## Share the project
An eight-slide, square-format social carousel is included for introducing the
project to publishing audiences.
- [Editable PowerPoint deck](docs/social/publishing-assurance-carousel.pptx)
- [1080 × 1080 posting images](docs/social/carousel)
- [Suggested post copy and accessibility text](docs/social/README.md)
## Documentation map
- [Architecture](docs/architecture.md)
- [Integration surfaces](docs/integration-surfaces.md)
- [Hosted MCP](docs/hosted-mcp.md)
- [Adapter contract](docs/adapter-contract.md)
- [Metrics](docs/metrics.md)
- [Privacy and release safety](docs/privacy-and-release.md)
- [Limitations](docs/limitations.md)
- [Roadmap](docs/roadmap.md)
## License
Apache License 2.0. See [`LICENSE`](LICENSE).