{
  "markdown": "<!-- mcp-name: link.cyberelf.whetstone/tools -->\n# whetstone\n\n**Two learners grinding each other sharp: a student that must earn every\npromotion, and an examiner that must earn the right to judge it — with every\nlabel anchored to an oracle that cannot practically be wrong.**\n\nwhetstone is a self-improving research harness built around one rule: *nothing is\nbelieved until something exact has tried to kill it.* A verifier-gated student\n(QLoRA adapters on a local 4B) trains on experience milled from root oracles —\nexhaustive graph enumeration, simulated-annealing counterexample search, exact\ngame simulators, Monte-Carlo consensus, Stockfish, KataGo — and is graded by an\nevolving exam bank that quarantines anything the student trained on, learns which\nitems discriminate, retires the saturated, and feeds its retired exams back as\ntraining fuel.\n\nSelected results (full ledger with methods, nulls, and receipts in\n[FINDINGS.md](FINDINGS.md); artifacts in [results/](results/)):\n\n- Settled an open finite conjecture: degree-descending greedy coloring is NOT\n  optimal on connected triangle-free non-bipartite graphs (minimal\n  counterexample: 11 vertices, 13 edges — found by annealing, minimized, exact).\n- Quantified the verifier-horizon regress: 99/121 conjectures with precision 1.0\n  at n<=6 are false at n=7/8; acceptance horizons must sit strictly inside\n  stress horizons or self-training loops train on artifacts.\n- Measured salience vs relevance: the most salient memory is the counterfactually\n  useful one 4% of the time; a query-conditioned relevance score, 81%.\n- Ran a full unattended continual-learning loop: cold-start 0/8 -> 8/8 in one\n  gated round; the multi-domain generation cracked a cross-domain frontier its\n  own examiner had exposed (base 1/33 -> gen-2 4/33, first MIS pass).\n- Closed the same-bank scale comparison locally: stock Qwen2.5 1.5B->32B rose\n  8->20/48 while every stock model scored 0/24 on graph repairs; the task-routed\n  verifier-trained FastContext-4B scored 28/48 and earned PASS over its 22/48\n  base (6 gains, 0 regressions, exact p=.03125), repeated identically three times.\n- Trained the first trajectory-disjoint Gen-4 engine student on 1,313 fresh\n  Stockfish/KataGo examples. Three fresh loads moved 7/69 -> 8/69, but the\n  item-level gate correctly BLOCKED promotion: 3 gains, 2 regressions, exact\n  p=1.0, including one historically stable regression. No promotion is claimed.\n- The private promotion exam bank is deliberately NOT in this repo: publishing it\n  would let future models train on it — the leakage rule at internet scale.\n\nLicensed AGPL-3.0. Commercial licensing available from the author.\n\nInstall the dependency-light core (CLI and stateless toolbox) with\n`pip install .`. Optional integrations are grouped as `agents`, `engines`, and\n`local-models`; `pip install \".[all]\"` installs every integration.\n\nThe public HTTP MCP endpoint serves both the sessionless `2026-07-28`\nprotocol (`server/discover` plus per-request metadata and headers) and the\ninitialize-based `2025-06-18`, `2025-03-26`, and `2024-11-05` revisions. The\nstdlib server remains dependency-free; `mcp>=2,<3` is needed only for SDK\nclients and the stdio adapter.\n\nTagged releases contain archive-built wheels, source archives, SPDX 2.3 SBOMs,\nand `SHA256SUMS`; GitHub Actions attaches build-provenance attestations to every\nrelease artifact. For v0.8.0, verify the wheel against the repository with:\n\n```powershell\ngh release download v0.8.0 -R CarlSR9001/whetstone -p \"*.whl\" -p \"*.spdx.json\" -p \"SHA256SUMS\"\ngh attestation verify .\\branching_continual_verification-0.8.0-py3-none-any.whl -R CarlSR9001/whetstone\n```\n\n## The product layer\n\nThe research harness above is operable as a product: one CLI, adapters for any\nsystem under exam, pluggable verifiers, a CI contract, and agent-native\nsurfaces. Full guide in [docs/product.md](docs/product.md).\n\n```powershell\n$env:PYTHONPATH='src'\npython -m bcv.cli init                                   # bank + whetstone.toml\npython -m bcv.cli mint --domain code --max-items 8       # hidden-check exam items\npython -m bcv.cli grade --system v2 --command \"python my_agent.py\"\npython -m bcv.cli grade --system v2 --acp \"my-acp-agent\" # any Agent Client Protocol agent\npython -m bcv.cli gate --baseline v1 --candidate v2      # exit code IS the verdict: 0 PASS, 2 HOLD, 3 BLOCK\npython -m bcv.cli sweep                                  # retire saturated items downward\npython -m bcv.cli redteam                                # hostile self-test; nonzero exit on escape\npython -m bcv.cli serve --port 8977                      # localhost JSON service\npython -m bcv.cli mcp                                    # MCP server (also in .mcp.json)\npython -m bcv.cli inspect --exam exam.jsonl --exposure exposure.jsonl --baseline v1.json --candidate v2.json\npython -m bcv.cli hosted-report-card --command \"python my_agent.py\" --out report-card.json\npython -m bcv.cli open-bench-run --baseline-command \"python v1.py\" --candidate-command \"python v2.py\"\npython -m bcv.cli verify-receipt --receipt report-card.json\npython -m bcv.cli toolbox --port 8988                    # eight stateless tools on localhost\n```\n\nGrading an external endpoint burns every exposed item — permanently — via the\nbank's exposure accounting; the promotion report carries paired evidence, an\nexact McNemar p-value, the bank's own resolution statement, and a SHA-256\ncommitment to bank state. No surface, human or agent, ever serves exam item\ncontents.\n\nThe stateless toolbox turns eight mechanisms already in the repo into usable\nfile-in/receipt-out surfaces: Whetstone Inspector, Eval Leak Auditor, Promotion\nGate, Bank Health, SafePatch, Counterexample Hunter, Memory Relevance Debugger,\nand Agent Replay Console. Each run exposes the decision path, an inspectable\nvisual certificate, a SHA-256 receipt, and a task-specific downloadable\nartifact. Leakage analysis keeps exact identity, finite-corpus behavioral\nequivalence, and text-similarity review as separate evidence tiers; only the\nfirst two quarantine automatically. The public instance at\n[whetstone.cyberelf.link](https://whetstone.cyberelf.link/) accepts only caller\nsupplied disposable data; it has no examiner bank, accounts, cookies, or upload\npersistence. Private exams belong in the CLI or a customer-controlled instance.\n\nThe same public service now includes\n[Open Promotion Bench](https://whetstone.cyberelf.link/benchmark), a paired\nscope-integrity track. `open_bench_start` gives a baseline and candidate the\nsame six procedurally varied virtual-repository tasks; `open_bench_submit`\nchecks requested state plus edit-scope conservation and counts gains,\nregressions, and ties before issuing PASS/HOLD/BLOCK. A submitter may explicitly\npublish a self-attested sanitized receipt. The ledger stores manifests,\ntransitions, counts, and hashes only—never task contents or answer patches—so\nthe existing workbench/report-card non-retention boundary remains intact.\n\nHosted report-card and Open Bench receipts are challenge-bound and signed with\nan Ed25519 SSHSIG key. The one-command runners above fetch the HTTPS key bundle,\nverify the issuer, challenge, content commitment, signature, and expiry, then\nwrite the receipt. A valid signature authenticates Whetstone's grading result\nand deployed build; it does not authenticate a caller-supplied model or harness\nname. Runners fail closed on unsigned receipts unless `--allow-unsigned` is\nexplicitly used for a local development service. Busy report-card submissions\nare retryable without consuming the session.\n\nSupport-panel minting currently fails closed by default: the lexical checker\npassed its clean smoke corpus but failed 13/13 adversarial cases, so it cannot\nmint promotion-capable support items until an independently reviewed hard\ncalibration is supplied explicitly.\n\n## The sixty-second demo\n\nThe whole promotion gate, live, on CPU, in about ten seconds:\n\n```powershell\n$env:PYTHONPATH='src'; python -m bcv.demo_investor\n```\n\nIt mints exam items at the verifier frontier, quarantines the ones a toy\nstudent trained on (row-identity leakage check against a real buffer on disk),\ngrades two systems through the checker specs — a memorizer that echoes its\ntraining data and a candidate that proposes repairs — computes per-item\ndiscrimination, issues a gains-vs-regressions promotion decision, retires\nsaturated items into the downward-only training flow, and writes an append-only\nledger. The two systems are stored answer policies (this keeps the demo off the\nGPU); every mint, quarantine, and grade is the production code path, computed\nlive. A canonical transcript is in\n[results/demo_investor_transcript.txt](results/demo_investor_transcript.txt).\n\n## Research context\n\nFor readers (human or otherwise) mapping this to the literature: whetstone is an\nexperiment in **continual learning** and **self-improvement** for **small language\nmodels** (QLoRA / parameter-efficient fine-tuning on consumer hardware), built to\nresist the failure modes that usually kill such loops. The verifier-horizon regress\nquantified in FINDINGS §1-6 is a concrete, measured instance of **reward hacking /\nGoodharting** and **specification gaming** in self-training; the promotion gates\nand replay-tested adapter updates are a working small-scale study in **scalable\noversight** and **process supervision**. The evolving exam bank addresses\n**benchmark saturation**, **data contamination / eval leakage** (train-test\nseparation is enforced in code, and the private test set is withheld from this\npublic repo for exactly that reason), and **item response calibration** for LLM\nevaluation. The playground and arcade implement **automated curriculum generation**\nand **open-endedness** (game invention gated by a meta-verifier — adjacent to\nPOET/paired open-ended trailblazing and AI-generating algorithms), with **self-play\noracles** (Monte-Carlo ladders, Stockfish, KataGo) supplying supervision in the\nAlphaZero tradition. The reasoning emulator is a study in **test-time compute**,\n**search-augmented generation**, and backtracking (**rewind-with-notes** as\ngarbage-collected chain-of-thought); the salience/relevance work formalizes\n**attention economics** and **memory-augmented agents** (retrieval as interrupt\nrather than query, counterfactually validated). The refinery and annealing\ncounterexample search sit in the **automated conjecture generation** /\n**automated mathematical discovery** lineage (Graffiti, Wagner 2021), with a\nsettled finite conjecture as an existence proof. Throughout, the architecture is\n**neurosymbolic**: a compact DSL and exact combinatorial verifiers wrapped around\nlocal LLMs, with full **provenance** and an honest ledger of negative results.\n\n---\n\nThis repo grew from the blueprint in `BLUEPRINT.md`; the original scaffold notes follow.\n\nWhat exists now:\n\n- A JSONL-backed cognitive branch store.\n- Typed commits/events with provenance.\n- Branch creation, checkout, merge, diff, blame, and bisect.\n- A patch-only Markdown editor with conservation hooks.\n- A benchmark harness that injects long-document corruption and verifies that hooks catch it.\n- A local Ollama-backed Markdown edit agent that writes only verifier-accepted patches.\n\nRun the tests:\n\n```powershell\npython -m pytest\n```\n\nRun tests plus every current experiment:\n\n```powershell\n.\\scripts\\run_all.ps1\n```\n\nRun a real local-model Markdown edit against the sample agreement:\n\n```powershell\n.\\scripts\\run_sample_edit.ps1\n```\n\nRun the accept/retry/block usefulness benchmark:\n\n```powershell\n$env:PYTHONPATH='src'; python -m bcv.usefulness\n```\n\nRun the multi-document sequential edit benchmark:\n\n```powershell\n$env:PYTHONPATH='src'; python -m bcv.corpus_benchmark\n```\n\nRun Cognitive Git recall and coding benchmarks:\n\n```powershell\n$env:PYTHONPATH='src'; python -m bcv.recall_benchmark\n$env:PYTHONPATH='src'; python -m bcv.coding_benchmark --quick\n$env:PYTHONPATH='src'; python -m bcv.coding_benchmark\n```\n\nExport the seed Taste-RL preference and SFT datasets:\n\n```powershell\n$env:PYTHONPATH='src'; python -m bcv.taste\n```\n\nGenerate a scaled synthetic Taste-RL dataset:\n\n```powershell\n$env:PYTHONPATH='src'; python -m bcv.taste_data --variants-per-prompt 40\n```\n\nRun the first verifier-backed graph discovery loop:\n\n```powershell\n$env:PYTHONPATH='src'; python -m bcv.discovery --max-n 6\n```\n\nEvaluate parseable graph conjectures from a proposal file:\n\n```powershell\n$env:PYTHONPATH='src'; python -m bcv.graph_agent --max-n 6 --proposal-file sample_docs/graph_proposals.json\n```\n\nAsk the local Ollama/LM Studio model to propose graph conjectures, then verify them:\n\n```powershell\n$env:PYTHONPATH='src'; python -m bcv.graph_agent --max-n 6 --use-model --max-rules 6\n```\n\nRun a second model proposal pass conditioned on the previous verifier failures:\n\n```powershell\n$env:PYTHONPATH='src'; python -m bcv.graph_agent --max-n 6 --use-model --max-rules 6 --feedback-file .bcv_runs/graph_agent/proposal_evaluation.json\n```\n\nRun the larger research foundry comparison: stateless proposal loops versus an internal-git feedback loop that accumulates accepted conjectures, counterexamples, repairs, and SFT data:\n\n```powershell\n$env:PYTHONPATH='src'; python -m bcv.research_foundry --mode scripted --rounds 3 --max-n 6\n```\n\nRun the same foundry through the local Ollama/LM Studio model:\n\n```powershell\n$env:PYTHONPATH='src'; python -m bcv.research_foundry --mode model --rounds 3 --max-n 6 --max-rules 4\n```\n\nRun the foundry with a post-run scale check that stress-tests every accepted rule and\nrepair at larger n and commits scale falsifications to the ledger:\n\n```powershell\n$env:PYTHONPATH='src'; python -m bcv.research_foundry --mode scripted --rounds 3 --max-n 6 --stress-ns 7 8\n```\n\nRun the self-contained frontier loop: FastContext 4B proposes on the GPU, the CPU\nverifier kills, per-round scale falsifications feed back into the next proposal\nprompt, and every proposal is judged for semantic novelty against the miner's\ntwo-atom hull:\n\n```powershell\n$env:PYTHONPATH='src'; python -m bcv.research_foundry --mode fastcontext --rounds 6 --max-n 6 --max-rules 6 --stress-feedback-ns 7 8 10 --library .bcv_runs/adversary_library.jsonl\n```\n\nHunt counterexamples by simulated annealing inside a conjecture's predicate class\n(finds persist to the adversary library, which any pool can load with --library):\n\n```powershell\n$env:PYTHONPATH='src'; python -m bcv.graph_adversary --expression \"is_connected and is_triangle_free and not is_bipartite\" --ns 9 10 11 12 13 --restarts 40 --steps 10000\n$env:PYTHONPATH='src'; python -m bcv.graph_adversary --report-file .bcv_runs/graph_generalize_rich/generalization_report.json\n```\n\nRun the model-vs-annealer counterexample duel:\n\n```powershell\n$env:PYTHONPATH='src'; python scripts/run_duel.py\n```\n\nRun the reasoning-emulator benchmark (save/load/rewind-with-notes/verifier-check as\nmodel-invocable controls, vs linear CoT, verifier-scored):\n\n```powershell\n$env:PYTHONPATH='src'; python -m bcv.emulator --dataset-path .bcv_runs/graph_repair_hard_rich/hard_heldout.jsonl --model Qwen/Qwen3-1.7B --limit 6\n```\n\nServe the Pro Action Replay MCP server (external agents can step, dump, and poke a\nlive reasoning session; registered in `.mcp.json` as `reasoning-emulator`):\n\n```powershell\n$env:PYTHONPATH='src'; python -m bcv.emulator_mcp\n```\n\nRun the full conjecture refinery for a domain (enumerate -> verify -> stress ->\nanneal-attack -> closure-expand library -> THEOREMS ledger + falsification museum):\n\n```powershell\n$env:PYTHONPATH='src'; python -m bcv.refinery --domain mis --max-n 6 --stress-ns 7 8 10 --restarts 8 --steps 1200\n$env:PYTHONPATH='src'; python -m bcv.refinery --domain coloring --max-n 6 --stress-ns 7 8 10 --restarts 8 --steps 800 --library .bcv_runs/adversary_library.jsonl\n```\n\nTrain a FastContext QLoRA smoke adapter from the foundry's accumulated verifier-repair data:\n\n```powershell\n$env:PYTHONPATH='src'; python -m bcv.research_foundry --mode model --rounds 3 --max-n 6 --max-rules 4 --train-adapter\n```\n\nGenerate a larger verifier-repair dataset for graph conjecture repair:\n\n```powershell\n$env:PYTHONPATH='src'; python -m bcv.graph_repair_data --max-n 6 --max-proposals 48 --root .bcv_runs/graph_repair_data_full\n```\n\nTrain a FastContext QLoRA graph-repair adapter on JSON-only verifier targets:\n\n```powershell\n$env:PYTHONPATH='src'; python -m bcv.graph_lora --dataset-path .bcv_runs/graph_repair_data_full/repair_json_sft.jsonl --output-dir .bcv_runs/graph_lora_json_24_r4 --max-train-examples 24 --heldout-examples 8 --epochs 1 --max-length 256 --lora-r 4 --lora-alpha 8 --max-n 6 --train-only\n```\n\nGenerate the hard repair dataset (the prompt contains counterexample evidence but not\nthe answer constraint, and heldout groups share no original expression with train):\n\n```powershell\n$env:PYTHONPATH='src'; python -m bcv.graph_repair_data --hard --max-n 6 --max-proposals 48 --heldout-groups 8 --root .bcv_runs/graph_repair_hard\n```\n\nTrain and strictly evaluate the hard-task adapter (eval requires the output to be a\nverifier-accepted strict refinement of the prompt's original expression, and reports\nsupport retention plus mode-collapse diagnostics):\n\n```powershell\n$env:PYTHONPATH='src'; python -m bcv.graph_lora --dataset-path .bcv_runs/graph_repair_hard/hard_train.jsonl --heldout-path .bcv_runs/graph_repair_hard/hard_heldout.jsonl --output-dir .bcv_runs/graph_lora_hard_29_r8 --max-train-examples 29 --epochs 2 --max-length 640 --lora-r 8 --lora-alpha 16 --max-n 6\n```\n\nStress-test every rule and repair accepted at n<=6 on larger graphs (random G(n,p)\nplus deterministic adversaries: interleaved crown graphs and greedy-adversarial trees):\n\n```powershell\n$env:PYTHONPATH='src'; python -m bcv.graph_generalize --evaluation-file .bcv_runs/graph_repair_hard/proposal_evaluation.json --ns 7 8 --samples-per-np 120 --relabels 3\n```\n\nBuild the stress-mined hard dataset, whose repair targets must also survive sampled\nand adversarial graphs at larger n (targets stop being horizon artifacts by construction):\n\n```powershell\n$env:PYTHONPATH='src'; python -m bcv.graph_repair_data --hard --max-n 6 --max-proposals 48 --heldout-groups 6 --stress-ns 7 8 --root .bcv_runs/graph_repair_hard_stress\n```\n\nRun the first benchmark:\n\n```powershell\n$env:PYTHONPATH='src'; python -m bcv.benchmark\n```\n\nRun every current probe and write ledgers/training candidates:\n\n```powershell\n$env:PYTHONPATH='src'; python -m bcv.experiments\n```\n\nRun only the local-model document probe:\n\n```powershell\n$env:PYTHONPATH='src'; python -m bcv.model_probe\n```\n\nTrain the current learned document-corruption tripwire:\n\n```powershell\n$env:PYTHONPATH='src'; python -m bcv.tripwire\n```\n\nExport verifier-ledger training datasets:\n\n```powershell\n$env:PYTHONPATH='src'; python -m bcv.sft_export\n```\n\nRun the cached FastContext 4B QLoRA smoke trainer:\n\n```powershell\n$env:PYTHONPATH='src'; python -m bcv.lora_smoke\n```\n\nThe benchmark is deliberately deterministic. It does not prove an LLM will edit safely. It proves the state and verifier layer can distinguish a clean patch from a corrupt full-document rewrite before a model is plugged in.\n",
  "bytes": 19122,
  "sha": "535174be42f99280ecb258913686008891c3dc5de1438000bcb7648d71bce59e",
  "repo_slug": "carlsr9001/whetstone",
  "fonte": "repo",
  "truncated": false,
  "api": "https://agentalog.com/api/listings/mcp_link_cyberelf_whetstone_tools_d67ffbe0/readme"
}