{
  "markdown": "# NodeBench AI\n\nEntity intelligence for any company, market, or question.\n\n**Live:** [nodebenchai.com](https://www.nodebenchai.com)  \n**npm:** `npx nodebench-mcp` / `npx nodebench-mcp-power` / `npx nodebench-mcp-admin`  \n**GitHub:** [HomenShum/nodebench-ai](https://github.com/HomenShum/nodebench-ai)\n\n**New to this codebase?** Read [`docs/START_HERE.md`](docs/START_HERE.md) — it\nfollows one user's question through the code in the order it actually runs, then\n[`.tours/`](.tours/) walks the same path inside the live source in VS Code\n(CodeTour extension). [`docs/codebase/`](docs/codebase/) has the stack,\nstructure, conventions, integrations, testing and the honest list of known\nproblems. [`docs/SIMPLIFICATION_REPORT.md`](docs/SIMPLIFICATION_REPORT.md)\nrecords what was measured and removed, with the command for every number.\n\n![Live graph rail replaying a committed eval transcript: 34 entities stream in, then a hover isolates one node's neighbourhood and a drag repositions it](demo/graph-rail/graph-rail-clip.gif)\n\n*Recorded from `demo/graph-rail/` replaying a committed eval transcript (persona-episode pack, 2026-01-05) through `session.observe()` — recorded events replayed, not re-verified; every label is a literal fixture substring, edges are traversal history, never evidence. Regenerate: `node scripts/record-graph-rail-clip.mjs`.*\n\n![NodeBench AI Home surface: research prompt with Quick answer / Deep research modes, and reusable public research cards below](.qa/evidence/2026-07-15-runtime-grounded-control-focus/baseline/exact-surfaces/home-desktop-light.png)\n\n## Product\n\nNodeBench is a research and reporting product built around five user-facing\nsurfaces:\n\n- `Home` = start quickly\n- `Reports` = reusable memory\n- `Chat` = do the work\n- `Inbox` = captures, nudges, alerts, automations, and unassigned review\n- `Me` = operator context and control\n\nDeep research opens in the separate Workspace surface at `nodebench.workspace`;\nit is not a sixth tab in the operating app.\n\nThe core idea is simple:\n\nusers do not just need a chatbot that answers once.\n\nThey need a system that can:\n\n- take a question, file, URL, or prior thread\n- search and synthesize with sources\n- turn the run into a reusable artifact\n- watch for meaningful change later\n- improve the next run from what it learned\n\n## Pending Release Candidate: Runtime-Grounded Control Focus\n\nThis candidate is not yet merged or deployed. It narrows the reachable product\nto controls backed by a real runtime capability and makes unavailable state\nexplicit:\n\n- Agents keeps ask, explicit swarm launch, active work, approvals, sources,\n  traces, receipts, exports, and provenance while removing projected metrics,\n  synthetic fallbacks, duplicate actions, and local-only controls.\n- Generated code remains copyable and exportable but never executes inside the\n  signed-in app origin. Runtime cards accept successful structured tool output,\n  not assistant prose, guessed search fields, or incomplete/error states.\n- Canonical owner-scoped streaming remains; an unreferenced bearer-stream\n  component and HTTP route were removed instead of retained as a second path.\n- Home, Reports, Chat, Inbox, and Me use live runtime data, owner-scoped where\n  private, or honest loading, empty, not-found, and unmeasured states instead of\n  reachable product fixtures. Desktop and mobile now share the same Inbox and\n  Me component trees; duplicate queue/profile renders and their hard-coded plan,\n  usage, connector, and local-draft projections were deleted.\n- Pipeline launches and schedules require an authenticated server identity,\n  while guest history, detail, bundle, stream, and evaluation reads require an\n  anonymous-session possession credential. Browser-supplied owner keys are no\n  longer an authority boundary, and trusted cron or MCP work stays on internal\n  contracts.\n- Authenticated launches consume durable per-owner admission: four units per\n  ten minutes and thirty per day, with composed runs costing two units. Server\n  input bounds and per-owner schedule caps close the remaining cost-abuse path.\n- Fresh pipeline attempts and recurring schedule occurrences are isolated by\n  durable attempt identities and execution-generation fences, so retries can\n  resume the same attempt without stale or overlapping workers rewriting it.\n- Research distinguishes sources consulted from citations actually bound in\n  the synthesis. Missing, malformed, or unbound evidence cannot be promoted to\n  a verified claim, and chat no longer injects citation markers into prose.\n- Operator telemetry and maintenance are deferred until deliberate disclosure\n  and require server-confirmed operator access.\n- Task sessions, traces, steps, evidence, approvals, swarms, and operational\n  history are owner-scoped. The unreachable parallel timeline, kanban, hook,\n  and public orchestrator were removed; due diligence retains only a bounded,\n  owner-checked internal task substrate. Unused public due-diligence, investor,\n  demo, and evaluation entrypoints were internalized or removed, and retained\n  job, branch, memo, and catch-path writes require the exact owner chain. Raw\n  orchestrator writes use internal contracts, MCP document and spreadsheet\n  operations require exact object ownership, and a TRACE completion fails\n  closed unless its receipt persists with an exact output hash.\n\nRelease evidence is intentionally pending. The source PR must merge through the\nrequired CI gates first; a follow-up evidence-seal PR must then record the\ncanonical squash SHA, exact-revision checks, preview assertions, production\ndeployment revision, and direct production browser verification before this\nsection can be described as shipped.\n\n## What Shipped\n\n- five-surface web app across `Home`, `Reports`, `Chat`, `Inbox`, and `Me`\n- separate deep-work Workspace shell at `nodebench.workspace`\n- typed search and reporting pipeline\n- hosted public research MCP for external apps and agents:\n  `https://nodebench-mcp-unified.onrender.com?profile=public-research`\n- Pi-AI pipeline lane on `Reports` with code-gen, design-gen, research,\n  composed runs, schedules, streaming previews, eval scorecard, and MCP HTTP\n  bridge\n- live SSE streaming with saved runtime state\n- Convex-backed product state for sessions, reports, entities, nudges, files,\n  and related objects\n- shared-context handoff and delegation plumbing\n- local and deployed server runtime for search, streaming, voice, and shared\n  context routes\n- `nodebench-mcp`, `nodebench-mcp-power`, and `nodebench-mcp-admin`\n  distribution lanes\n- builder-facing Oracle, dogfood, eval, replay, and control-plane\n  infrastructure\n\n## Live graph rail on a LIVE Convex backend\n\n### The rail inside the product UI\n\n![EntityProfilePage at /#entity/Anthropic loading against a live Convex dev deployment: the Live Graph rail populates reactively from the page's own queries, then grows by one node on camera when a real storeEntityContext write lands](demo/graph-rail-live/product-rail-live.gif)\n\n*The rail lives in the real product page —\n`apps/web/src/features/research/components/EntityGraphRail.tsx`, mounted in\n`EntityProfilePage` (`/#entity/<name>`, the grammar digest links emit). It is\nfed by the SAME reactive queries the page already holds\n(`entityContexts.getEntityContext`, `relationshipGraph.getEntityGraph`,\n`adaptiveEntityQueries.getAdaptiveProfile`) — zero new subscriptions. Counts\nstay undefined and every edge is traversal: research prose measures nothing.\nRegenerate: `node demo/graph-rail-live/record-product-rail.mjs` (exits\nnonzero if the rail is missing, stays under 4 nodes, or fails to grow after\nthe live write). See `docs/GRAPH_INTEGRATION.md` for the event taxonomy.*\n\n### The standalone live viewer\n\n![The rail at zero, then NodeBench's real storeEntityContext mutations landing on a live dev deployment, the rail populating reactively over WebSocket to 12 entities](demo/graph-rail-live/live-convex-rail.gif)\n\n*No replay file: `demo/graph-rail-live/` subscribes to the real\n`domains/knowledge/entityContexts:getEntityContext` query on an isolated dev\ndeployment, and the repo's own seed script writes through the real mutation\non camera. All edges traversal — research prose is not a measurement.\nRegenerate: `node demo/graph-rail-live/record-live.mjs <dev-deployment-url>`\n(see the script header for the one-time deployment setup).*\n\n## Live graph rail (recorded events)\n\n![Live graph rail replaying recorded research events](demo/graph-rail/graph-rail.png)\n\n`demo/graph-rail/` feeds the [NodeGraph Live](https://github.com/HomenShum/NodeGraph)\nrenderer (vendored at `vendor/nodegraph-live/`, pending npm publish) with real\nentity-research events from a committed eval transcript\n(`benchmarks/history/archived-2026-q1/persona-episode-eval-pack-20260105-153100.json`)\n— it is a replay of recorded events, not a live agent run: every label is a\nverbatim string from that fixture, all counts render as unknown because the\ntranscript holds no measured pair counts, and no assertion edges are drawn\nbecause no record carries a complete source receipt.\n`node scripts/capture-graph-rail.mjs` regenerates the capture and exits nonzero\non an empty rail, any console error, or any rendered label that does not appear\nin the fixture.\n\n## Hosted Public Research MCP\n\nNodeBench can be used as a public research memory and tool server from any\nagent or app without forcing signup before the first useful result.\n\nUse the hosted MCP endpoint:\n\n```txt\nhttps://nodebench-mcp-unified.onrender.com?profile=public-research\n```\n\nFor Gmail/job-match style integrations, use the smaller profile:\n\n```txt\nhttps://nodebench-mcp-unified.onrender.com?profile=gmail-research\n```\n\nPublic profiles are anonymous by default, but still metered. Responses include:\n\n```txt\nx-nodebench-request-id\nx-nodebench-profile\nx-nodebench-auth-mode\nx-nodebench-account-key\n```\n\nApps should send stable, non-sensitive client headers:\n\n```http\nx-nodebench-client: your-app-name\nx-nodebench-client-version: 1.0.0\nx-nodebench-client-id: stable-install-or-workspace-id\n```\n\n`x-nodebench-client-id` lets NodeBench attribute anonymous usage and estimated\ncosts without requiring a NodeBench token. Do not put private email text,\nresume text, API keys, or user secrets in this header.\n\n### Progressive Sign-In And Linking\n\nThe intended user flow is:\n\n```text\nFirst public dossier works without signup\n  -> show sources, freshness, and confidence\n  -> offer \"Link NodeBench\" after value is visible\n  -> linked users get stable history, higher budgets, team usage, webhooks,\n     token management, billing controls, and reusable private workspace context\n```\n\nDo not block public-source research behind login. Promote sign-in when the user\nwants persistence, shared team memory, budget controls, private workspace\nlinking, or API/MCP tokens.\n\nSee [MCP_TOOL_PROFILES.md](./docs/guides/MCP_TOOL_PROFILES.md) for the full\nprofile list, tool catalog, account attribution, and cost tracking contract.\n\n## Product At A Glance\n\n```text\nUSER SURFACES\n-------------\nHome      -> start quickly\nReports   -> reusable report memory\nChat      -> answer, sources, trace, follow-ups\nInbox     -> captures, nudges, automations, alerts, unassigned items\nMe        -> operator context, permissions, controls\n\nBACKEND\n-------\nConvex tables and product state for sessions, reports, entities, nudges,\nfiles, shared context, and evaluation artifacts\n\nRUNTIME\n-------\nsearch pipeline\n  -> answer packet\n  -> saved report\n  -> tracked entity / tracked theme / follow-up task\n  -> nudge or prep brief\n  -> resumed chat or reopened report\n\nCOMPOUNDING LOOP\n----------------\nquestion\n  -> answer\n  -> saved report\n  -> watch item\n  -> useful nudge\n  -> better next run\n\nDISTRIBUTION\n------------\nnodebenchai.com\nnodebench.workspace\nnodebench-mcp\nnodebench-mcp-power\nnodebench-mcp-admin\n```\n\n## Event Intelligence Serving Model\n\nEvent serving extends the same budgeted search route used by the main app and\nMCP runtime. NodeBench treats search as a memory-building operation with a\nbudget. For events, NodeBench checks the event corpus and workspace memory\nbefore live search, then persists useful results as entities, claims, sources,\nand workspace context.\n\nScratchNode is the lightweight live-event sidecar for this model: a disposable\nroom that turns public chat and sourced `/ask` answers into a public wiki while\nkeeping attendee notes private. It complements Luma, Slack, Eventbrite, and\nother event surfaces instead of replacing them; the explicit handoff to\nNodeBench opens `https://nodebenchai.com/events/:eventSlug/private` with\nprivate-note continuation context and no URL-borne `ownerKey`.\n\nThe event flow is:\n\n```text\nBefore event\n  -> build event corpus\n\nDuring event\n  -> capture messy notes instantly\n\nAfter event\n  -> turn captures into report, cards, follow-ups, and reusable memory\n```\n\nThe product model is:\n\n```text\nScratchNode sidecar room + public event corpus + private NodeBench continuation\n```\n\nEvent corpus and capture data stay separated:\n\n- `Shared event corpus` = public event info, speakers, sponsors, company pages,\n  sessions, and public source cache.\n- `Private captures` = what a user personally heard, wrote, recorded, or\n  photographed; ScratchNode private notes never enter the public feed, public\n  wiki, or public `/ask` cache.\n- `Team/org memory` = shared only inside the fund, company, or workspace.\n- `Event aggregate insights` = opt-in or anonymized only.\n\nDuring the event, most captures should hit the event corpus first and avoid paid\nsearch:\n\n```text\nvoice memo / text / screenshot\n  -> captureRouter\n  -> active event corpus\n  -> entity and claim extraction\n  -> active event session attachment\n  -> budget policy\n  -> ack + next action\n```\n\nExample mobile ack:\n\n```text\nSaved to Ship Demo Day session\nDetected 1 person | 1 company | 2 claims | 1 follow-up\nUsing event corpus | 0 paid calls\n```\n\nAfter the event, the report opens in `nodebench.workspace`:\n\n```text\nBrief      -> post-event memo\nCards      -> people, companies, products, themes\nNotebook   -> raw notes, transcripts, screenshot OCR, cleaned notes\nSources    -> field notes, public evidence, verification status\nChat       -> follow-up questions and deeper refreshes\nMap        -> graph view later\n```\n\nThe canonical spec lives in\n[EVENT_INTELLIGENCE_SERVING_MODEL.md](./docs/architecture/EVENT_INTELLIGENCE_SERVING_MODEL.md).\nThe ScratchNode/NodeBench privacy boundary lives in\n[SCRATCHNODE_NODEBENCH_BOUNDARY.md](./docs/architecture/SCRATCHNODE_NODEBENCH_BOUNDARY.md).\n\n## Why This Design\n\nNodeBench is designed around a few product realities:\n\n1. A useful answer should not disappear after one chat turn.\n2. Saved work should become reusable memory, not a dead archive row.\n3. The product should bring the user back only when something meaningful\n   changes.\n4. The system should gradually learn how the user works without forcing a heavy\n   onboarding flow.\n5. Operator context should improve future runs without turning the system into\n   corporate-speak or fake-agreeable sludge.\n\nThat drives the current design:\n\n- answer-first execution\n- advisor mode by design via dynamic routing:\n  - fast executive lane for routine work\n  - deeper advisor lane for ambiguity, planning, and harder reasoning\n  - similar in spirit to Claude Code's official `opusplan` split:\n    stronger planning lane, cheaper execution lane\n- saved artifacts as first-class objects\n- visible sources and traceability\n- a five-page loop instead of five unrelated tabs\n- future `Harness v2` work focused on specification, operator context, and\n  compounding behavior\n\nPlain English:\n\n```text\nNodeBench should not spend the most expensive reasoning path on every request.\nIt should move fast by default, then go deeper when the task, evidence, or user\nrequest justifies it.\n```\n\nThe detailed implementation, verification, and evaluation plan for this mode\nlives in:\n\n- [HARNESS_V2_PROPOSAL.md](./docs/archive/2026-q1/architecture-superseded/HARNESS_V2_PROPOSAL.md) (archived)\n- [HARNESS_V2_BUILD_PLAN.md](./docs/archive/2026-q1/architecture-superseded/HARNESS_V2_BUILD_PLAN.md) (archived)\n\n## How The Five Pages Compound\n\nNodeBench should not feel like five separate destinations.\n\nThe intended product behavior is:\n\n```text\nHome\n  -> start quickly\n\nReports\n  -> turn that artifact into reusable memory\n\nChat\n  -> do the work\n  -> create the first useful artifact\n\nInbox\n  -> triage captures, nudges, automations, alerts, and unassigned items\n\nMe\n  -> improve how the next run is handled\n\nWorkspace\n  -> open deep Brief / Cards / Notebook / Sources / Chat / Map work\n  -> lives at nodebench.workspace, not in the operating tab bar\n\nNext Home or Chat run\n  -> starts with more context than before\n```\n\nThe shortest version of the compounding loop is:\n\n```text\nquestion\n  -> answer\n  -> saved report\n  -> watch item\n  -> useful nudge\n  -> better next run\n```\n\nPlain-English artifact flow:\n\n```text\ninput\n  -> answer packet\n  -> saved report\n  -> tracked entity / tracked theme / follow-up task\n  -> nudge or prep brief\n  -> resumed report or resumed chat\n  -> user correction or confirmation\n  -> updated operator context\n  -> better next run\n```\n\nWhat each page contributes:\n\n- `Home` starts the run with the least friction possible\n- `Reports` turns those into a durable report the user can reopen, refresh, and\n  reuse\n- `Chat` creates the answer, sources, trace, entities, and next actions\n- `Inbox` collects nudges, captures, automations, alerts, and unassigned items\n- `Me` stores the operator context that improves the next answer\n- `Workspace` owns recursive cards, notebook editing, source verification, and\n  long-lived intelligence memory\n\n## Current Legacy Infrastructure\n\nNodeBench is not starting from zero. The repo already contains a substantial\nlegacy stack that works today.\n\nCurrent legacy foundation:\n\n- five-surface web product\n- Convex-backed canonical data layer\n- local and deployed server runtime\n- harness v1 planning and execution path\n- shared-context handoff and delegation support\n- MCP distribution lanes\n- builder-facing evaluation and control-plane systems\n\nWhat that means:\n\n- the problem is not missing architecture\n- the problem is product behavior, workflow compression, and clearer\n  cross-surface compounding\n\n## Roadmap\n\nThe near-term goal is:\n\n```text\nkeep the working legacy foundation\nremove accidental complexity\nadd specification-aware operator context\nship one clear compounding workflow\n```\n\nMain tasks still to finish:\n\n- [ ] make `Home -> Reports -> Chat -> Inbox -> Me` behave like one continuous\n      workflow instead of five adjacent surfaces\n- [ ] turn harness v1 into the clearer v2 shape described in\n      [HARNESS_V2_PROPOSAL](docs/archive/2026-q1/architecture-superseded/HARNESS_V2_PROPOSAL.md)\n- [ ] ship `Layer 0` operator context so the system can learn useful workflow\n      patterns without forcing a heavy onboarding flow\n- [ ] support permissioned transcript ingestion from NodeBench chats first, then\n      optional external logs such as Claude Code JSONL transcripts for\n      `nodebench-mcp`\n- [ ] add style-drift guardrails so the system learns judgment and workflow\n      without overfitting to corporate voice, filler, or sycophancy\n- [ ] add anticipatory prep behavior so the system can prepare the user before\n      important interactions, not only answer after the fact\n- [ ] make saved reports behave like reusable memory, not storage\n- [ ] keep `Nudges` as an Inbox section with at least one working daily trigger\n- [ ] make `Me` clearly improve future runs by exposing what context is being\n      used and why\n- [ ] finish the `nodebench-mcp` v3 cut-and-split plan so default runtime,\n      power runtime, and admin runtime are clearly separated\n- [ ] instrument real latency, real cost, real artifact completion, and real\n      reuse across both web and MCP flows\n- [ ] keep README, runtime behavior, and exposed tool counts in sync so the\n      public story matches the actual system\n- [ ] keep dogfood, eval, and builder-control infrastructure as internal\n      leverage instead of letting it leak into the main user-facing product\n\n## Quick Start\n\n### Web app\n\nOpen [nodebenchai.com](https://www.nodebenchai.com) and start in `Home`.\n\n### MCP\n\n```bash\n# Claude Code\nclaude mcp add nodebench -- npx -y nodebench-mcp\n\n# Claude Code power lane\nclaude mcp add nodebench-power -- npx -y nodebench-mcp-power\n\n# Claude Code admin lane\nclaude mcp add nodebench-admin -- npx -y nodebench-mcp-admin\n\n# Cursor\nnpx nodebench-mcp --preset cursor\n\n# Generic MCP client\nnpx nodebench-mcp\n```\n\n### Local development\n\n```bash\ngit clone https://github.com/HomenShum/nodebench-ai.git\ncd nodebench-ai\nnvm use            # Node version from .nvmrc\nnpm install\n\n# Frontend only. This is the honest first run: it starts, and every product\n# route shows the \"Convex backend not configured\" card until you supply a real\n# VITE_CONVEX_URL. That card is the designed state, not a failure.\nnpx vite --port 5173\n\n# Production build\nnpm run build\n```\n\n`npm run dev` runs three processes in parallel (`vite`, `convex dev`,\nand the voice worker). Two of them block on credentials — `convex dev` wants an\ninteractive Convex login and the voice worker wants `.env.local` — so it is not\nthe command to start with on a fresh clone.\n\n`cp .env.example .env.local` on its own does **not** get you a working backend:\nthe file ships `VITE_CONVEX_URL=https://your-project.convex.cloud`, which is a\nplaceholder and is rejected on purpose (`apps/web/src/lib/convexUrl.ts`). Run\n`npx convex dev` to provision a real deployment, then put its URL there.\n\n## Architecture\n\n```text\nnodebenchai.com (React + Vite + Tailwind)\n    |\nConvex Cloud (sessions, reports, entities, nudges, files, product state)\n    |\nserver runtime + search pipeline + SSE\n    |\nanswer packet\n    |\nsaved report\n    |\ntracked entities / watch conditions / nudges\n    |\nfuture runs with better operator context\n```\n\n## Student Learning Lessons\n\nThe notebook and diligence stack in this repo are a good example of a common\nproduct engineering tradeoff:\n\n- the best user experience is one notebook that feels continuous\n- the safest current runtime is still layered and block-addressable underneath\n\nFor NodeBench, that means:\n\n- `founder` is a trait and diligence block, not a permanent sixth tab\n- diligence should use one generic pipeline, not many narrow `*Identify.ts`\n  features\n- the runtime should stay `scratchpad-first -> structuring pass ->\n  deterministic merge`\n- user-owned prose should feel local-first and calm while typing\n- live agent output should arrive as overlays or decorations first, not as\n  direct document mutations\n- accepted agent output should become frozen, user-owned notebook content\n- provenance should stay available, but secondary to the reading and writing\n  flow\n\nWhy the notebook does **not** use one giant live editor model yet:\n\n- collaboration is more reliable when the system can address bounded sections\n- provenance, evidence, and contribution logs need stable attachment points\n- background agent updates should not compete with user keystrokes\n- deterministic section-level merge is easier to reason about than whole-page\n  mutation churn\n\nThe practical rule in this repo is:\n\n```text\nUX should feel monolithic.\nRuntime should stay layered.\nTyping should be local-first.\nAgent output should be overlay-first.\nAccepted output should become owned prose.\n```\n\nCurrent notebook refactor lessons:\n\n- hide the block machinery from the reading path\n- keep chrome quiet and move metadata to hover or focus\n- isolate the notebook surface from page-level re-render churn\n- favor one memoized notebook boundary over many inline object props\n- treat live diligence as read-only reference overlay until the user accepts it\n- when accepted, materialize a frozen notebook snapshot with explicit provenance\n- anchor live overlays at the notebook surface, not inside the first editable row\n- let Convex projection rows carry real source metadata so the UI is not forced to reconstruct trust state from prose alone\n- use one generic projection producer for overlays: report save writes the same structured rows that page-load backfill and manual refresh re-run\n- when moving beyond report-backed overlays, stream raw scratchpad only in a secondary rail and emit structured projection rows on checkpoint rather than dumping scratchpad prose into the notebook body\n- if checkpoint structure comes from an LLM, keep it block-scoped and schema-bound: `scratchpad checkpoint -> JSON -> validation/repair -> deterministic fallback -> projection row`\n- let the model structure intermediate JSON, but keep merge, persistence, and notebook ownership deterministic\n- ship generic diligence primitives first, then block-specific renderers\n\nFor students reading the code, the most relevant docs are:\n\n- [`AGENT_PIPELINE`](docs/architecture/AGENT_PIPELINE.md)\n- [`DILIGENCE_BLOCKS`](docs/architecture/DILIGENCE_BLOCKS.md)\n- [`SCRATCHPAD_PATTERN`](docs/architecture/SCRATCHPAD_PATTERN.md)\n- [`PROSEMIRROR_DECORATIONS`](docs/architecture/PROSEMIRROR_DECORATIONS.md)\n- [`SESSION_ARTIFACTS`](docs/architecture/SESSION_ARTIFACTS.md)\n\nThe live notebook refactor is deliberately incremental:\n\n- current shipped slices make the notebook feel more continuous and reduce\n  per-keystroke render churn\n- current shipped slices also move live diligence into notebook-surface\n  overlays instead of seeded block-like records and freeze accepted snapshots\n- the end state is one notebook experience with layered internals, not a raw\n  block UI and not a brittle giant document runtime\n\n### Key tech\n\n- Frontend: React, Vite, TypeScript, Tailwind CSS\n- Backend: Convex\n- Search: Linkup + Gemini extraction + grounding pipeline\n- MCP server: Node.js + TypeScript\n- Realtime runtime: SSE + Convex-backed persistence\n\n## API Keys\n\nSet these in `.env.local` for local work or in Convex / Vercel for deployed\nenvironments.\n\n| Key | Required | Purpose |\n| --- | --- | --- |\n| `GEMINI_API_KEY` | Yes | classification, extraction, synthesis |\n| `LINKUP_API_KEY` | Recommended | web search and sourced answers |\n| `VITE_CONVEX_URL` | Yes | Convex deployment URL |\n\n## Codebase map\n\nTop-3 levels, annotated. See [`docs/ARCHITECTURE.md`](docs/ARCHITECTURE.md) for the\npipeline diagram and [`docs/architecture/README.md`](docs/architecture/README.md)\nfor the 13 canonical architecture docs.\n\n```text\nnodebench-ai/\n├── README.md                   ← you are here\n├── CONTRIBUTING.md             ← contribution bar\n├── CLAUDE.md                   ← Claude Code conventions for this repo\n├── AGENTS.md                   ← agent methodology + eval bench\n├── LICENSE                     ← MIT\n│\n├── apps/\n│   └── web/src/                ← React frontend (Vite)\n│       ├── features/           ← feature-first (Home · Reports · Chat · Inbox · Me · Workspace · entities · agents · …)\n│       │   └── <feature>/      ← views · components · hooks · lib · __tests__ (colocated)\n│       ├── shared/             ← shared UI primitives, hooks, utils\n│       ├── lib/                ← registry, analytics, error reporting\n│       └── layouts/            ← shell + cockpit + public\n│\n├── workers/node/               ← Node runtime (Express + MCP gateway)\n│   ├── pipeline/               ← agent harness runtime + diligence blocks\n│   ├── routes/                 ← HTTP routes (search, harness, founder episodes)\n│   ├── mcpGateway.ts           ← WebSocket MCP gateway\n│   └── services/               ← shared services\n│\n├── backend/convex/             ← Convex backend\n│   ├── domains/                ← domain folders (agents · product · research · founder · search · …)\n│   ├── schema.ts               ← database schema (includes agentScratchpads)\n│   └── crons/                  ← scheduled jobs\n│\n├── packages/\n│   ├── mcp-local/              ← the published nodebench-mcp npm package (MIT)\n│   ├── mcp-client/             ← typed client SDK\n│   └── convex-mcp-nodebench/   ← Convex-side MCP auditor\n│\n├── .claude/\n│   ├── README.md               ← map of the .claude/ layout\n│   ├── rules/                  ← 31 modular rules with related_ cross-refs\n│   ├── skills/                 ← reusable how-to procedures\n│   ├── agents/                 ← subagent configs\n│   └── commands/               ← custom slash commands\n│\n├── docs/\n│   ├── README.md               ← docs tree map\n│   ├── ARCHITECTURE.md         ← top-level pipeline diagram\n│   ├── ONBOARDING.md           ← 30-minute new-contributor path\n│   ├── architecture/           ← 13 canonical specs + plans/ + README index\n│   ├── agents/                 ← agent docs + bootstrap configs\n│   ├── guides/                 ← how-to for builders\n│   ├── decisions/              ← ADRs\n│   ├── changelog/              ← release notes\n│   ├── product/                ← product decisions\n│   ├── qa/                     ← QA protocols\n│   └── archive/                ← superseded content, provenance-only\n│\n├── evals/\n│   └── e2e/                    ← end-to-end eval suites\n│\n├── scripts/                    ← dogfood, eval harness, one-offs\n└── public/                     ← static assets served by Vite + Vercel\n```\n\n## Related Docs\n\n**Start here:** [`docs/ONBOARDING.md`](docs/ONBOARDING.md) · [`docs/ARCHITECTURE.md`](docs/ARCHITECTURE.md) · [`docs/architecture/README.md`](docs/architecture/README.md)\n\nThe 13 canonical architecture docs are organized in 4 tiers. See [`docs/architecture/README.md`](docs/architecture/README.md) for the indexed map:\n\n- **Tier 1 (core pipeline):** [`AGENT_PIPELINE`](docs/architecture/AGENT_PIPELINE.md) · [`DILIGENCE_BLOCKS`](docs/architecture/DILIGENCE_BLOCKS.md) · [`USER_FEEDBACK_SECURITY`](docs/architecture/USER_FEEDBACK_SECURITY.md)\n- **Tier 2 (sub-patterns):** [`SCRATCHPAD_PATTERN`](docs/architecture/SCRATCHPAD_PATTERN.md) · [`PROSEMIRROR_DECORATIONS`](docs/architecture/PROSEMIRROR_DECORATIONS.md) · [`AGENT_OBSERVABILITY`](docs/architecture/AGENT_OBSERVABILITY.md) · [`SESSION_ARTIFACTS`](docs/architecture/SESSION_ARTIFACTS.md)\n- **Tier 3 (features):** [`FOUNDER_FEATURE`](docs/architecture/FOUNDER_FEATURE.md) · [`REPORTS_AND_ENTITIES`](docs/architecture/REPORTS_AND_ENTITIES.md) · [`AUTH_AND_SHARING`](docs/architecture/AUTH_AND_SHARING.md)\n- **Tier 4 (cross-cutting):** [`MCP_INTEGRATION`](docs/architecture/MCP_INTEGRATION.md) · [`EVAL_AND_FLYWHEEL`](docs/architecture/EVAL_AND_FLYWHEEL.md) · [`DESIGN_SYSTEM`](docs/architecture/DESIGN_SYSTEM.md)\n\nActive architecture addenda:\n[`GRAPH_SEARCH_AGENT_CONTEXT`](docs/architecture/GRAPH_SEARCH_AGENT_CONTEXT.md)\ncaptures the graph/search/agent-context strategy, exact product questions,\nscale projection, node attention model, and human-vs-agent retrieval split.\n\nHistorical specs are preserved in [`docs/archive/2026-q1/`](docs/archive/2026-q1/INDEX.md).\n\n## Production Readiness & Evaluation\n\nNodeBench ships with a comprehensive evaluation harness that proves correctness\nacross 32+ scenarios, 9 user personas, and 9 feature categories. This is not\nhand-wavy \"it works\" — it is measured, versioned, and reproducible.\n\n### Latest Published Run Results\n\n**Pi-AI pipeline cascade:** merged to `main` on 2026-04-30 at\n`2a541037874c0f8c675ab393d5c08f50123cf6d2`.\n\n| Lane | Result |\n|------|--------|\n| PR chain | #211 -> #212 -> #213 -> #214 -> #215 -> #216 all merged |\n| Production surface | `https://www.nodebenchai.com/?surface=packets` |\n| MCP bridge | `https://agile-caribou-964.convex.site/mcp/pipeline/*` behind `MCP_SECRET` |\n| Code-gen run | `pipeline_mokobe4y_6n23be` succeeded, `verified`, 6 files, 32.9s, about `$0.001` |\n| Research streaming | `pipeline_mokpvi1b_yoj8ot` completed with 4,317 streamed characters |\n| Linkup research | `pipeline_mol2wj2j_2lgx2u` succeeded with 18 snippets across 5 sub-questions |\n| Composed pipeline | `research_then_code` completed stage 1 research and stage 2 code-gen |\n| Schedule workflow | once schedule swept by cron/manual sweep and auto-disabled after run |\n| Design output | design-gen produced a PNG stored in Convex storage |\n| UI launcher | DOM-submitted composed run updated the reactive run list |\n| Pipeline scorecard | 41.7% verified, Brier 0.135 across 12 runs |\n\nThe implementation is mounted on the Reports surface:\n\n- `PipelineLauncher`\n- `PipelineSchedulesPanel`\n- `PipelineEvalScorecard`\n- `PipelineRunsPanel`\n- `EntityFindingsPanel`\n\nThe detailed handoff is in\n[`docs/handoff/PI_AI_PIPELINES_HANDOFF.md`](docs/handoff/PI_AI_PIPELINES_HANDOFF.md).\n\n**Workflow-loop eval bank:** added on 2026-04-30 to test the full product loop,\nnot just answer text.\n\n```text\nquery / capture\n  -> memory search\n  -> entity resolution\n  -> report update\n  -> notebook update\n  -> graph edges\n  -> sources / claims\n  -> follow-up / export\n```\n\n| Eval bank | Result |\n|-----------|--------|\n| Total workflow cases | 124 |\n| Minimum P0 suite | 30 cases |\n| Coverage categories | 11 |\n| Score dimensions | 12 |\n| Validator | `apps/web/src/features/evaluation/data/nodebenchWorkflowEvalBank.test.ts` |\n| Latest local check | `npx vitest run apps/web/src/features/evaluation/data/nodebenchWorkflowEvalBank.test.ts` -> 4/4 passed |\n\nThe eval bank lives in\n[`apps/web/src/features/evaluation/data/nodebenchWorkflowEvalBank.ts`](apps/web/src/features/evaluation/data/nodebenchWorkflowEvalBank.ts).\n\n### Two-Layer Judge Architecture\n\nEvery production run is evaluated by two independent systems:\n\n**Layer 1: Deterministic Boolean Gates** (`server/pipeline/diligenceJudge.ts`)\n- 10 strict pass/fail checks: tier validity, latency budget, token tracking,\n  source capture, terminal status\n- Verdicts: `verified` | `provisionally_verified` | `needs_review` | `failed`\n- Zero LLM involvement — pure deterministic validation\n\n**Layer 2: LLM Semantic Scoring** (`server/pipeline/diligenceLlmJudge.ts`)\n- 5 dimensions scored [0,1]: prose quality, citation coherence, source\n  credibility, tier appropriateness, overall semantic fit\n- Prompt version tracking (`llmjudge-v1`) for cohort separation\n- Bounded: 30s timeout, 512KB response cap, honest error reporting\n\nThis dual-layer approach means hallucinations and quality regressions are\ncaught by **two independent systems** before they reach users.\n\n### Current Production Status\n\n**Latest Full-Stack Eval:** `2026-04-23T06:46:53Z`\n\n```text\nOverall Pass Rate:     100% ✅\nLLM Judge Average:     9.6/10 (target: ≥7 for production)\nDogfood Score:         100/100 (0 real issues)\nEntity Resolution:     100% ✅\nFactual Accuracy:      90.6% ✅\nNo Hallucinations:     90.6% ✅\nActionable Output:     100% ✅\nAnswer Control:        100% ✅ (all 8 dimensions)\nFeature Breadth:       100% ✅ (31 scenarios)\nRetention/Continuity:  4/4 passed ✅\n```\n\nAll production gates **passing**:\n- ✅ Expanded Feature Coverage Production Gate\n- ✅ Answer Control Production Gate  \n- ✅ Dogfood Production Gate\n- ✅ Notebook Capacity Production Gate\n- ✅ History Soak Production Gate\n\n**Note:** The only outstanding item is p95 latency optimization (174s vs 90s\ntarget) — a performance enhancement, not a correctness blocker. The system is\n**production-ready for all quality scenarios**.\n\n### Evaluation Coverage\n\n**Capability Eval — 32 Persona Scenarios**\n\n| Persona | Example Query | Status |\n|---------|---------------|--------|\n| JPM Startup Banker | \"DISCO — worth reaching out? Fastest debrief\" | ✅ 100% |\n| Early Stage VC | \"OpenAutoGLM — what's the wedge?\" | ✅ 100% |\n| CTO Tech Lead | \"QuickJS — do I have exposure?\" | ✅ 100% |\n| Enterprise Exec | \"Gemini 3 — procurement next step?\" | ✅ 100% |\n| Ecosystem Partner | \"SoundCloud VPN — who benefits?\" | ✅ 100% |\n| Founder Strategy | \"Salesforce Agentforce — counter-positioning?\" | ✅ 100% |\n| Academic R&D | \"RyR2/Alzheimer's — literature anchor?\" | ✅ 100% |\n| Quant Analyst | \"DISCO — extract funding signal\" | ✅ 100% |\n| Product Designer | \"DISCO — schema-dense UI card JSON\" | ✅ 100% |\n| Sales Engineer | \"DISCO — share-ready outbound summary\" | ✅ 100% |\n\n**Expanded Feature Breadth — 31 Scenarios**\n\n| Category | Count | Pass Rate |\n|----------|-------|-----------|\n| Calendar | 3 | 100% ✅ |\n| Disclosure | 4 | 100% ✅ |\n| Document | 3 | 100% ✅ |\n| Hybrid | 4 | 100% ✅ |\n| Media | 3 | 100% ✅ |\n| Skills | 4 | 100% ✅ |\n| Spreadsheet | 3 | 100% ✅ |\n| Tools | 4 | 100% ✅ |\n| Web | 3 | 100% ✅ |\n\n**Answer Control — 8 Dimensions**\n\n- Entity resolution: 100% ✅\n- Retrieval relevance: 100% ✅\n- Claim support: 100% ✅\n- Final response quality: 100% ✅\n- Trajectory quality: 100% ✅\n- Actionability: 100% ✅\n- Artifact decision quality: 100% ✅\n- Ambiguity recovery: 100% ✅\n\n### How to Verify\n\nRun the full production evaluation suite:\n\n```bash\n# Full 8-phase evaluation (typecheck → build → capability → expanded →\n# answer-control → dogfood → notebook → history)\nnpm run eval\n\n# Quick verification (3 scenarios)\nnpm run eval:quick-slice\n\n# Individual lanes\nnpm run eval:capability      # 32 persona scenarios\nnpm run eval:feature-breadth # 31 feature scenarios  \nnpm run eval:retention       # Wiki continuity suite\n```\n\nAll artifacts are versioned in `docs/architecture/benchmarks/`:\n- `full-stack-eval-latest.md` — aggregate summary\n- `comprehensive-eval-*.md` — capability results\n- `expanded-eval-*.md` — feature breadth results\n- `product-answer-control-eval-*.md` — answer control results\n\n### What \"Production Ready\" Means Here\n\n1. **Deterministic gates pass** — no regressions in core correctness\n2. **LLM judge scores ≥7** — semantic quality validated by independent LLM\n3. **Dogfood score ≥85** — internal usage shows no real issues\n4. **All 32 persona scenarios pass** — diverse user types handled correctly\n5. **All 31 feature scenarios pass** — broad surface area covered\n6. **Retention/continuity passes** — long-term memory works\n7. **Answer control 100%** — artifact decisions, ambiguity recovery solid\n\nThe system meets all of these. The only remaining work is latency optimization\n— making fast answers even faster, not making broken answers work.\n\n### Model Strategy\n\n- **Primary:** `moonshotai/kimi-k2.6` (OpenRouter) — 100% capability pass\n- **Fallback:** `gpt-5.4` — automatic retry on empty/missing debrief\n- **Judge:** `kimi-k2.6` — 9.6/10 average across all scenarios\n\nKimi is the primary lane. GPT-5.4 remains the safety fallback until Kimi's\nfirst-attempt stability improves, but both paths are production-tested.\n\n## Product Suite\n\n```text\nNodeBench AI   = flagship user surface\nnodebench-mcp  = workflow lane\nAttrition.sh   = measured replay + optimization lane\n```\n\nAttrition is not a third flagship. It is the measurable optimization lane for\nthe same NodeBench workflow.\n\n## License\n\nMIT\n",
  "bytes": 38599,
  "sha": "960a8431b4d56509ec80aa43e4b9ff0c04570b6e886473f010decce056d34cd8",
  "repo_slug": "homenshum/nodebench-ai",
  "fonte": "repo",
  "truncated": false,
  "api": "https://agentalog.com/api/listings/plg_homenshum_nodebench_ai_nodebench_7893c055/readme"
}