{
  "markdown": "# TheSecondBrain\n\n**A personal knowledge vault that compounds over time — powered by LLMs, lived in the terminal.**\n\n> *Inspired by the [LLM Wiki pattern](inspiration/llm-wiki.md) by Andrej Karpathy: instead of re-deriving knowledge on every query, the LLM incrementally builds and maintains a persistent, interlinked wiki — so knowledge accumulates rather than evaporates.*\n\n---\n\n## What Is This?\n\nMost AI document tools work like search: drop files in, ask questions, the LLM retrieves relevant chunks and answers. The knowledge is never kept. Ask the same question tomorrow and it rediscovers from scratch.\n\n**TheSecondBrain is different.** When you add a source, the LLM reads it and *writes wiki pages* — extracting concepts, entities, key learnings, and wiring them together with cross-references. Every question you ask enriches the wiki further. The knowledge compounds.\n\nYou curate sources and ask questions. The LLM writes and maintains everything else.\n\n---\n\n## Project Status\n\n**Status:** Beta\n\nTheSecondBrain is currently tuned for **internal/team use first** and careful external experimentation. It is useful today, but it is **not production-grade software yet**.\n\nCurrent expectations:\n\n- Core wiki path handling is tested and canonicalized around `wiki/...`\n- CI runs build, vet, tests, and formatting checks on PRs\n- Docs are kept aligned with current behavior\n- Release tags produce downloadable binaries for macOS and Linux\n\nKnown limitations:\n\n- **PDF ingestion is not supported yet**\n- Vault repair for previously corrupted `wiki/wiki/...` layouts is warning-only for now\n- This remains a single-user local tool, not a multi-user or hosted product\n\n---\n\n## Architecture\n\n```\n┌─────────────────────────────────────────────────────────────┐\n│                      brain  (TUI)                           │\n│          Go + Bubble Tea  ·  runs in current directory      │\n└───────────────────┬─────────────────────────────────────────┘\n                    │\n        ┌───────────┼───────────┐\n        ▼           ▼           ▼\n   /analyze       Query      /fixwiki\n  (ingest)       (RAG)      (correct)\n        │           │           │\n        ▼           ▼           │\n┌───────────┐  ┌──────────┐    │\n│   wiki/   │  │knowledge │◄───┘\n│           │  │  -base/  │\n│ sources/  │  │          │\n│ entities/ │  │embeddings│\n│ concepts/ │  │amendments│\n│synthesis/ │  │ metadata │\n│ index.md  │  └──────────┘\n│  log.md   │\n└───────────┘\n        ▲\n        │  LLM writes all wiki pages\n        │\n┌───────────┐\n│   raw/    │  ← You drop anything here\n│           │     (docs, images, code,\n│ supported │      notes, repos)\n│   files   │\n└───────────┘\n\nGlobal config: ~/.config/secondbrain/  (API key, model settings)\nProject:       ./  (current directory — like git)\n```\n\n**Three layers:**\n\n| Layer | Who owns it | What lives here |\n|-------|-------------|-----------------|\n| `raw/` | You | Source files — immutable, never modified by the brain |\n| `wiki/` | LLM | Structured knowledge — concepts, entities, summaries, synthesis |\n| `knowledge-base/` | System | Embeddings, metadata, amendment audit trail |\n\n**Two config scopes:**\n\n- **Global** (`~/.config/secondbrain/`) — API key, model settings. Shared across all projects.\n- **Project** (current directory) — `raw/`, `wiki/`, `knowledge-base/`. Like a git repository: `brain` operates in wherever you invoke it.\n\n---\n\n## Top 3 Use Cases\n\n### 1. Deep Research\nReading papers, articles, and reports on a topic over weeks? Drop each one into `raw/` as you go. The brain builds an interlinked wiki of concepts, entities, and findings — cross-referencing everything automatically. By the end you have a structured knowledge base, not a pile of highlights.\n\n**Flow:** Drop paper → `/pull` → wiki pages created for concepts + authors + findings → ask follow-up questions → `/save` compelling syntheses\n\n### 2. Codebase Documentation\nPoint the brain at a cloned repository. It reads the architecture, patterns, modules, and dependencies — and writes a living wiki that stays current as the code evolves. New teammates onboard by querying the brain rather than excavating the code.\n\n**Flow:** Clone repo into `raw/` → `/pull` → wiki/projects/repo-name.md created with architecture, patterns, open questions → ask \"how does the auth flow work?\"\n\n### 3. Team Knowledge Base\nFeed the brain Slack threads, meeting transcripts, design docs, and customer call notes. It maintains a shared wiki that no one has to manually update — because the LLM does the bookkeeping no one wants to do.\n\n**Flow:** Export Slack/Notion → drop into `raw/` → wiki grows with decisions, entities, and concepts → team queries the brain instead of searching Slack\n\n---\n\n## Getting Started\n\n### Install\n\n**One command — no Git, no Go, no technical knowledge needed:**\n\n```bash\ncurl -fsSL https://raw.githubusercontent.com/ORG028658/TheSecondBrain/main/install.sh | bash\n```\n\nThe script:\n- Detects your OS and chip automatically (macOS Intel / Apple Silicon, Linux x86 / ARM)\n- Downloads the pre-built binary from the latest GitHub release\n- Installs it to `~/.local/bin/brain` — no `sudo` required\n- Adds `~/.local/bin` to your shell PATH automatically\n- Tells you exactly what to run when it's done\n\nAfter the script finishes, run the command it prints (usually `source ~/.zshrc && brain`).\n\n**Verify the install worked:**\n\n```bash\nwhich brain   # → /Users/<you>/.local/bin/brain\nbrain\n```\n\n---\n\n### Uninstall\n\n```bash\nbrain --uninstall\n```\n\nRemoves the binary and `~/.config/secondbrain/` (API key + settings).\nYour vault data (`raw/`, `wiki/`, `knowledge-base/`) is never touched.\n\n---\n\n**For developers — build from source:**\n\n```bash\n# Requires Go 1.22+\ngit clone https://github.com/ORG028658/TheSecondBrain\ncd TheSecondBrain\nbash install.sh   # builds from source, installs to /usr/local/bin\n```\n\n### First Run\n\n```bash\nmkdir my-project && cd my-project\nbrain\n```\n\nOn first launch, a setup wizard asks for your **Rakuten AI Gateway key**. It creates:\n- `~/.config/secondbrain/config.yaml` — global settings\n- `~/.config/secondbrain/.env` — API key (never committed)\n- `raw/`, `wiki/`, `knowledge-base/` — in the current directory\n\n### Supported Platforms\n\n- macOS\n- Linux\n- Go version: see [`tui/go.mod`](tui/go.mod)\n\n### Supported Inputs\n\nCurrently supported:\n\n- markdown, text, HTML, and common config/data formats\n- source code and repository trees\n- common image formats (`.jpg`, `.jpeg`, `.png`, `.gif`, `.webp`)\n\nNot supported yet:\n\n- PDF ingestion\n\n### Basic Workflow\n\n```bash\n# 1. Drop files into raw/\ncp ~/Downloads/research-notes.md raw/\ncp -r ~/code/my-android-app raw/\n\n# 2. Process (or files are auto-analyzed within 3 seconds of being dropped)\n/pull\n\n# Alternative: ingest an existing directory without copying files into raw/\n# brain reads from the project root itself — useful for codebases, mono-repos, etc.\nbrain --current-dir      # session-level: all pulls use the project root as source\n/pull --current-dir      # one-shot: single pull from the project root\n\n# 3. Ask questions\nWhat design patterns does my-android-app use?\n\n# 4. Keep useful answers\n/save Android Architecture Overview\n```\n\n---\n\n## Features\n\n### Ingest Pipeline\n- **Many text, code, config, and image types** — markdown, text, code (`.kt`, `.py`, `.go`), HTML, YAML, JSON, and common images\n- **Nested folders** — the entire `raw/` tree is walked, any depth\n- **Multi-page ingest** — one source typically creates 5–15 wiki pages: a source summary, entity pages (people, tools, companies), concept pages, all interlinked\n- **Auto-watch** — file watcher monitors `raw/`; new files trigger analysis automatically after a 3-second debounce\n- **Hash-based change detection** — unchanged files are skipped; only new/modified files are processed\n- **Image analysis** — images are described by vision AI and integrated into relevant wiki pages\n- **Explicitly unsupported right now** — PDFs are skipped instead of partially parsed\n\n### Wiki Structure\n```\nwiki/\n  sources/       ← One summary page per raw source\n  entities/      ← People, organisations, products, tools\n  concepts/      ← Ideas, patterns, theories, techniques\n  synthesis/     ← Filed query results and analyses\n  index.md       ← Auto-maintained master catalog\n  log.md         ← Append-only operation history\n```\n\nEvery wiki page has:\n- **YAML frontmatter** — type, title, tags, sources, created, updated\n- **`[[WikiLink]]` syntax** — internal cross-references between pages\n- **Knowledge extraction, not summarisation** — concepts are explained, not described\n\n### RAG Query Engine\n- **Similarity threshold filtering** — chunks below `min_similarity` (default 0.25) are discarded before reaching the LLM, removing noise\n- **LLM confidence-scored references** — each cited reference includes an AI-calculated confidence % and reason (not just cosine similarity — the LLM scores based on actual usage)\n- **Conversation history** — last 6 turns passed to every query for follow-up awareness\n- **Strict wiki-only answers** — the LLM never uses outside knowledge; if it's not in the wiki it says so\n\n### Wiki Corrections & Amendment Audit Trail\n- **Natural language corrections** — say \"that's wrong, it should be X\" and the brain finds the relevant page, proposes a correction, and asks for confirmation\n- **`/fixwiki <name> <correction>`** — explicit correction by page name or fuzzy match (e.g. `/fixwiki transformer activation should be ReLU`)\n- **Contradiction detection** — before applying any correction, the LLM analyses whether it contradicts the current content (`CONSISTENT` / `CONTRADICTORY`)\n- **Force-apply** — type `force` to override the system's recommendation; your data, your call\n- **Amendment records** — every correction is recorded in `knowledge-base/amendments/YYYYMMDD-HHmmss-slug.md` with: original content, proposed change, system analysis, and outcome\n- **`/amendments`** — list all amendment records with status icons (`✓` applied, `⚡` force-applied)\n\n### Research Gap Tracking\n- **`/gap <topic>`** — flags a missing topic and creates a research stub in `wiki/sources/` with sections for what's needed, suggested sources, and why it matters\n- Varied, conversational \"not in wiki\" responses — not a fixed error message\n- Stubs are indexed so they surface in future searches\n\n### Synthesis Pages\n- **`/save <title>`** — saves the last answer as `wiki/synthesis/slug.md` with proper frontmatter and source citations\n- Filed syntheses are re-indexed and become searchable knowledge\n- All saves logged to `wiki/log.md`\n\n### TUI\n- **Streaming output** — answers appear token by token\n- **Scroll** — `PgUp`/`PgDn` to scroll chat history; a scroll hint appears in the footer when not at the bottom (auto-follows otherwise)\n- **Command history** — `↑`/`↓` arrows to navigate previous inputs (like a shell)\n- **Clipboard** — `Ctrl+Y` copies the last answer\n- **Shell passthrough** — `!<command>` runs any shell command from the project directory (pipes, `&&`, `cd` all work)\n- **File-in-chat** — mention a file path (e.g. `/path/to/doc.md`) and it's automatically copied to `raw/` with an explanation\n- **Sidebar** — press `1`/`2`/`3` to switch between Chat, Commands, and Status panes\n- **Brain logo** with live stats in the header — wiki page count, KB chunk count, watcher indicator\n\n---\n\n## Commands Reference\n\n| Command | Description |\n|---------|-------------|\n| `/pull` | Full pipeline: scan `raw/` → extract knowledge → update `wiki/` → sync embeddings |\n| `/pull --current-dir` | Same as `/pull` but uses the project root as the source directory instead of `raw/` |\n| `/analyze` | Force re-analyze `raw/` (reprocess all files) |\n| `/analyze --current-dir` | Same as `/analyze` but uses the project root instead of `raw/` |\n| `/sync` | Re-embed changed wiki pages (after manual edits) |\n| `/save <title>` | Save last answer as `wiki/synthesis/<slug>.md` |\n| `/fixwiki <name> <fix>` | Correct a wiki page by name or path |\n| `/gap <topic>` | Flag a missing topic — creates a research stub |\n| `/amendments` | List all amendment audit records |\n| `/lint` | Wiki health check — broken links, orphans, stubs, contradictions |\n| `/status` | Show project dir, raw file count, wiki pages, KB chunks, API key status |\n| `/config` | Show global config (dir, model, embeddings, paths) |\n| `/config key` | Show masked API key |\n| `/config reset` | Remove global config — next launch triggers setup |\n| `/logout` | Same as `/config reset` — removes `~/.config/secondbrain/` |\n| `/tips` | Show quick-start guide |\n| `/help` | Show all commands |\n| `!<command>` | Run a shell command from the project directory |\n\n---\n\n## Keyboard Shortcuts\n\n| Key | Action |\n|-----|--------|\n| `↑` / `↓` | Navigate command history |\n| `PgUp` / `PgDn` | Scroll chat (stops auto-follow when scrolled up) |\n| `Ctrl+Y` | Copy last answer to clipboard |\n| `Ctrl+C` | Quit |\n| `Esc` | Cancel current operation (query, pull, analyze) or wiki confirmation |\n| `1` / `2` / `3` | Switch sidebar pane: Chat / Commands / Status (when input is empty) |\n| `confirm` | (in confirmation prompts) Apply a wiki correction |\n| `force` | (in confirmation prompts) Force-apply despite contradictions |\n\n---\n\n## Releases\n\nInstall the latest release:\n\n```bash\ngo install github.com/ORG028658/TheSecondBrain/tui@latest\n```\n\nInstall a specific version:\n\n```bash\ngo install github.com/ORG028658/TheSecondBrain/tui@v0.1.0\n```\n\nTagged releases also publish prebuilt archives for macOS and Linux. See [RELEASE.md](RELEASE.md) for the release process and backup guidance.\n\n---\n\n## Configuration\n\n**Global** (`~/.config/secondbrain/config.yaml`):\n```yaml\nllm:\n  model: \"gpt-4o\"\n  max_tokens: 4096\n  base_url: \"https://api.ai.public.rakuten-it.com/openai/v1\"\n\nembeddings:\n  model: \"text-embedding-3-small\"\n  base_url: \"https://api.ai.public.rakuten-it.com/openai/v1\"\n\nrag:\n  chunk_size: 1500      # characters per chunk\n  top_k: 5              # max chunks retrieved per query\n  min_similarity: 0.25  # discard chunks below this cosine score (raise for stricter relevance)\n```\n\n**Secrets** (`~/.config/secondbrain/.env`):\n```\nLLM_COMPATIBLE_API_KEY=your_api_key_here\n```\n\n### Compatible API Providers\n\nTheSecondBrain works with **any OpenAI-compatible API endpoint**. Change `base_url` and `model` in `config.yaml`, then set `LLM_COMPATIBLE_API_KEY` to that provider's key:\n\n| Provider | `base_url` | Example model |\n|----------|-----------|---------------|\n| [OpenAI](https://platform.openai.com) | `https://api.openai.com/v1` | `gpt-4o` |\n| [Groq](https://groq.com) | `https://api.groq.com/openai/v1` | `llama-3.3-70b-versatile` |\n| [Azure OpenAI](https://azure.microsoft.com/products/ai-services/openai-service) | `https://<resource>.openai.azure.com/openai` | your deployment name |\n| [Ollama](https://ollama.com) (local, free) | `http://localhost:11434/v1` | `llama3.2` |\n| [Together AI](https://together.ai) | `https://api.together.xyz/v1` | `meta-llama/Llama-3-70b` |\n| Rakuten AI Gateway | `https://api.ai.public.rakuten-it.com/openai/v1` | `gpt-4o` |\n\n> **Anthropic / Claude:** Anthropic's native API is not OpenAI-compatible. Access Claude models through [AWS Bedrock](https://aws.amazon.com/bedrock/) or [Google Vertex AI](https://cloud.google.com/vertex-ai), both of which expose OpenAI-compatible endpoints.\n\n---\n\n## Directory Layout\n\n```\n<your-project>/\n├── raw/                          ← Drop any files here (immutable)\n├── wiki/\n│   ├── sources/                  ← One summary per raw source\n│   ├── entities/                 ← People, orgs, products, tools\n│   ├── concepts/                 ← Ideas, patterns, theories\n│   ├── synthesis/                ← Saved query results\n│   ├── index.md                  ← Auto-maintained catalog\n│   └── log.md                    ← Operation history\n└── knowledge-base/\n    ├── embeddings/store.json     ← Vector store (flat JSON, cosine search)\n    ├── metadata/sources.json     ← Content hashes for change detection\n    ├── amendments/               ← Correction audit trail\n    └── output/                   ← Reports and exports\n\n~/.config/secondbrain/\n├── config.yaml                   ← Global model + RAG settings\n└── .env                          ← API key (600 permissions, never committed)\n```\n\n---\n\n## Troubleshooting\n\n- `brain: command not found`\n  Add `$(go env GOPATH)/bin` or `$GOBIN` to your `PATH`.\n- `401` or unauthorized errors\n  Check `~/.config/secondbrain/.env` and confirm `LLM_COMPATIBLE_API_KEY` is valid for the configured provider.\n- Warning about nested `wiki/wiki/...`\n  This usually means the vault was touched by an older path bug. New writes are normalized, but you should inspect and migrate nested markdown files before trusting mixed results.\n- PDF files do nothing\n  PDF ingestion is not implemented yet in this beta.\n\n## Backup and Recovery\n\nBack up these paths regularly:\n\n- your project `wiki/`\n- your project `knowledge-base/`\n- your project `raw/` if the original source material is not stored elsewhere\n- `~/.config/secondbrain/` if you want to preserve config and API settings\n\nTreat `wiki/` and `knowledge-base/amendments/` as source-of-truth user data. `knowledge-base/embeddings/store.json` can usually be rebuilt with `/pull` or `/sync`.\n\n---\n\n## How It Works — The Pipeline\n\n```\nDrop file into raw/\n        │\n        ▼  (auto after 3s, or /pull)\n[Analyzer — LLM]\n  · Reads file content (text, code, or vision for images)\n  · Checks wiki/index.md for existing related pages\n  · Returns JSON: array of pages to create/update + log entry\n        │\n        ▼\nwiki/ pages written\n  wiki/sources/slug.md        ← source summary\n  wiki/entities/name.md       ← per entity mentioned\n  wiki/concepts/name.md       ← per concept covered\n  (typically 5–15 pages per source)\n        │\n        ▼  (/sync or auto after /pull)\n[Embeddings — Rakuten AI Gateway]\n  · Chunks each wiki page by paragraph\n  · Embeds via text-embedding-3-small\n  · Stores vectors in knowledge-base/embeddings/store.json\n        │\n        ▼  (on question)\n[RAG Query]\n  · Embeds question\n  · Cosine search → top-K chunks filtered by min_similarity\n  · LLM answers from wiki context only\n  · LLM scores each reference by actual contribution\n  · Conversation history passed for follow-up awareness\n```\n\n---\n\n## The LLM Schema\n\n`CLAUDE.md` at the project root is the brain's operating manual. It defines:\n- **10 canonical rules** — never touch `raw/`, always use wikilinks, always update the index, etc.\n- **Page templates** — for sources, entities, concepts, and synthesis pages\n- **Four operations** — Ingest, Query, Lint, and Onboarding with step-by-step workflows\n- **Knowledge extraction standard** — the LLM extracts *how things work*, not just *what things are*\n\nThis file is automatically loaded into every Claude Code session that opens the project directory.\n\n---\n\n## Dependencies\n\n| Component | Library |\n|-----------|---------|\n| TUI | `charmbracelet/bubbletea` + `bubbles` + `lipgloss` |\n| LLM (wiki + RAG answers) | OpenAI-compatible API (Rakuten AI Gateway → `gpt-4o`) |\n| Embeddings | Rakuten AI Gateway → `text-embedding-3-small` |\n| Vector store | Flat JSON with cosine similarity (zero dependencies) |\n| File watching | `fsnotify/fsnotify` |\n| Config | `gopkg.in/yaml.v3` + `joho/godotenv` |\n| Clipboard | `atotto/clipboard` |\n\n---\n\n## Project Docs\n\n- [CONTRIBUTING.md](CONTRIBUTING.md)\n- [SECURITY.md](SECURITY.md)\n- [CODE_OF_CONDUCT.md](CODE_OF_CONDUCT.md)\n- [ROADMAP.md](ROADMAP.md)\n- [RELEASE.md](RELEASE.md)\n- [PRIVACY.md](PRIVACY.md)\n",
  "bytes": 19465,
  "sha": "9c70eb6fbcfc03f3937a45b3d9f23ea4208ada36a1be867475aaba724a745759",
  "repo_slug": "org028658/thesecondbrain",
  "fonte": "repo",
  "truncated": false,
  "api": "https://agentalog.com/api/listings/plg_org028658_thesecondbrain_second_brain_281988df/readme"
}