{
  "markdown": "# Knowledge Base\n\nA general-purpose **documentation dump ground**. Point firecrawl at a site, scrape\nits docs into clean frontmattered Markdown, and reference them from any project.\nEach subject is a **topic** — a self-contained [OKF](okf-pack/) space.\n\n```\n.\n├── index.md            # catalog of all topics\n├── okf-pack/           # shared reference layer: format rules, operating spec, authoring craft\n│   ├── okf-rulebook.md          # OKF v0.1 format spec (single source of truth)\n│   ├── okf-space.md             # setup & operating spec\n│   ├── concept-authoring.md     # on-demand concept-writing craft\n│   └── topic-AGENTS.template.md\n├── scripts/            # firecrawl ingestion pipeline (parameterized by topic)\n├── topics/\n│   └── <topic>/\n│       ├── AGENTS.md   # this topic's schema layer (read every session)\n│       ├── raw/        # IMMUTABLE scraped docs — read, never edit\n│       │   ├── SOURCES.md          # scrape config (URLs + firecrawl flags)\n│       │   └── <site>/CONTENTS.md  # mechanical manifest per scraped site\n│       └── wiki/       # light OKF bundle: index.md + log.md (+ on-demand concepts)\n└── .firecrawl/         # raw scrape cache + map JSON (gitignored)\n```\n\n## The two layers (per topic)\n\n| Layer | Path | Owner | Mutability |\n|-------|------|-------|-----------|\n| **Raw sources** | `raw/` | firecrawl + you | Immutable. Regenerable from `raw/SOURCES.md`. |\n| **The wiki** | `wiki/` | the LLM | Derived. **Light by default** — a catalog over `raw/`, not a rewrite of it. |\n\nThe split is the immutability boundary: `raw/` is ground truth; `wiki/` is derived\nand regenerable. See [`okf-pack/okf-space.md`](okf-pack/okf-space.md) for the full model.\n\n### Wiki mode is LIGHT\n\nBig scrapes (hundreds of pages) should **not** be re-read and rewritten into\nconcept pages — that burns tokens for little gain when the raw Markdown is already\nclean. The default ingest is mechanical and ~free: scrape → regenerate `CONTENTS.md`\nmanifests → refresh `wiki/index.md` → log it. Full OKF synthesis (overviews,\ncomparisons, classification) is **opt-in, on a named subset, only when asked**.\nEach topic's `AGENTS.md` spells this out.\n\n## Add a new topic\n\n```bash\n# 1. Scaffold (creates raw/, wiki/ seeded, AGENTS.md from the template)\nscripts/new_topic.sh blender\n\n# 2. Declare sources: edit topics/blender/raw/SOURCES.md (the ```sources block)\n\n# 3. Scrape — map → scrape → transform → index, all sites in SOURCES.md\nscripts/scrape_topic.sh blender\n#    First run per site writes raw/<site>.urls.txt and stops so you can hand-edit\n#    it (drop translations, blog, auto-gen API dumps); re-run to continue.\n\n# 4. (zero-token) Refresh wiki/index.md to catalog the new raw sources; log it.\n# 5. (optional, costs tokens) Ask the agent to synthesize overview/concept pages.\n\n# 6. Reference topics/blender/raw/ (verbatim) or wiki/ (curated) in any project.\n```\n\n## Pipeline scripts (`scripts/`)\n\n| Script | Purpose |\n|--------|---------|\n| `new_topic.sh <slug>` | Scaffold a topic folder from the template. |\n| `scrape_topic.sh <slug>` | Run the full pipeline for every site in `<slug>/raw/SOURCES.md`. |\n| `scrape_urls.sh <urls> <raw-dir>` | Scrape a URL list to raw JSON (concurrency 5, resumable; `MAX_PARALLEL` to override). |\n| `firecrawl_to_md.py` | Transform raw Firecrawl JSON → Markdown with frontmatter. |\n| `gen_index.py <site-dir> \"<Title>\"` | Generate a site's `CONTENTS.md` manifest. |\n\n### `firecrawl_to_md.py` per-site flags (set via `extra_flags` in `SOURCES.md`)\n\n| Flag | Purpose |\n|------|---------|\n| `--base-tags a,b,c` | Tags prepended to every doc (path segments appended automatically). |\n| `--strip-title-suffix REGEX` | Remove a trailing title suffix, e.g. `\"\\s*\\|\\s*Tauri\\s*$\"`. |\n| `--strip-path-prefix /p/q` | Trim a URL path prefix before mirroring (docs.rs, GitHub, …). |\n| `--cut-before REGEX` | Drop leading site chrome before the first matching line (docs.rs). |\n| `--skip-404` | Skip pages whose title/content indicate a 404 (dead sitemap URLs). |\n\nThe cleaner also strips, for all sites: `Skip to content` / `Section titled` jump\nlinks, `Edit in GitHub` / `Copy as Markdown` rows, docs.rs `[Source]`/`[§]` nav\nlinks, self-linked headings, and rustdoc `Available on crate feature` preambles.\n\n## Scraped-doc frontmatter schema\n\nEvery file under `raw/<site>/` carries YAML frontmatter so it can be filtered,\nindexed, and traced to source:\n\n```yaml\n---\ntitle:        \"SQL\"\ndescription:  \"Plugin providing an interface…\"\nsource_url:   \"https://v2.tauri.app/plugin/sql\"\nsite:         \"tauri-v2\"\nsite_host:    \"v2.tauri.app\"\nsection:      \"plugin\"\ntags:         [\"tauri\", \"tauri-v2\", \"plugin\", \"sql\"]\nlanguage:     \"en\"\nscraped_date: 2026-06-15\n---\n```\n\n`raw/` docs keep this firecrawl frontmatter as-is — they are **sources**, so OKF's\n`type:`-required conformance (rulebook §7) applies only inside `wiki/`.\n\n## Notes\n\n- `scrape_urls.sh` and `scrape_topic.sh` skip already-downloaded URLs — safe to re-run.\n- Firecrawl's own `download`/`crawl` produce plain Markdown with **no** frontmatter;\n  that's why we scrape to JSON and transform separately.\n- For large auto-generated API references (docs.rs crates, framework dumps), curate\n  a `*.urls.txt` subset — full crates are thousands of pages and burn credits.\n",
  "bytes": 5266,
  "sha": "04a95d4edf9e561965ab3a3d7d71beb0b3024b177377adfee5bde1d0994331c4",
  "repo_slug": "tesfandiari1/knowledge-base",
  "fonte": "repo",
  "truncated": false,
  "api": "https://agentalog.com/api/listings/okf_tesfandiari1_knowledge_base_topics_premi_6f51f970/readme"
}