Sources
Bundle OKF 0.1 · 2 conceitos · tesfandiari1/knowledge-base
Open source Repository Open in the app JSON README (API)
About
# Sources
Verbatim scraped docs live in `../raw/`. Browse the per-site manifests:
* [tauri-v2](../raw/tauri-v2/CONTENTS.md) - Tauri 2.0 docs (79 pages) from v2.tauri.app.
* [awesome-tauri](../raw/awesome-tauri/CONTENTS.md) - curated ecosystem list.
# Concepts
_Synthesized on demand only (see ../AGENTS.md → Wiki mode). Empty by default._
Details
- Kind
- OKF bundles
- Topic
- Productivity
- Publisher
- tesfandiari1
- Origin
- okf_github
- Category
- dados
- Version
- 0.1
- Last push
- 2026-06-24T20:57:49Z
- Repository state
- ativo
- Language
- Python
- Added
- 2026-09-08 02:26:02
- Updated
- 2026-09-08 02:26:02
- Origin id
tesfandiari1/knowledge-base:topics/tauri/wiki/index.md
README
# Knowledge Base A general-purpose **documentation dump ground**. Point firecrawl at a site, scrape its docs into clean frontmattered Markdown, and reference them from any project. Each subject is a **topic** — a self-contained [OKF](okf-pack/) space. ``` . ├── index.md # catalog of all topics ├── okf-pack/ # shared reference layer: format rules, operating spec, authoring craft │ ├── okf-rulebook.md # OKF v0.1 format spec (single source of truth) │ ├── okf-space.md # setup & operating spec │ ├── concept-authoring.md # on-demand concept-writing craft │ └── topic-AGENTS.template.md ├── scripts/ # firecrawl ingestion pipeline (parameterized by topic) ├── topics/ │ └── <topic>/ │ ├── AGENTS.md # this topic's schema layer (read every session) │ ├── raw/ # IMMUTABLE scraped docs — read, never edit │ │ ├── SOURCES.md # scrape config (URLs + firecrawl flags) │ │ └── <site>/CONTENTS.md # mechanical manifest per scraped site │ └── wiki/ # light OKF bundle: index.md + log.md (+ on-demand concepts) └── .firecrawl/ # raw scrape cache + map JSON (gitignored) ``` ## The two layers (per topic) | Layer | Path | Owner | Mutability | |-------|------|-------|-----------| | **Raw sources** | `raw/` | firecrawl + you | Immutable. Regenerable from `raw/SOURCES.md`. | | **The wiki** | `wiki/` | the LLM | Derived. **Light by default** — a catalog over `raw/`, not a rewrite of it. | The split is the immutability boundary: `raw/` is ground truth; `wiki/` is derived and regenerable. See [`okf-pack/okf-space.md`](okf-pack/okf-space.md) for the full model. ### Wiki mode is LIGHT Big scrapes (hundreds of pages) should **not** be re-read and rewritten into concept pages — that burns tokens for little gain when the raw Markdown is already clean. The default ingest is mechanical and ~free: scrape → regenerate `CONTENTS.md` manifests → refresh `wiki/index.md` → log it. Full OKF synthesis (overviews, comparisons, classification) is **opt-in, on a named subset, only when asked**. Each topic's `AGENTS.md` spells this out. ## Add a new topic ```bash # 1. Scaffold (creates raw/, wiki/ seeded, AGENTS.md from the template) scripts/new_topic.sh blender # 2. Declare sources: edit topics/blender/raw/SOURCES.md (the ```sources block) # 3. Scrape — map → scrape → transform → index, all sites in SOURCES.md scripts/scrape_topic.sh blender # First run per site writes raw/<site>.urls.txt and stops so you can hand-edit # it (drop translations, blog, auto-gen API dumps); re-run to continue. # 4. (zero-token) Refresh wiki/index.md to catalog the new raw sources; log it. # 5. (optional, costs tokens) Ask the agent to synthesize overview/concept pages. # 6. Reference topics/blender/raw/ (verbatim) or wiki/ (curated) in any project. ``` ## Pipeline scripts (`scripts/`) | Script | Purpose | |--------|---------| | `new_topic.sh <slug>` | Scaffold a topic folder from the template. | | `scrape_topic.sh <slug>` | Run the full pipeline for every site in `<slug>/raw/SOURCES.md`. | | `scrape_urls.sh <urls> <raw-dir>` | Scrape a URL list to raw JSON (concurrency 5, resumable; `MAX_PARALLEL` to override). | | `firecrawl_to_md.py` | Transform raw Firecrawl JSON → Markdown with frontmatter. | | `gen_index.py <site-dir> "<Title>"` | Generate a site's `CONTENTS.md` manifest. | ### `firecrawl_to_md.py` per-site flags (set via `extra_flags` in `SOURCES.md`) | Flag | Purpose | |------|---------| | `--base-tags a,b,c` | Tags prepended to every doc (path segments appended automatically). | | `--strip-title-suffix REGEX` | Remove a trailing title suffix, e.g. `"\s*\|\s*Tauri\s*$"`. | | `--strip-path-prefix /p/q` | Trim a URL path prefix before mirroring (docs.rs, GitHub, …). | | `--cut-before REGEX` | Drop leading site chrome before the first matching line (docs.rs). | | `--skip-404` | Skip pages whose title/content indicate a 404 (dead sitemap URLs). | The cleaner also strips, for all sites: `Skip to content` / `Section titled` jump links, `Edit in GitHub` / `Copy as Markdown` rows, docs.rs `[Source]`/`[§]` nav links, self-linked headings, and rustdoc `Available on crate feature` preambles. ## Scraped-doc frontmatter schema Every file under `raw/<site>/` carries YAML frontmatter so it can be filtered, indexed, and traced to source: ```yaml --- title: "SQL" description: "Plugin providing an interface…" source_url: "https://v2.tauri.app/plugin/sql" site: "tauri-v2" site_host: "v2.tauri.app" section: "plugin" tags: ["tauri", "tauri-v2", "plugin", "sql"] language: "en" scraped_date: 2026-06-15 --- ``` `raw/` docs keep this firecrawl frontmatter as-is — they are **sources**, so OKF's `type:`-required conformance (rulebook §7) applies only inside `wiki/`. ## Notes - `scrape_urls.sh` and `scrape_topic.sh` skip already-downloaded URLs — safe to re-run. - Firecrawl's own `download`/`crawl` produce plain Markdown with **no** frontmatter; that's why we scrape to JSON and transform separately. - For large auto-generated API references (docs.rs crates, framework dumps), curate a `*.urls.txt` subset — full crates are thousands of pages and burn credits.