Thresher Knowledge Bundle
Bundle OKF 0.1 · 10 conceitos · ecurtin2/gmailjanitor
Open source Repository Open in the app JSON README (API)
About
# Thresher Knowledge Bundle
Explainable active learning for text — wheat from the chaff. This bundle documents
why Thresher is built the way it is, how the crates fit together, and how we
record experiments as we evaluate performance on classification tasks.
# Design
* [System design](design/system-design.md) - Goals, why-this-way rationale, and citations
* [UI design system](design/ui-design-system.md) - Tokens, components, and Insights chart patterns
# Architecture
* [Architecture overview](architecture/overview.md) - Crate layers, module map, local-first runtime
* [Data flow](architecture/data-flow.md) - Ingest through Verdict and Explanation
* [Training pipeline](architecture/training-pipeline.md) - Feedback → LR → CORELS → snapshot
* [Ports and adapters](architecture/ports-and-adapters.md) - Who defines each port and who implements it
# Decisions
* [Decision log](decisions/) - Architecture-level decision records
# Experiments
* [Experiment log](experiments/) - Runs, studi
Details
- Kind
- OKF bundles
- Topic
- Maps, weather & travel
- Publisher
- ecurtin2
- Origin
- okf_github
- Category
- dados
- Version
- 0.1
- Last push
- 2026-07-19T15:09:52Z
- Repository state
- ativo
- Language
- Rust
- License
- Apache-2.0
- Added
- 2026-09-08 22:11:44
- Updated
- 2026-09-08 22:11:44
- Origin id
ecurtin2/gmailjanitor:docs/knowledge/index.md
README
# Thresher
**Explainable active learning for text.** Separate the wheat from the chaff.
Thresher is a Rust library and desktop app for classifying text documents with
an explainable rule-list model. You label a few examples; Thresher learns a short,
human-readable rule list and uses it to triage the rest — fast enough for
microsecond inference, with every verdict carrying an explanation.
**First use case:** Gmail inbox cleanup (spam vs keep). The core is source-agnostic —
future adapters (RSS, tickets, files) plug in via hexagonal ports.
## Architecture
```
crates/
corels/ Standalone CORELS-style rule-list learner
thresher-core/ Domain types + ports (no I/O)
thresher-ml/ Sparse logistic regression sub-models
thresher-features/ Document → binary meta-features
thresher-store/ SQLite persistence
thresher-gmail/ Gmail REST adapter (OAuth2)
thresher-app/ Application services
thresher-ui/ iced desktop UI
```
### How decisions are made
The CORELS rule list is the **only** decision-maker. It never sees raw text — only
named binary meta-features:
| Feature family | Examples |
| --- | --- |
| Lists | `allowlist:sender`, `blocklist:sender`, `blocklist_domain:sender` |
| Regex | `regex:unsubscribe` (user-defined over any field) |
| Source flags | `flag:has_list_unsubscribe_header`, `flag:is_bulk_precedence` |
| Derived | `all_caps:subject`, `link_heavy:body_snippet`, length buckets |
| Sub-models | `lr_score:>0.5`, `lr_score:>0.8`, `lr_score:>0.95` |
Sparse logistic regression (and future sub-models) feed **bucketized** scores into
that explainable layer; they do not decide alone.
### Active learning loop
1. **Ingest** documents from a source (Gmail).
2. **Review** an uncertainty-ordered queue (near decision boundary / model disagreement first).
3. **Train** — fit LR → binarize features → run CORELS → persist snapshot.
4. **Triage** — inspect verdicts + explanations; apply labels (v1: labels only).
## Credits
The `corels` crate is a **clean-room** Rust implementation inspired by the
CORELS (Certifiably Optimal RulE ListS) algorithm and papers. We are grateful to
the CORELS authors for open research and documentation:
- Elaine Angelino, Nicholas Larus-Stone, Daniel Alabi, Margo Seltzer, and Cynthia Rudin.
**Learning Certifiably Optimal Rule Lists for Categorical Data**. JMLR, 2018.
- Nicholas Larus-Stone, Elaine Angelino, Daniel Alabi, Margo Seltzer, Vassilios Kaxiras,
Aditya Saligrama, Cynthia Rudin. **Systems Optimizations for Learning Certifiably
Optimal Rule Lists**. SysML, 2018.
- Elaine Angelino, Nicholas Larus-Stone, Daniel Alabi, Margo Seltzer, Cynthia Rudin.
**Learning certifiably optimal rule lists for categorical data**. KDD, 2017.
Reference project: [github.com/corels/corels](https://github.com/corels/corels)
(GPL-3.0). Thresher’s learner was written from the published algorithms, not
ported from that source, and is licensed MIT OR Apache-2.0.
## Quick start
```bash
# Or use the justfile:
just build
just test
just clippy
# Offline demo (no Gmail): seed synthetic mail + launch UI
just demo
# Or step by step:
just seed-fresh # 80 synthetic emails, 25 pre-labeled
just run # iced UI
```
Inside the UI: label with `s`/`h`, undo with `u`, then **Train** and open **Triage**.
### Gmail setup
1. Create a Google Cloud project and enable the **Gmail API**.
2. Create OAuth 2.0 credentials (Desktop / installed app).
3. Either:
```bash
export THRESHER_GMAIL_CLIENT_ID="….apps.googleusercontent.com"
export THRESHER_GMAIL_CLIENT_SECRET="…"
```
or place `~/.config/thresher/gmail_credentials.json`:
```json
{
"client_id": "….apps.googleusercontent.com",
"client_secret": "…"
}
```
4. In the UI: **Gmail** → authorize in the browser → **Ingest** → label with
`s` (spam) / `h` (keep) → **Train** → **Triage** → **Apply spam labels**.
Tokens are cached at `~/.config/thresher/gmail_token.json` (mode `600` on Unix).
Local data lives under `~/.local/share/thresher/thresher.db` (or the platform
equivalent from `dirs::data_dir`).
### Keyboard shortcuts (Review tab)
| Key | Action |
| --- | --- |
| `j` / `k` | Next / previous document |
| `s` | Label as spam (positive) |
| `h` | Label as keep (negative) |
### Actions (v1)
v1 actions are **labels only** (e.g. apply `Thresher/Spam`). No automatic delete
or mark-as-read — those variants exist in the domain model but are gated off.
## Experiments
The `thresher-bench` crate simulates an active-learning user over standard
datasets (SMS spam, Enron-Spam, RT polarity), recording labeling-efficiency and
held-out learning curves plus train/classify timings (OpenTelemetry meters in
`thresher-app`).
```bash
# Download datasets (cached) and run baseline-style sims
just bench # all three datasets
just bench dataset=sms # one dataset
# Browse a dataset in the iced UI
just seed-dataset sms
just run
```
Writeups live under [`docs/knowledge/experiments/`](docs/knowledge/experiments/).
See [`AGENTS.md`](AGENTS.md) for how to log new runs.
## Adding a new source
1. Map your items into `thresher_core::Document` (`fields` + `flags` + `source_ref`).
2. Implement `DocumentSource` and `ActionSink`.
3. Wire the adapter into the UI / app the same way as `thresher-gmail`.
No changes to the learner or feature pipeline are required unless you want
source-specific defaults.
## License
Licensed under either of:
- Apache License, Version 2.0 ([LICENSE-APACHE](LICENSE-APACHE))
- MIT license ([LICENSE-MIT](LICENSE-MIT))
at your option.