{
  "markdown": "<p align=\"center\"><img width=\"381\" height=\"256\" alt=\"ChatGPT Image Jul 24, 2026, 10_32_36 PM\" src=\"https://github.com/user-attachments/assets/a2c7e4d3-aa5e-4b17-b305-0b6676f3304d\" /></p>\n\n# SemanticDF (SDF)\n\nA **semantic layer for Apache Spark** (JVM/Scala), inspired by\nthe [Boring Semantic Layer](https://github.com/boringdata/boring-semantic-layer) (Python/Ibis).\n\nA `SemanticTable` is a deferred, source-agnostic definition that compiles to a Spark\n`DataFrame` at a batch terminal (`.toDataFrame(spark)` / `.execute(spark)`) or a\n`StreamingQuery` at the streaming terminal (`.toStreamingQuery(spark, opts)`). It is *not*\na `DataFrame` itself — it captures *what* you want (dimensions, measures, joins, filters,\ngrains) so the engine can decide *how* to compute it. The same definition serves both\nbatch and streaming sources; only the terminal differs.\n\n## What problems SemanticDF solves\n\nModern data teams have a **plumbing problem**: every dashboard, notebook, or AI agent\nneeds the same business metrics defined somewhere — usually duplicated across queries,\nspreadsheets, and tribal lore. When the metric changes (`\"total_passengers\"` now means\ndeplanements not boarded-then-deplaned), every copy drifts. SemanticDF puts the metric\ndefinitions in **one checked-in source** — a small Scala DSL or a YAML model — and\ngives every consumer (your code, your tests, your LLM agent) the same compile-time\nguarantee that they're asking for the right thing.\n\n## What you can do with it\n\n- **Define a metric once, query it everywhere.** A `SemanticTable` is an immutable\n  description. Use it from `flights.query(...)` in code, from a YAML model in\n  `models/flights.yml`, from a dbt `manifest.json`, from an Apache Ossie\n  YAML, or from an MCP agent that calls `query` / `describe_model` over JSON.\n- **Calc + percent-of-total measures with no expression-tree surgery.** A measure that\n  references other measures (`t.total / t.flight_count`) resolves by name against the\n  aggregated DataFrame; a percent-of-total (`t.total / t.all(t.total)`) cross-joins a\n  broadcasted totals row.\n- **Compile-time typo safety on the query side.** The optional typeclass layer\n  (`SemanticField[T]` phantom types) catches dimension-vs-measure confusion at the\n  call site rather than at first execution. `ResultDecoder.derive[T]` does the same\n  for the *result* side of a query.\n- **One model across batch and streaming.** The op tree is source-agnostic;\n  only the execution terminal differs (`.toDataFrame(...)` for batch,\n  `.toStreamingQuery(...)` for Structured Streaming).\n- **A models → agents bridge.** `okfgen` produces OKF markdown an LLM can read;\n  the MCP server exposes the tools (`list_models`, `describe_model`, `query`,\n  `explain`, `introspect`, `audit_log`) over stdio or REST.\n- **Result cache for repeated LLM-agent queries.** Opt in with\n  `.withResultCache(ResultCache.inMemory(256))`; the second identical\n  `query()` returns from cache without re-executing the Spark plan.\n  Cache keys are stable SHA-256s of the request shape\n  (model + measures + dimensions + where + having + orderBy + limit),\n  so semantically-equivalent queries share a cache entry. Drop all\n  entries for a model in one call via `cache.invalidateModel(\"orders\")`,\n  or set a per-model `version: Int` and let the cache auto-evict stale\n  entries on the next read after the model rebuilds.\n- **Per-query audit log for LLM-agent observability.** Opt in with\n  `.withAuditSink(AuditSink.inMemory())`; every `query()` emits an\n  `AuditEvent` recording the model, request shape, elapsed time, row\n  count, and status. The MCP `audit_log` tool exposes the recent\n  event stream back to the agent for self-introspection\n  (\"what did I just query?\" / \"did my last query timeout?\").\n- **A typeclass for interchange formats.** `SemanticMetadataAdapter[Source, P]`\n  is the unified entry point. Today there are three instances:\n  `DbtAdapter` (dbt `manifest.json`), `OssieReader` (Apache Ossie YAML),\n  and `SDFAdapter` (the cross-process `SemanticManifest` JSON).\n  Future formats plug in as a new `object` and inherit the\n  `loadSemanticTables(source, resolve)` entry point.\n- **Cluster-mode safe.** A `SemanticTable` is Java-serializable, so capturing it in a closure\n  (UDF, broadcast variable, `spark-submit --master yarn|k8s`) round-trips through\n  Spark's deploy mode without `NotSerializableException`. The internal op tree,\n  dimension/measure lambdas, and cache key derivation all cross the JVM boundary safely.\n- **Run as a long-running service.** The `semanticdf-platform` module is a standalone\n   Restate-native runtime that exposes the library over HTTP. Models, queries, and\n   streaming queries get durable state, a replayable audit log, and crash recovery — no\n   glue code required on your side. See [The platform](#the-platform) below.\n\n\n## When (and when not) to use it\n\n- **Good fit:** small-to-mid data teams with a stable set of business metrics,\n  already on Spark 3.5+ (or 4.x), who want one definition everyone shares — including\n  LLM agents.\n- **Not yet:** stream-stream joins (only static-stream joins are supported today —\n  `join_one(batchTable, streamingModel, ...)`); heavy-numeric ML workloads without\n  rollup needs; sub-second interactive dashboards where another tool's tighter\n  latency matters more than metric consistency.\n\n## Where to read next\n\n- **[`docs/getting-started.md`](docs/getting-started.md) — 5-minute paste-and-run setup (Maven + SparkSession + first query)**\n- [`docs/guide.md`](docs/guide.md) — narrative walkthrough: how SemanticDF works, in plain English\n- **[`semanticdf-platform/README.md`](semanticdf-platform/README.md)** — the standalone Restate-native platform runtime (long-running JVM with a Restate ingress, post-crash query reconciliation, bulk-startup recovery). Ships as a separate Maven module that depends on the library.\n- [`DESIGN.md`](DESIGN.md) — architecture of record (decisions, the hard problems)\n- **[`docs/design/multi-engine-design.md`](docs/design/multi-engine-design.md)** — the engine-portable design: `Engine[R]` contract, portable IR (`RelOp`), portable result types, capability surfaces, CAS publication contract. The reference for engine-adapter authors.\n- **[`docs/design/v0.3.1-feature-parity-backlog.md`](docs/design/v0.3.1-feature-parity-backlog.md)** — the 7 gaps between the v0.3.0 portable design and full feature parity with the legacy Spark library (Spark-on-legacy path, `t.all`, joins, predicate unification, rollup compile, catalog adapter). Prioritized roadmap.\n- [`docs/DOCS_MAP.md`](docs/DOCS_MAP.md) — wayfinding guide: which doc to read for which question\n- [`docs/GLOSSARY.md`](docs/GLOSSARY.md) — terms-of-art (op tree, BaseScope, MeasureScope, expression-tree surgery, …)\n- [`docs/adr/`](docs/adr/) — recorded decisions\n- [`RELEASE.md`](RELEASE.md) — version-by-version changelog\n- [`docs/known-limitations.md`](docs/known-limitations.md) — current scope & guardrails (what's in, what's deferred, with workarounds)\n- [`examples/`](examples/) — runnable end-to-end examples\n\n## Engine-portable core (in progress)\n\nSemanticDF is evolving from a **Spark-only semantic layer** to an\n**engine-agnostic semantic data platform**. The portable core lives\nunder `io.semanticdf.core.*` (in `semanticdf-core/`); engine adapters\nimplement `Engine[R]` and live in `adapters/semanticdf-*/`.\n\n| Module | Role |\n|---|---|\n| `semanticdf-core` | Portable ADTs: `Model`, `Dimension`, `Measure`, `FilterSpec`, `JoinSpec`, `RelOp`, `Engine[R]`, `ExecutionPlan`, `ResultValue`, `CatalogAdapter`, etc. **Zero Spark imports.** |\n| `adapters/semanticdf-spark` | Legacy fluent library + `SparkEngineProvider` (implements `Engine[R]` against the portable core). |\n| `adapters/semanticdf-trino` | Trino engine adapter — `TrinoEngine`, `TrinoEngineProvider`, `TrinoQueryCompiler`. |\n| `adapters/semanticdf-duckdb` | In-process DuckDB engine adapter — `DuckDBEngine`. |\n| `adapters/semanticdf-unity-catalog` | REST catalog adapter (read-only) over Unity Catalog. |\n| `adapters/semanticdf-hive-metastore` | Thrift catalog adapter (read-only) over Hive Metastore. |\n| `semanticdf-mcp` | MCP server with engine registry (`MCPEngineProvider` + `MCPEngineRegistry`); routes queries to the chosen engine provider. |\n\n**Status of the migration**: the portable types are in place; engine\nadapters for Spark, Trino, and DuckDB compile against them and round-\ntrip queries end-to-end. The legacy fluent API (`SemanticTable.query(...).execute(spark)`)\ncoexists with the new portable types until consumers migrate. The\nremaining migration step is wiring `SemanticTableCore` (the fluent\nAPI) to emit portable `RelOp` and route through `Engine[R]` instead\nof compiling directly to Spark plans — tracked as a follow-on.\n\n**Catalog identity + CAS** (design §5.3): `core/catalog/` defines\n`CatalogRef`, `CatalogIdentity`, `PublishMode` (`CreateOnly` /\n`Upsert` / `CompareAndSet`), `PublishResult`, and `CatalogAdapter`.\nAdapters publish models/rollups/extension blobs with per-identity\natomic publication semantics.\n\n## The platform\n\n`semanticdf-platform` is a **standalone Restate-native runtime** that\nturns the library into a long-running service. The library compiles\nsemantic definitions to Spark plans; the platform puts a durable\ningress in front of them, persists models and audit logs to Postgres,\nand reconciles streaming queries across JVM crashes.\n\nYou can use the library directly (embed in your own app, call\n`SemanticTable.toDataFrame(spark)`), or you can run the platform and\ntalk to it over HTTP. The two are independent — the library has no\nRestate dependency; the platform has no Spark dialect of its own.\n\n### Library vs. platform at a glance\n\n| | Library (`io.semanticdf:semanticdf_2.13`) | Platform (`semanticdf-platform/`) |\n|---|---|---|\n| **Lifetime** | Embedded in your JVM | Long-running JVM (5 services + Restate ingress) |\n| **API style** | Scala DSL + YAML | HTTP (raw or via MCP) |\n| **Durability** | Your app's call | Restate journal + Postgres |\n| **Streaming** | Spark Structured Streaming in your app | Stream registry survives JVM crashes |\n| **Crash recovery** | n/a | Auto-replay + bulk-startup sweep |\n| **Spark Connect** | ✓ via `SdfSession` | ✓ opt-in via `SEMANTICDF_SPARK_CONNECT_URL` |\n\nThe platform is the recommended deployment for production: it gives\nyou model versioning, audit replay, streaming lifecycle, and a\ndurable cache (opt-in via `SEMANTICDF_RESULT_CACHE=memory`) without\nre-implementing them in your app.\n\n### Platform get started\n\nThe platform is a separate Maven project that depends on the library.\nFor a copy-paste runnable setup, see\n[`semanticdf-platform/README.md`](semanticdf-platform/README.md). Quick\nsteps:\n\n1. **Build the library**:\n   ```bash\n   mvn install -DskipTests    # produces semanticdf_2.13-0.2.1.jar in ~/.m2\n   ```\n2. **Start a Restate dev server** (single-node, in-memory journal):\n   ```bash\n   docker run --rm --name restate -d -p 8080:8080 -p 9070:9070 -p 9071:9071 \\\n       docker.io/restatedev/restate:latest\n   ```\n3. **Start the platform**:\n   ```bash\n   cd semanticdf-platform\n   mvn exec:java -Dexec.mainClass=io.semanticdf.platform.PlatformApplication -Plocal\n   ```\n   The `-Plocal` profile bundles Spark into the runtime classpath (default scope is `provided` for slim production JARs). The platform ships a `.mvn/jvm.config` with the `--add-opens` flags Spark 3.5.x needs on JDK 17, so no `MAVEN_OPTS` shell wrapper is required. The platform listens on `http://localhost:8080`. The Restate ingress from step 2 is on `8080`.\n4. **Register a model**:\n   ```bash\n   curl -X POST http://localhost:8080/ModelService/flights/register \\\n       -H \"Content-Type: application/json\" \\\n       -d '{\"modelName\":\"flights\",\"yaml\":\"flights:\\n  table: flights_tbl\\n  dimensions:\\n    carrier: carrier\\n  measures:\\n    rows: \\\"count(*)\\\"\\n\"}'\n   ```\n5. **Query it**:\n   ```bash\n   curl -X POST http://localhost:8080/QueryService/runQuery \\\n       -H \"Content-Type: application/json\" \\\n       -d '{\"modelName\":\"flights\",\"measures\":[\"rows\"],\"dimensions\":[\"carrier\"],\"where\":\"\"}'\n   ```\n\nThe platform is opt-in for the durable substrate. By default it runs\nend-to-end in journal-only mode (no Postgres). Set the env vars to\nopt-in to durable persistence:\n\n| Env var | When true | Default |\n|---|---|---|\n| `SEMANTICDF_MODELS_PERSIST` | `ModelService.register` writes to Postgres | false |\n| `SEMANTICDF_AUDIT_PERSIST` | `AuditService` writes to Postgres | false |\n| `SEMANTICDF_RESULT_CACHE` | `memory` enables the LRU query cache | `noop` |\n| `SEMANTICDF_SPARK_CONNECT_URL` | Use Spark Connect (remote cluster) | unset |\n| `RESTATE_INGRESS_URL` | Register against an external Restate | unset |\n\n### Platform tour\n\nFive services wired into one Restate endpoint:\n\n| Service | Type | Key | Job |\n|---|---|---|---|\n| `ModelService` | `@VirtualObject` | model name | Compile YAML, persist, hot-reload |\n| `QueryService` | `@Service` (stateless) | — | Execute queries, cache results |\n| `StreamingService` | `@Workflow` | stream-id | Start, monitor, reconcile |\n| `AuditService` | `@VirtualObject` | tenant | Replay-safe audit log |\n| `CatalogService` | `@Service` (stateless) | — | List / describe models |\n\nState placement rule: **Restate journal = coordination (recent, recoverable from replay); Postgres = record (durable, queryable).**\n\nFor the full architecture, see\n[`docs/design/platform-architecture.md`](docs/design/platform-architecture.md).\nFor the rationale and trade-offs, see\n[`docs/design/platform-services-completion-plan.md`](docs/design/platform-services-completion-plan.md).\n\n### When to use the platform vs. the library\n\nUse the **platform** when:\n- You want one canonical set of metric definitions shared across many consumers (dashboards, notebooks, agents).\n- You need streaming queries that survive JVM restarts.\n- You want a replayable audit log of every query.\n- You're wiring an LLM agent to your data — the platform's durable ingress + model registry is a natural fit.\n\nUse the **library directly** when:\n- You're embedding semantic compilation in a single app (e.g., a Spark workload job).\n- You don't need cross-process state.\n- You want minimal dependencies (no Restate, no Postgres).\n\n## Build\n\nRequires **JDK 17** and **Maven 3.9+**. Spark is on the classpath as `provided` (it comes\nfrom your cluster/runtime).\n\n```bash\nmvn test                      # Spark 3.5.8 (default)\nmvn -Pspark4 test             # Spark 4.1.1 (latest stable)\n```\n\n## Quick start\n\nAdd the Maven dep and paste this into your project:\n\n```scala\nimport io.semanticdf._\nimport org.apache.spark.sql.SparkSession\nimport org.apache.spark.sql.functions.{count, lit, sum}\n\nimplicit val spark = SparkSession.builder().master(\"local[2]\").getOrCreate()\nimport spark.implicits._\n\nval flights = Seq(\n  (\"AA\", 100, 5), (\"UA\",  80, 3), (\"DL\", 150, 6),\n).toDF(\"carrier\", \"distance\", \"passengers\")\n\nval flightsModel = toSemanticTable(flights, name = Some(\"flights\"))\n  .withDimensions(Dimension(\"carrier\", t => t(\"carrier\")))\n  .withMeasures(\n    Measure(\"flight_count\",     t => count(lit(1))),\n    Measure(\"total_passengers\",  t => sum(t(\"passengers\"))),\n    Measure(\"avg_passengers\",    t => t(\"total_passengers\") / t(\"flight_count\")),\n  )\n\nflightsModel.groupBy(\"carrier\").aggregate(\"flight_count\", \"avg_passengers\").execute.show\n```\n\n> For a full walkthrough (prerequisites, Maven coordinates, troubleshooting) see **[`docs/getting-started.md`](docs/getting-started.md)**.\n\nOnce this runs, continue with [`docs/guide.md`](docs/guide.md) for the narrative walkthrough that explains how the compilation works under the hood.\n\n## CLI Tools\n\nTwo tools live in `src/main/scala/io/semanticdf/tools/`, both runnable via `mvn exec:java`:\n\n### docsgen — YAML model → browsable HTML\n\n```bash\nmvn exec:java \\\n  -Dexec.mainClass=io.semanticdf.tools.Main \\\n  -Dexec.args=\"docsgen --path examples/starter/models/ --out docs/index.html\"\n# Open docs/index.html in a browser\n```\n\nReads one YAML file or a directory of `.yml` files and emits a self-contained HTML page (sidebar nav, per-model cards, dimension/measure/join tables, time/entity/pii badges). No Spark needed; no external dependencies.\n\n### introspect — DataFrame → YAML model starter\n\n```bash\nmvn exec:java \\\n  -Dexec.mainClass=io.semanticdf.tools.Main \\\n  -Dexec.args=\"introspect --path examples/starter/data/flights.csv --format csv --model flights\"\n# Writes a starter YAML to stdout (or --out models/flights.yml to write to a file).\n```\n\nReads a data file via Spark, infers dimensions (StringType → dim, NumericType → sum/avg, TimestampType → time dimension with `is_time_dimension: true`), and emits a starter YAML model. Edit the output to refine types, add descriptions, and customise expressions.\n\n**Note — JDK 17 + Spark needs `--add-opens` flags** for any command that touches Spark (which includes `introspect`). Without them, the JVM crashes with `sun.nio.ch.DirectBuffer` access errors. Either set `MAVEN_OPTS` to the full flag set (see [`docs/runtime-quickstart.md`](docs/runtime-quickstart.md#traps) trap #1) or, for project-local reproducibility, drop a `.mvn/jvm.config` with one flag per line. `docsgen` does not need Spark, so it works without the flags.\n\n### okfgen — YAML models → agent knowledge catalog (sidecar markdown)\n\n```bash\nmvn exec:java \\\n  -Dexec.mainClass=io.semanticdf.tools.Main \\\n  -Dexec.args=\"okfgen --path examples/starter/models/ --out docs/agents/reference/starter/\"\n# Writes one Markdown concept doc per model under the --out directory.\n```\n\nGenerates per-model **sidecar Markdown** (OKF — the agent knowledge format) for an\nexternal agent catalog. Each `models/foo.yml` becomes `agents/reference/<project>/foo.md`,\nwith one-line dimensions/measures/joins/filters plus a row-by-row examples reference.\nThe YAML stays the engine source of truth — OKF is a publishing layer, not a schema\nreplacement. See [`docs/agents/okf-mapping.md`](docs/agents/okf-mapping.md) for the\nmapping rules and output format. `make okfgen-check` is the CI drift check — it re-runs\nokfgen to a tempdir and `diff -ru`'s the result against the committed bundle.\n\n## MCP server (`semanticdf-mcp`)\n\nThe `semanticdf-mcp/` sibling module is a Model Context Protocol server that\nexposes semanticdf to any MCP-compatible client (Claude Desktop, Cursor, Continue)\nover **stdio**. The six tools from\n[`docs/agents/mcp-contract.md`](docs/agents/mcp-contract.md) v5:\n\n| Tool | Purpose |\n|---|---|\n| `list_models`    | Reports loaded models (name + description) |\n| `describe_model` | Full schema (dimensions, measures, joins, filters, version) + optional OKF sidecar |\n| `query`          | Runs a query, returns rows + columns |\n| `explain`        | Same request shape, no execution — emits the semantic plan |\n| `introspect`     | Auto-generate starter YAML from a DataFrame |\n| `audit_log`      | Returns the recent `AuditEvent` stream — what the agent has queried, when, and whether it succeeded |\n\n### Run the server\n\n```bash\nmvn install -DskipTests                                # install parent library to local ~/.m2\ncd semanticdf-mcp && mvn package\nmvn exec:java -Dexec.mainClass=io.semanticdf.mcp.Main \\\n  -Dexec.args=\"--models ../examples/starter/models/ \\\n               --data ../examples/starter/data-config.yaml \\\n               --okf-bundle /tmp/okf/\"\n```\n\nAll three flags are required:\n\n| Flag | What |\n|---|---|\n| `--models <dir>`     | directory of `*.yml` model files |\n| `--data <file>`      | data-config YAML (`data:` block per the contract) |\n| `--okf-bundle <dir>` | where OkfGen writes the OKF markdown; server reads it into memory at startup |\n\n### Wire up a client (Claude Desktop example)\n\n```json\n{\n  \"mcpServers\": {\n    \"semanticdf\": {\n      \"command\": \"java\",\n      \"args\": [\n        \"-jar\",\n        \"/path/to/semanticdf-mcp/target/semanticdf-mcp_2.13-<version>.jar\",\n        \"--models\",\n        \"/path/to/your/models\",\n        \"--data\",\n        \"/path/to/your/data-config.yaml\",\n        \"--okf-bundle\",\n        \"/tmp/okf/\"\n      ]\n    }\n  }\n}\n```\n\nThe server source lives in [`semanticdf-mcp/`](semanticdf-mcp/README.md). See\n[`docs/agents/mcp-contract.md`](docs/agents/mcp-contract.md) v2 for the\nrequest/response schema of every tool.\n\n## Capabilities\n\n### Calc measures (name-based compilation)\n\nA calc measure references *other measures* by name. The compiler classifies base vs calc\nautomatically, pulls transitive dependencies, and applies calcs in topological layers.\n\n```scala\n// Request only a leaf calc — its deps (avg → total_distance + flight_count) are pulled.\nflights.groupBy(\"carrier\").aggregate(\"avg_distance_per_flight\").execute(spark)\n```\n\n- Calc-of-calc chains resolve by name across layers; cycles raise a clear error.\n- Typos give a \"did you mean?\" suggestion instead of a crash.\n\n### Percent-of-total (`t.all`)\n\n`t.all(\"measure\")` resolves to the grand total — the same measure aggregated with no group\nkeys, cross-joined into the result. **The formula is recomputed at zero grain**, so\nnon-sum totals are correct:\n\n```scala\n.withMeasures(\n  Measure(\"total_passengers\", t => sum(t(\"passengers\"))),\n  // pct sums to 1.0 by construction:\n  Measure(\"pct_of_total\", t => t(\"total_passengers\") / t.all(\"total_passengers\")),\n)\n```\n\n`t.all(\"avg_distance_per_flight\")` returns **225** (grand avg = 6750/30), *not* 675 (the\nsum of per-group averages). That's the classic BI trap, fixed.\n\n> **Division by zero:** Spark's `/` returns `null` on zero/missing denominators (correct SQL\n> semantics). If you want an explicit default (e.g. `0.0` instead of `null`), use\n> `CalcHelpers.safeDivide(num, denom, defaultValue = 0.0)`.\n\n### Window functions\n\nA Measure is just `SemanticScope => Column`, so any Spark window function is legal\ninside the lambda. The window evaluates against the post-aggregation DataFrame (Pass 2\nof the calc layer), so it can reference group-by keys and base measures by name.\n\n```scala\nimport org.apache.spark.sql.expressions.Window\nimport org.apache.spark.sql.functions.{row_number, rank, sum}\n\nval st = toSemanticTable(flightsDf, name = Some(\"flights\"))\n  .withDimensions(\n    Dimension(\"carrier\", t => t(\"carrier\")),\n    Dimension(\"origin\",  t => t(\"origin\")),\n  )\n  .withMeasures(\n    Measure(\"flight_count\", t => count(lit(1))),\n    // rank within each carrier by origin:\n    Measure(\"rank_per_carrier_origin\",\n      t => row_number().over(Window.partitionBy(t(\"carrier\")).orderBy(t(\"origin\")))),\n    // running total of total_passengers across origins per carrier:\n    Measure(\"running_total\",\n      t => sum(t(\"total_passengers\")).over(\n        Window.partitionBy(t(\"carrier\")).orderBy(t(\"origin\")))),\n  )\n  .groupBy(\"carrier\", \"origin\")\n  .aggregate(\"flight_count\", \"rank_per_carrier_origin\", \"running_total\")\n```\n\nWindow functions work in the Scala DSL. The YAML loader also accepts raw SQL\nwindow expressions in `measures:` (e.g. `row_number() over (partition by carrier\norder by origin)`); the parser blocklist covers `row_number`, `rank`, `dense_rank`,\n`lag`, `lead`, `ntile`, `first_value`, `last_value`, plus window-frame SQL\nkeywords (`order`, `rows`, `range`, `between`, `unbounded`, `preceding`,\n`following`, `and`, `current`, `asc`, `desc`, `nulls`).\n\n> **Known limitation:** a window function that references group-by keys (e.g.\n> `Window.partitionBy(t(\"carrier\"))`) cannot be combined with `t.all(...)` for\n> percent-of-total — the zero-grain totals table has no group-by keys, so the\n> window evaluation fails. Workaround: use a window that doesn't reference\n> group-by keys (e.g. `Window.orderBy(...)` only), or compute percent-of-total\n> as a separate measure.\n\n### Transforms (per-row computations, applied at model-load)\n\nPer-row logic — `datediff(...)`, `case when ...`, window functions — is\napplied to the source DataFrame at model-load time via the YAML `transforms:`\nblock or Scala `withTransforms(...)`. Transformed columns become part of\nthe source DataFrame and are visible to subsequent filters and measures.\nOrder matters (no automatic topological sort).\n\nFor a worked example with the YAML + Scala equivalents, the model-load\nlifecycle, and what fields downstream measures see, see\n[`docs/guide.md` → Transforms](docs/guide.md#dimensions-measures-and-transforms).\n\n### Filters (pre-join row-level hygiene, applied at model-load)\n\nRow-level **hygiene** — drop rows missing a required field, drop cancelled\norders, dedup before aggregation — doesn't fit a query, because it\ngoverns which rows the model *contains*, not which rows a particular\nquery returns. Declare it on the model via `filters:` (YAML) or\n`withRowFilter(...)` (Scala DSL). Filters run **pre-agg, pre-join**,\nagainst this model's source table only.\n\n```yaml\nflights:\n  table: flights_csv\n  filters:\n    require_origin_and_carrier:\n      expr: \"origin IS NOT NULL AND carrier IS NOT NULL\"\n      description: \"Drop rows with null origin or carrier.\"\n  dimensions:\n    carrier: carrier\n```\n\n`SparkFilterValidator` enforces pre-join visibility at load time — a\nfilter referencing a joined-side column is rejected. For the Scala DSL\nform, the visibility rules, and a worked example, see\n[`docs/guide.md` → Filters](docs/guide.md#filters-pre-join-hygiene-public-where-for-query-time).\n\nAlso see [`docs/calc-author-guide.md`](docs/calc-author-guide.md) for the\ndetailed validator rules.\n\n### Joins (`join_one` / `join_many` / `join_cross`)\n\n```scala\nval orders  = toSemanticTable(ordersDf, name = Some(\"orders\"))\nval items   = toSemanticTable(lineItemsDf, name = Some(\"line_items\"))\n\n// join_many pre-aggregates each side at the join-key grain to prevent fan-out inflation.\nval joined = orders.join_many(items, on = \"order_id\")\n  .withMeasures(\n    Measure(\"orders.total_qty\", t => sum(t(\"line_items.qty\"))),\n  )\n```\n\n- `join_one` — one-to-one / parent-child (post-agg safe).\n- `join_many` — one-to-many; **both sides pre-aggregated** at join-key grain before joining\n  to prevent fact inflation.\n- `join_cross` — Cartesian product.\n- Merged model uses left-precedence; prefixed names (`\"orders.total_qty\"`) resolve correctly.\n\n### Filters — WHERE/HAVING auto-routing\n\n```scala\nflights.where(\"carrier\" === \"AA\")                      // dimension → WHERE (pre-agg)\n       .where(\"total_passengers\" > 600)                 // measure   → HAVING (post-agg)\n       .where((\"carrier\" === \"AA\") and (\"total\" > 100)) // AND-split: WHERE + HAVING\n       .where((\"carrier\" === \"AA\") or (\"total\" > 800))  // OR-whole (can't split)\n```\n\n- `where(pred)` routes automatically: dimension predicates → pre-agg, measure predicates →\n  post-agg. `And` compounds split per-condition; `Or`/`Not` mixing dim+measure stay whole.\n- `having(pred)` forces post-agg.\n- DSL: `===` `=!=` `>` `>=` `<` `<=` `in` `notIn` `isNull` `isNotNull`, plus `and`/`or`/`.not`.\n  (Standard `==`/`!=` are `final` on `Any` and return `Boolean` — unusable for a deferred DSL.)\n\n### Order, limit, and one-shot `query()`\n\n```scala\n// Fluent chain:\nflights.groupBy(\"carrier\").aggregate(\"total_passengers\")\n  .orderBy(SortKey.desc(\"total_passengers\")).limit(10)\n  .execute(spark)   // top-10 carriers\n\n// Or a one-shot bundle:\nflights.query(\n  measures   = Seq(\"total_passengers\"),\n  dimensions = Seq(\"carrier\"),\n  having     = Some(\"total_passengers\" > 600),\n  orderBy    = Seq(SortKey.desc(\"total_passengers\")),\n  limit      = Some(10),\n).execute(spark)\n```\n\n### Querying from a notebook via `spark.sql(...)`\n\nFor notebook / SQL-first consumers, a compiled `SemanticTable` can be registered\nas a Spark temp view and queried with plain SQL via\n`.createOrReplaceTempView(\"flights_view\")` followed by `spark.sql(...)`. The view is the\n*compiled output* of the model — joins, pre-join filters, and pre-aggregation\nall happen *before* the SQL queries the view.\n\nSee [`docs/guide.md` → Notebook escape hatch](docs/guide.md#notebook-escape-hatch--raw-sql-via-a-temp-view)\nfor the worked example, scoping rules, and a multi-cell notebook workflow.\n\n### Typed queries (compile-time safety)\n\nThe string-based API above is convenient but typo-prone — a wrong field name is a runtime\nerror. An **additive** typed API catches those mistakes at compile time.\n\n```scala\n// Declare phantom types + implicit typeclass witnesses (one-time, per field):\nobject Flights {\n  sealed trait Carrier\n  sealed trait Origin\n  sealed trait TotalPassengers\n  sealed trait FlightCount\n\n  implicit val carrier: SemanticDimension[Carrier]           = SemanticDimension.of[Carrier](\"carrier\")\n  implicit val origin:  SemanticDimension[Origin]            = SemanticDimension.of[Origin](\"origin\")\n  implicit val pax:     SemanticMeasure[TotalPassengers]     = SemanticMeasure.of[TotalPassengers](\"total_passengers\")\n  implicit val count:   SemanticMeasure[FlightCount]         = SemanticMeasure.of[FlightCount](\"flight_count\")\n}\nimport Flights._\n\n// Typed query — wrong ref types are caught at compile time:\nval st = toSemanticTable(flightsDf, name = Some(\"flights\"))\nval rows = st.groupByDimensions(carrier)\n              .aggregateMeasures(pax, count)\n              .orderBy(SortKey.desc(pax))\n              .limit(10)\n              .execute(spark)\n\n// Typed predicate (operator kind is in the method name, not a runtime string):\nval highPax = st.where(Predicate.Gt(pax, 600)).execute(spark)\n\n// Typed measure declaration — name read from the SemanticMeasure witness:\nimport org.apache.spark.sql.functions.row_number\nimport org.apache.spark.sql.expressions.Window\nval enriched = st.withMeasures(pax, t => row_number().over(Window.partitionBy(t(\"carrier\")).orderBy(t(\"total_passengers\").desc)))\n\n// Compile-time guarantees:\n//   groupByDimensions(pax)          // COMPILE ERROR — pax is a Measure, not a Dimension\n//   aggregateMeasures(carrier)      // COMPILE ERROR — carrier is a Dimension, not a Measure\n//   Compare.Greater(pax, 600)       // COMPILE ERROR — typo; only Eq/Ne/Lt/Le/Gt/Ge compile\n//   Predicate.Gt(pax, \"six hundred\")  // compiles — predicate.value is Any (fails at runtime, not compile time)\n```\n\n- Pure additions to the library: the string API is unchanged. Zero runtime overhead —\n  `groupByDimensions`/`aggregateMeasures` and `Compare.Gt` compile to the same Spark\n  `Column` expressions as the string forms.\n- Arities 1–4 are fully type-checked at compile time; the `…All(refs)` overloads do a\n  single runtime check for arity 5+ (rare in practice).\n- The `FieldRef[T]` carrier is a value class — no allocation on the hot path.\n- See [the typeclass-design rationale](docs/backlog-type-safety.md) for the design rationale\n  and what's still deferred (the typed-arithmetic DSL (planned) — see\n  [`docs/backlog-type-safety.md`](docs/backlog-type-safety.md) §E3). The `ResultDecoder[T]`\n  typeclass (including macro derivation for case classes via `ResultDecoder.derive[T]`)\n  and the `queryAs[T]: Dataset[T]` terminal are shipped.\n\n### Typed query results — `ResultDecoder[T]`\n\nThe same compile-time guarantee applies to the **output** side. `SemanticTable.collectAs[T]`\nreturns a `Seq[T]` rather than untyped `Seq[Row]`, plumbed through a small\ntypeclass:\n\n```scala\n// Built-in primitive decoders read column 0 of each row:\nval names: Seq[String] = table.collectAs[String](spark)\nval counts: Seq[Long]  = table.collectAs[Long](spark)\n\n// Case-class decoders — derive[T] generates the instance at compile time:\ncase class CarrierCount(carrier: String, count: Long)\nimplicit val decoder: ResultDecoder[CarrierCount] = ResultDecoder.derive[CarrierCount]\nval typed: Seq[CarrierCount] = table.collectAs[CarrierCount](spark)\n```\n\nThe macro (`ResultDecoder.derive[T]`) is a Scala 2 blackbox macro that inspects the\ncase class's primary constructor and emits one `row.getX(i)` call per field.\nSupported field types: `String`, `Int`, `Long`, `Double`, `Float`, `Boolean`,\n`Short`, `Byte`, `java.math.BigDecimal`. Unsupported field types\n(`java.time.Instant`, sealed traits, `Option[T]`, nested case classes, ...)\nproduce a **compile-time error** pointing at the offending constructor parameter,\nso the user can either rename, restructure, or supply a manual\n`ResultDecoder[T]` instance via `implicit val`.\n\nFor richer shapes that the macro doesn't support, the manual form is just as\nconcise as a one-line `val`:\n```scala\nimplicit val decoder: ResultDecoder[Foo] = new ResultDecoder[Foo] {\n  def decode(row: Row): Foo = Foo(row.getString(0), foo.bar(row))\n}\n```\n\n### Time semantics\n\n```scala\nval st = toSemanticTable(flightsWithTimeDf, name = Some(\"flights\"))\n  .withDimensions(\n    Dimension.time(\"ts\", t => t(\"ts\"), smallestTimeGrain = Some(\"day\")),\n  )\n  .withMeasures(Measure(\"total_passengers\", t => sum(t(\"passengers\"))))\n\nst.atTimeGrain(\"ts\", \"month\").groupBy(\"ts\").aggregate(\"total_passengers\").execute(spark)\n// groups by truncated month (date_trunc)\n\n// Or via query():\nst.query(\n  measures   = Seq(\"total_passengers\"),\n  dimensions = Seq(\"ts\"),\n  timeGrain  = Some(\"month\"),\n  timeRange  = Some((\"2024-01-01\", \"2024-02-28\")),  // filters raw ts, pre-truncation\n).execute(spark)\n```\n\n- `Dimension.time(...)` marks a timestamp dimension; `smallestTimeGrain` floors requests.\n- `atTimeGrain(dim, \"month\")` overrides the dimension's expr with `date_trunc`.\n- Grain too fine (e.g. `\"hour\"` when `smallestTimeGrain = \"day\"`) raises a clear error.\n- `time_range` filters the raw column; `time_grain` affects only grouping.\n\n### EXPLAIN — op tree, Spark plan, and semantic intent\n\nThree flavours of plan inspection, each for a different debugging need:\n\n```scala\nmodel.explain()                // op tree shape (no Spark compilation)\nmodel.explain(spark)           // Catalyst physical plan\nmodel.explainSemantic(spark)   // WHY: filter routing, pulled measures, etc.\n```\n\n`explainSemantic` is the one a developer usually wants. See\n[`docs/guide.md` → How a query compiles](docs/guide.md#how-a-query-compiles)\nfor the worked example with sample output.\n\n## API reference\n\n| Method | Description |\n|---|---|\n| `toSemanticTable(df, name?)` | Construct a semantic model from a base `DataFrame`. |\n| `.withDimensions(...)` / `.withMeasures(...)` | Immutable model extension. Typed `withMeasures(measure, expr)` overload accepts a `SemanticMeasure` witness directly via subtyping. |\n| `.withTransforms(transforms*)` | Per-row logic (e.g. `datediff`, `case when`) applied to source data at model-load. Mirrors the YAML `transforms:` block. |\n| `.withRowFilter(name, expr, description: Option[String], metadata: Map[String, String])` | Attach a pre-join row filter (Spark SQL string) declared in the model. Mirrors the YAML `filters:` block. `SparkFilterValidator` enforces pre-join column visibility (source + transforms; joined-side columns not visible) at load time. |\n| `.version(v: Int)` | Set the model's version (forward-compat hint for consumers). `table.version` reads the current value (0 = unversioned). |\n| `.join_one(other, on)` / `.join_many(other, on)` / `.join_cross(other)` | Joins. |\n| `.where(pred)` / `.having(pred)` | Filters (auto-routed WHERE/HAVING). |\n| `.groupBy(keys...).aggregate(measures...)` | Group-by + aggregate. |\n| `.groupByDimensions[D1..D4](refs...)` / `.groupByDimensionsAll(refs)` | Typed group-by — ref kind (dimension) is checked at compile time (arity 5+ at runtime). |\n| `.aggregateMeasures[M1..M4](refs...)` / `.aggregateMeasuresAll(refs)` | Typed aggregate — ref kind (measure) is checked at compile time (arity 5+ at runtime). |\n| `Predicate.Eq/Ne/Gt/Ge/Lt/Le/in/notIn/isNull/isNotNull[F](ref, v)` | Typed predicate factories — `ref: FieldRef[F]` with `SemanticField[F]` witness. |\n| `Compare.Gt(field, value)` / `Compare.Eq(field, value)` / etc. | Sealed comparison ADT — operator kind (Eq/Ne/Lt/Le/Gt/Ge) is in the type, not a string. `Compare.apply(\"gt\", ...)` legacy factory is preserved. |\n| `.atTimeGrain(dim, grain)` | Truncate a time dimension for grouping. |\n| `.orderBy(keys...)` / `.limit(n)` | Terminal ordering / top-N. `SortKey.asc(ref)` / `SortKey.desc(ref)` accept typed `SemanticField` witnesses. |\n| `.query(measures, dimensions?, where?, having?, orderBy?, limit?, timeGrain?, timeGrains?, timeRange?)` | One-shot bundle. |\n| `.queryAs[T](measures, dimensions?, where?, having?, orderBy?, limit?, timeGrain?, timeGrains?, timeRange?)(implicit spark, decoder: ResultDecoder[T], encoder: Encoder[T]): Dataset[T]` | Typed one-shot bundle. Same shape as `.query` but returns a `Dataset[T]`, decoding rows into a case class via the implicit `ResultDecoder[T]` (use `ResultDecoder.derive[T]` for the case-class witness) and `Encoder[T]` (use `import spark.implicits._`). Compile-time type-safety on result field names and types. |\n| `Measure.typed[T](name: String, expr: TypedSemanticScope => TypedColumn[T]): Measure` | Typed measure factory. Same shape as the `Measure` case class but the lambda's return type is type-checked at compile time via the phantom `T`. Compose with `TypedArithmetic.{divide, plus, minus, multiply}` for type-checked arithmetic. The typed form lowers to a plain `Measure` at runtime — works with `withMeasures(...)` as usual. Zero runtime overhead, no memory leak. |\n| `TypedArithmetic.{divide, plus, minus, multiply}[T, U, R](a: Column, b: Column)(implicit nt: Numeric[T], nu: Numeric[U], nr: Numeric[R]): TypedColumn[R]` | Typed arithmetic ops for measure lambdas. The compiler requires `Numeric[T]`, `Numeric[U]`, `Numeric[R]` to be in implicit scope — `String` and other non-numeric types fail at compile time. Returns a `TypedColumn[R]` (value class wrapping `Column`); implicit conversion to `Column` makes the typed form drop-in compatible with the untyped `SemanticScope => Column` lambda. The function body is just the corresponding Spark `Column` op — type parameters are erased. |\n| `.toDataFrame(spark)` / `.execute(spark)` | Batch terminal (compile to `DataFrame`). With `implicit val spark: SparkSession` in scope, both can be called without the argument (`.toDataFrame` / `.execute`). |\n| `.previewSchema(spark)` | Output schema (compile to `StructType`, no rows). |\n| `.withHint(strategy, params*)` | Apply a Spark planner hint (e.g. `\"broadcast\"`, `\"repartition\", n`). |\n| `.withAuditSink(sink: AuditSink)` | Install an `io.semanticdf.audit.AuditSink` — every `query()` / `execute()` / `toDataFrame()` emits an `AuditEvent` (model, request shape, elapsed, status). Default `NoOp` (no overhead). Requires a non-empty `auditRequest` (set by `query(...)`; cleared by post-query shape-changers like `withDimensions`/`withMeasures`/`withRowFilter`/`withTransforms`/`where`/`having`/`orderBy`/`limit`/`atTimeGrain`); otherwise `.toDataFrame()` throws `IllegalStateException`. See [the runtime-tuning walk-through](docs/tutorial-runtime-tuning.md). |\n| `.withResultCache(cache: ResultCache)` | Install an `io.semanticdf.cache.ResultCache` — identical `query()` calls return from cache without re-executing the Spark plan. Default `NoOp`. Cache keys are stable SHA-256s of the request shape. Same `auditRequest` requirement as `withAuditSink`; cleared by post-query shape-changers together with the request via `invalidateAuditRequest`. See [the runtime-tuning walk-through](docs/tutorial-runtime-tuning.md). |\n| `.withMaterialize(level: StorageLevel)` | Opt-in DataFrame persistence on the fast path of `toDataFrame` (no audit, no cache) — `df.persist(level)` is applied so multiple actions on the returned `DataFrame` reuse the persisted storage instead of re-executing the Spark plan. Call `df.unpersist()` on the returned DataFrame to release. Default `None` (no persist). Storage level choice is the operator's responsibility — `MEMORY_ONLY` on a large query can OOM the cluster. Audit/cache paths return a `parallelize`-based DataFrame that's effectively `MEMORY_ONLY` for the call's duration; `withMaterialize` does not apply there (the user never sees the compiled DF). See `docs/design/with-materialize.md`. |\n| `.withMaxRows(n: Int)` | Cap on rows returned per query. `n = 0` disables (escape hatch); `n < 0` throws. Cap fires on cache-miss and audit-only paths of `toDataFrameInternal` — the fast path (no audit, no cache) skips the cap. Default 100,000. See [the runtime-tuning walk-through](docs/tutorial-runtime-tuning.md). |\n| `.withBroadcastJoinThreshold(bytes: Long)` | Opt-in `broadcast(right)` on equi-joins when `right.stats.sizeInBytes < bytes`. `bytes = 0` disables (no override); `bytes < 0` throws. LEFT-wins / RIGHT-fallback precedence at join construction. Streaming queries: no-op (AQE is disabled by Spark for streaming DataFrames via `ResolveWriteToStream`). See [the runtime-tuning walk-through](docs/tutorial-runtime-tuning.md). |\n| `.withSalt(n: Int)` | Opt-in skew-handling hint — translates to `spark.sql.adaptive.skewJoin.skewedPartitionFactor = n` (and re-enables the parent AQE flag). `n = 0` converts to `None` (silently disable); `n < 0` throws. Spark AQE handles skew by splitting each skewed partition and replicating the matching partition on the other side — a custom `(rand() * n)` salt column would produce WRONG results in shuffled joins because LEFT/RIGHT executors have different RNG sequences. Streaming: no-op (same Spark limitation). See [the runtime-tuning walk-through](docs/tutorial-runtime-tuning.md). |\n| `.validate()` | Compile-free structural check; returns `ValidationResult(errors, warnings, isValid)` for CI pre-flight. |\n| `.joins: Seq[JoinInfo]` | All join edges in the model (left/right keys, cardinality: one/one_to_many/many_to_many/cross). Captures join keys at construction time (no compile required). |\n| `.measureKind(name): MeasureKind` | Classify a measure as `Base` / `Calc` / `Window` — useful for tooling that needs to know which measures have a known-name calc dependency chain. |\n| `.sourceTable: Option[String]` | Back-reference to the originating table name (when loaded from a registered/catalog source via `YamlLoader`). |\n| `.filters: Seq[SemanticFilter]` | The model's pre-join row filters in declaration order (`name`, `expr`, `description`, `metadata`). |\n| `.dimensions: Map[String, Dimension]` / `.measures: Map[String, Measure]` / `.findDimension(name)` / `.findMeasure(name)` | Catalog accessors. |\n| `.createOrReplaceTempView(name)` / `.createTempView(name)` / `.createOrReplaceGlobalTempView(name)` | Compile to `DataFrame` and register as a Spark temp view (session or global). All three take `(implicit spark: SparkSession)` — call from inside a `SparkSession.builder()` block. |\n| `.explain()` | Print the SemanticDF op-tree summary (no Spark compile). |\n| `.explain(spark)` | Run the full query and print Spark's **simple** physical plan. |\n| `.explainExtended(spark)` | Run the full query and print Spark's **extended/cost** plan (incl. logical-plan sections). |\n| `.explainSemantic(spark?)` / `.explainSemantic(spark?, Scope)` | Multi-section human-readable plan: filter routing, transitive deps, join strategies, warnings. | |\n\n`Dimension.time(...)` / `Dimension.entity(...)` are ergonomic factories. `Predicate._`\nbrings the filter DSL into scope. `SortKey.desc(...)` / bare `String` (ascending) drive\n`orderBy`. `CalcHelpers.safeDivide(num, denom, defaultValue)` guards zero/missing denominators\nwith an explicit default (Spark `/` returns null on div-by-zero — correct SQL semantics;\nuse `safeDivide` only when null is undesirable).\n\n## Runnable examples\n\nIn-library examples are in `src/main/scala/io/semanticdf/examples/`.\nCompile once with `mvn compile`, then run any example:\n\n```bash\nmvn compile -q\n\nmvn scala:run -DmainClass=io.semanticdf.examples.FlightsBasic\nmvn scala:run -DmainClass=io.semanticdf.examples.FlightsPctTotals\nmvn scala:run -DmainClass=io.semanticdf.examples.OrdersJoinMany\nmvn scala:run -DmainClass=io.semanticdf.examples.FiltersRouting\nmvn scala:run -DmainClass=io.semanticdf.examples.TimeSeries\nmvn scala:run -DmainClass=io.semanticdf.examples.Benchmark\n```\n\nOr submit as a Spark app:\n```bash\nmvn package -q\nspark-submit --class io.semanticdf.examples.FlightsBasic target/semanticdf_2.13-*.jar\n```\n\n## Consumer-facing templates\n\nThe `examples/` directory holds **consumer templates** — standalone Maven sub-projects\nthat show how to *use* SemanticDF in your own codebase. Each is a runnable, copy-pasteable\nproject. They depend on SemanticDF from your local `~/.m2` (run `mvn install` on the\nparent first).\n\n| Template | What it teaches |\n|---|---|\n| [`examples/starter`](examples/starter/README.md) | 7 queries (group-by, pct-of-total, joins, time-grain, filter, top-N window, MoM window) — the canonical \"hello world\" |\n| [`examples/pipeline`](examples/pipeline/README.md) | Full BI lifecycle — raw CSV → ETL → cleaned parquet → declarative YAML queries |\n| [`examples/window-analytics`](examples/window-analytics/README.md) | Window functions: top-N per group, period-over-period, running totals |\n| [`examples/customer-analytics`](examples/customer-analytics/README.md) | RFM segmentation + cohort activity (calc-of-calc composition) |\n| [`examples/operations-analytics`](examples/operations-analytics/README.md) | Order fulfillment time, on-time rate, anomaly detection (z-score) |\n| [`examples/telco-analytics`](examples/telco-analytics/README.md) | Telco: monthly ARPU per plan, promotion effectiveness, roaming revenue |\n| [`examples/hospital`](examples/hospital/README.md) | Hospital: data cleansing workflow (dedup, normalize, fill), ALOS, 30-day readmission rate |\n| [`examples/dbt-reader`](examples/dbt-reader/README.md) | Load a dbt `manifest.json` as a semantic model — the adapter pattern for a third-party interchange format |\n| [`examples/joined-manifest-e2e`](examples/joined-manifest-e2e/README.md) | Cross-process joined-manifest workflow: write → JSON artifact → read via `SDFAdapter` |\n| [`examples/runtime-tuning`](examples/runtime-tuning/README.md) | All six runtime knobs in a customer analytics dashboard — caps, caching, audit, broadcast, materialize, skew handling. Companion to [the walk-through](docs/tutorial-runtime-tuning.md). |\n| [`examples/skewed-join`](examples/skewed-join/README.md) | `withSalt(5)` against a 1M-event star-schema join with a 90/10 hot-key distribution; verifies AQE config + correctness. |\n\nRun any of them:\n\n```bash\ncd examples/window-analytics   # or any other template\nmvn scala:run -DmainClass=com.example.windowanalytics.Main\n```\n\n## Cross-version compatibility\n\nVerified green on both Spark lines (Spark 3.5.8 default + Spark 4.1.1 via `-Pspark4`). All library and MCP tests pass on each JDK 17. The total test count grows with each release; see the surefire reports for the current number.\n\n| Spark | Scala | Status |\n|---|---|---|\n| 3.5.8 (default) | 2.13.18 | ✅ |\n| 4.1.1 (`-Pspark4`) | 2.13.18 | ✅ |\n\nNo code shims are needed — the codebase uses only Spark APIs stable across 3.5→4.x.\n\n## Design & decisions\n\n- **[`DESIGN.md`](DESIGN.md)** — architecture of record: op tree, scopes, calc compilation,\n  joins, percent-of-total, filters, time semantics, invariants.\n- **[`docs/runtime-quickstart.md`](docs/runtime-quickstart.md)** — JDK/Scala/Spark/Maven\n  matrix, build & test commands, CLI tools, the four runtime traps (Java-17\n  module flags, `scala:run` arg leak, deprecated import, version files).\n- **[`docs/tutorial-runtime-tuning.md`](docs/tutorial-runtime-tuning.md)** — the six runtime knobs (`withMaxRows`, `withResultCache`, `withAuditSink`, `withBroadcastJoinThreshold`, `withMaterialize`, `withSalt`) in one walk-through. Decision tree, when-to-use matrix, real-world scenario, anti-patterns.\n- **[`docs/known-limitations.md`](docs/known-limitations.md)** — current scope & guardrails (batch-only, per-session security, symmetric join keys, etc.) with workarounds and roadmap hints. Read before first consumer.\n- **[`docs/calc-author-guide.md`](docs/calc-author-guide.md)** — how to write correct calc measures: ratio, pct-of-total, calc-of-calc, `safeDivide`.\n- **[`docs/backlog-type-safety.md`](docs/backlog-type-safety.md)** — the open list of deferred features and their priority ordering.\n- **[`docs/adr/`](docs/adr/)** — recorded decisions:\n  - [0001](docs/adr/0001-adopt-karpathy-guidelines-not-app-design.md) — karpathy guidelines adopted (think-before-coding, simplicity-first, surgical changes, goal-driven execution); app-design plugin/portal rejected.\n  - [0002](docs/adr/0002-streaming-batch-first-streaming-shaped.md) — batch-first, streaming-shaped (DSL/source-agnostic op tree; batch and streaming terminals share the model definition).\n  - [0003](docs/adr/0003-re-sequence-calc-proof-first.md) — re-sequence to prove name-based calc compilation early.\n",
  "bytes": 48418,
  "sha": "d467928f0bdad81ab573a26470b0a56da121aa830daf53040bd1ac08f02aabd7",
  "repo_slug": "echoenv/semanticdf",
  "fonte": "repo",
  "truncated": false,
  "api": "https://agentalog.com/api/listings/okf_echoenv_semanticdf_docs_agents_reference_4f8a1c3a/readme"
}