{
  "markdown": "# Interactive Data Systems Lab\n\nRunnable Python and Rust data-system internals with an MCP-native runtime copilot layer.\n\nThis repository combines two things:\n\n- a local-first engineering lab for PostgreSQL-like and Databricks-like internals,\n- a Runtime Copilot MCP surface for diagnostics, explainability, regression checks, and operational memory.\n\n## Core capabilities\n\n- Storage and planner internals: heap tables, B-tree indexing, selectivity, and plan choice.\n- Persistence and replay: WAL/checkpoint style flows and deterministic state transitions.\n- Workflow and write-path modeling: event-first architecture, idempotency, retry semantics.\n- Explainable runtime operations: traced runs, failure summaries, regression verdicts, baseline compare.\n- MCP access: machine-usable operational interface instead of ad hoc shell scripts.\n\n## Who this is for\n\n- Data engineers learning warehouse and query-engine internals.\n- Platform and infrastructure engineers teaching storage and execution fundamentals.\n- Teams building onboarding labs, workshops, and demo environments.\n\n## Quick start\n\nRequirements:\n\n- Python 3.10+ (tested with Python 3.14)\n\nSetup:\n\n```bash\npython3 -m venv .venv\nsource .venv/bin/activate\npython -m pip install \"psycopg[binary]\"\n```\n\nRun end-to-end flow:\n\n```bash\ncargo run --bin e2e_flow\n```\n\nRun core demos:\n\n```bash\n.venv/bin/python mini_pg_like.py\n.venv/bin/python mini_databricks_clone.py\ncargo run --bin mini_pg_like\ncargo run --bin mini_databricks_clone\n```\n\n## What this repository contains\n\n- `mini_pg_like.py`: PostgreSQL-like toy engine with heap table, B-tree index, and planner output.\n- `mini_databricks_clone.py`: Databricks-like toy platform with versioning, partitions, DAGs, and events.\n- `src/bin/mini_pg_like.rs`: Rust PostgreSQL-like demo.\n- `src/bin/mini_databricks_clone.rs`: Rust Databricks-like demo.\n- `src/lib.rs`, `src/common.rs`, `src/pg.rs`: shared Rust core modules.\n- `mcp_engine_server.py`: MCP runtime adapter for diagnostics and regression workflows.\n\n## Why this exists\n\nMost internals content stops at diagrams. This project stays runnable and inspectable:\n\n- compare Python and Rust implementations of the same system ideas,\n- trace write-path behavior with concrete events and state transitions,\n- run explainable regression checks through MCP,\n- turn runtime operations into a discoverable control surface for AI clients.\n\n## MCP Adapter Layer\n\nThis repo also includes a minimal MCP server that wraps the lab operations:\n\n- `mcp_engine_server.py`\n- Cursor config: `.cursor/mcp.json`\n\nCurrent MCP tool list for this release (`47` tools total):\n\nEngine state and runtime:\n- `init_engine`\n- `insert_row`\n- `upsert_row`\n- `create_index`\n- `explain_customer`\n- `reindex_project`\n- `run_e2e_flow`\n\nExplainability and demos:\n- `explain_run`\n- `demo_explain_run`\n- `demo_explain_run_failure`\n- `demo_explain_semantic_failure`\n- `demo_explain_idempotency_conflict`\n- `demo_explain_concurrency_failure_storm`\n- `explain_regression_suite`\n\nTrace and retrieval:\n- `record_tool_trace`\n- `similar_incidents`\n- `refresh_trace_path`\n- `refresh_docs_path`\n- `memory_upsert`\n- `memory_search`\n\nSLO and ROI:\n- `health_check`\n- `benchmark_calls`\n- `scenario_load_test`\n- `capture_roi_baseline`\n- `report_drift_bug`\n- `decision_gate`\n\nSchema evaluation (verdict + report via MCP, no external Postgres):\n- `schema_load_tool`\n- `schema_explain_tool`\n- `schema_evaluate_tool`\n- `schema_evaluate_full_tool`\n\nProject contract and regression:\n- `project_manifest`\n- `project_capabilities`\n- `project_tool_catalog`\n- `project_get_defaults`\n- `project_run_regression`\n- `project_capture_baseline`\n- `project_compare_baseline`\n\nGeneric project state:\n- `project_list_entities`\n- `project_get_entity`\n- `project_upsert_entity`\n- `project_delete_entity`\n- `project_append_event`\n- `project_ingest_trace`\n- `project_explain_run`\n- `project_export_state`\n\nGeneric heuristics:\n- `project_list_heuristics`\n- `project_run_heuristic`\n\nFor machine-readable discovery, prefer:\n- `project_tool_catalog`\n- `project_get_defaults`\n\nCurrent heuristic profiles available through `project_run_heuristic`:\n- `pain_structure`\n- `naive_bias`\n- `price_distribution`\n- `liquidity_signals`\n- `price_liquidity_matrix`\n- `cross_category`\n- `sale_format`\n- `speed_signals`\n- `trust_signals`\n\nIf Cursor MCP auto-discovery is enabled, restart Cursor and connect `mini-data-engine`.\nDefault MCP runtime data paths are under `tests/artifacts/mcp/*`.\n\n### Cursor approval setup (reduce repeated prompts)\n\nIf Cursor keeps asking for MCP or command approval on every call, apply this once:\n\n1. Enable workspace trust in Cursor user settings:\n\n```json\n\"security.workspace.trust.enabled\": true\n```\n\n2. In Cursor, open `Settings -> Agents -> Auto-Run` and set:\n   - `Auto-run mode`: `Run in Sandbox`\n   - `MCP Allowlist`: add `mini-data-engine` tools you use often\n   - `Command Allowlist`: add frequently used safe commands\n\n3. Keep this repo opened as the same trusted workspace and reload the window once.\n\nNotes:\n- MCP server approval and per-tool allowlist behavior are enforced by Cursor security settings.\n- In some Cursor versions, allowlist behavior can be best-effort and still prompt in edge cases.\n\nFastest way to see the new explainability use case in action through MCP:\n\n```text\ndemo_explain_run\n```\n\nThat single tool call creates a traced run, records step-level events under one `run_id`,\nand returns an explanation with:\n\n- ordered timeline\n- tool path\n- total elapsed time\n- failure summary if anything breaks\n\nYou can then replay the same explanation directly with:\n\n```text\nexplain_run(run_id=\"...\")\n```\n\n## Explain Regression Suite\n\nUse `explain_regression_suite` when you want regression checks to run through MCP and come back as explainable run summaries instead of isolated test output.\n\nThe suite drives the current validation surface through the MCP layer, attaches `run_id` traces, and returns explain output for each check so regressions can be inspected with the same mechanism used for runtime incidents.\n\nIt currently runs:\n\n- Python unit tests via `python -m unittest discover`\n- Rust tests via `cargo test`, including the current `engine_cli` integration tests\n- `health_check`\n- `benchmark_calls`\n- `scenario_load_test`\n- explainability control demos\n\nThe explainability demos intentionally include both positive and negative controls:\n\n- `demo_explain_run` as `expected_success`\n- `demo_explain_run_failure` as `expected_failure`\n- `demo_explain_semantic_failure` as `expected_failure`\n- `demo_explain_idempotency_conflict` as `expected_failure`\n- `demo_explain_concurrency_failure_storm` as `expected_failure`\n\nThat means the suite is not only checking that the happy path stays green. It also checks that the explain layer still classifies and summarizes known failure classes correctly.\n\nThe current regression surface covers:\n\n- happy-path traced execution\n- runtime/path failures\n- semantic data validation failures\n- idempotency conflict failures\n- concurrency and failure-storm control scenarios\n- sampled benchmark and scenario SLO regressions\n\nFastest MCP call for the full regression bundle:\n\n```text\nexplain_regression_suite\n```\n\nUse it as the top-level MCP regression entrypoint when you want one answer that includes:\n\n- which checks passed\n- which failures were expected controls\n- explain summaries for each traced run\n- early signals that a latency or behavior regression appeared\n\nThe MCP layer is an access interface, not the core product idea. The core of the repository is the runnable lab itself.\n\nProduct note:\n- [`PRODUCT_NOTE_RUNTIME_EXPLAINABILITY.md`](/Users/ll/Documents/data-engineering-runtime-lab/PRODUCT_NOTE_RUNTIME_EXPLAINABILITY.md)\n  Short note describing the runtime explainability use case, the required signals, and the `explain_run` MVP.\n- [`PRODUCT_NOTE_RUNTIME_COPILOT.md`](/Users/ll/Documents/data-engineering-runtime-lab/PRODUCT_NOTE_RUNTIME_COPILOT.md)\n  Product framing for `Runtime Copilot` as an MCP-native operational brain.\n- [`EXPLAIN_REGRESSION_SUITE_FEASIBILITY.md`](/Users/ll/Documents/data-engineering-runtime-lab/EXPLAIN_REGRESSION_SUITE_FEASIBILITY.md)\n  Short article describing what this repository validated about explain-first regression suites and where the current denominator still stays narrow.\n\nUse in Codex:\n- skill package: [`codex/skills/runtime-copilot/SKILL.md`](/Users/ll/Documents/data-engineering-runtime-lab/codex/skills/runtime-copilot/SKILL.md)\n- automation examples: [`codex/automations`](/Users/ll/Documents/data-engineering-runtime-lab/codex/automations)\n- guide: [`docs/use-in-codex.md`](/Users/ll/Documents/data-engineering-runtime-lab/docs/use-in-codex.md)\n\nRun persistent engine CLI (productization path):\n\n```bash\n# Initialize storage\ncargo run --bin engine_cli -- init ./tests/artifacts/engine/data orders\n\n# Insert and upsert (WAL append)\ncargo run --bin engine_cli -- insert ./tests/artifacts/engine/data orders 1 4242 50\ncargo run --bin engine_cli -- upsert ./tests/artifacts/engine/data orders 1 4242 55\n\n# Build index and explain\ncargo run --bin engine_cli -- index ./tests/artifacts/engine/data orders\ncargo run --bin engine_cli -- explain ./tests/artifacts/engine/data orders 4242\n\n# Write snapshot and truncate WAL\ncargo run --bin engine_cli -- checkpoint ./tests/artifacts/engine/data orders\n\n# Transaction simulation: begin/commit/rollback semantics,\n# per-table write lock, snapshot read, and conflict detection\ncargo run --bin engine_cli -- tx-demo ./tests/artifacts/engine/data orders\n\n# Crash/restart recovery for transaction journals\ncargo run --bin engine_cli -- tx-recovery-list ./tests/artifacts/engine/data orders\ncargo run --bin engine_cli -- tx-recovery-commit ./tests/artifacts/engine/data orders <tx_id>\ncargo run --bin engine_cli -- tx-recovery-rollback ./tests/artifacts/engine/data orders <tx_id>\n```\n\n## What You Should See\n\n- In `mini_pg_like.py`: selective predicate switches to `Index Scan`; non-selective stays `Seq Scan`.\n- In `mini_databricks_clone.py`: layer-by-layer demo output, workflow DAG order/metrics, and canonical events count from the single write path.\n- In `mini_pg_like` (Rust): same planner behavior with shared core modules.\n- In `mini_databricks_clone` (Rust): same layered demo using shared Rust library code.\n- In `engine_cli` (Rust): persistent snapshot + WAL replay flow with simple operational commands.\n- In `engine_cli tx-demo`: explicit transaction scopes, snapshot reads, per-table write lock, and concurrent upsert conflict detection.\n- In `engine_cli tx-recovery-*`: staged transaction operations survive process restarts via per-transaction journal files and can be committed or rolled back explicitly.\n- In `e2e_flow`: one command runs write path, checkpoint, bronze->silver transform, planner explain, and DuckDB SQL validation on persisted data.\n\n## Technical Design Backbone\n\n[`TECHNICAL_DESIGN_GENERIC.md`](./TECHNICAL_DESIGN_GENERIC.md) captures the architectural discipline behind the code:\n\n- cross-layer reasoning (`Idea -> API -> Runtime -> Storage -> Perf`),\n- deterministic state transitions,\n- event-first design,\n- adapter contracts,\n- DAG-driven orchestration,\n- measurable go/no-go criteria.\n\nIt is not a separate product claim. It is the review and implementation spine used across the lab.\n\n## Docker package\n\n### Use the **local** build (recommended for development)\n\nBuild the image from this repo so MCP uses your local code (including schema tools) instead of the GitHub image:\n\n```bash\n./scripts/docker-build-local.sh\n```\n\nThis builds `mini-data-engine:local`. To drive **another project** (e.g. threads) with this MCP, set Cursor MCP to use the local image and mount that project as workspace:\n\n- Copy [.cursor/mcp.docker.local.json](.cursor/mcp.docker.local.json) into your **project’s** `.cursor/mcp.json` (or merge the `mcpServers` entry into your Cursor user config).\n- Open the project you want to drive (e.g. threads). `${workspaceFolder}` will be that project; the container gets `WORKSPACE_ROOT=/workspace` and your project mounted at `/workspace`, so e.g. `schema_path=\"schema.sql\"` resolves to that project’s file.\n\nExample local config (uses `mini-data-engine:local` and mounts current workspace as `/workspace`):\n\n```json\n{\n  \"mcpServers\": {\n    \"mini-data-engine\": {\n      \"command\": \"docker\",\n      \"args\": [\n        \"run\", \"--rm\", \"-i\",\n        \"-e\", \"WORKSPACE_ROOT=/workspace\",\n        \"-v\", \"${workspaceFolder}:/workspace\",\n        \"-v\", \"${workspaceFolder}/tests/artifacts:/app/tests/artifacts\",\n        \"mini-data-engine:local\"\n      ]\n    }\n  }\n}\n```\n\n### Use the published image (GHCR)\n\nImage is published to GHCR:\n\n- `ghcr.io/kroq86/data-engineering-runtime-lab:latest`\n\nPull:\n\n```bash\ndocker pull ghcr.io/kroq86/data-engineering-runtime-lab:latest\n```\n\nUse in Cursor MCP config (example):\n\n```json\n{\n  \"mcpServers\": {\n    \"mini-data-engine\": {\n      \"command\": \"docker\",\n      \"args\": [\n        \"run\",\n        \"--rm\",\n        \"-i\",\n        \"-v\",\n        \"${workspaceFolder}/tests/artifacts:/app/tests/artifacts\",\n        \"ghcr.io/kroq86/data-engineering-runtime-lab:latest\"\n      ]\n    }\n  }\n}\n```\n\n## Loom stack\n\nMCP surface for **[loom-ops](https://github.com/kroq86/loom-ops)** and ops runbooks. Ecosystem: [ECOSYSTEM.md](https://github.com/kroq86/loom-stack/blob/main/docs/ECOSYSTEM.md)\n\n```bash\npip install ops-runtime-mcp\nops-runtime-mcp   # stdio MCP; see docs/use-in-codex.md\n```\n",
  "bytes": 13419,
  "sha": "cbef34e7dee10773ca14fe9055d41eae1e3fbd8fb5521c50287e965d8d8874b5",
  "repo_slug": "kroq86/data-engineering-runtime-lab",
  "fonte": "repo",
  "truncated": false,
  "api": "https://agentalog.com/api/listings/mcp_io_github_kroq86_mini_data_engine_50dc5c45/readme"
}