{ "markdown": "
\n\n\"Misata\"\n\n# Misata\n\n**You declare the outcome. Misata generates the data that provably matches it.**\n\nRealistic, relational rows that hit exact revenue curves, fraud rates, referential integrity, and statistical structure. From a sentence, YAML, or your database. No real data, no ML model.\n\n[![PyPI version](https://img.shields.io/pypi/v/misata.svg?style=flat-square&color=E89030)](https://pypi.org/project/misata/)\n[![Python versions](https://img.shields.io/pypi/pyversions/misata.svg?style=flat-square)](https://pypi.org/project/misata/)\n[![CI](https://img.shields.io/github/actions/workflow/status/rasinmuhammed/misata/ci.yml?branch=main&style=flat-square&label=tests)](https://github.com/rasinmuhammed/misata/actions)\n[![License](https://img.shields.io/github/license/rasinmuhammed/misata.svg?style=flat-square)](LICENSE)\n[![Open in Colab](https://img.shields.io/badge/Open%20in-Colab-F9AB00?style=flat-square&logo=googlecolab&logoColor=white)](https://colab.research.google.com/github/rasinmuhammed/misata/blob/main/notebooks/quickstart.ipynb)\n[![Paper](https://img.shields.io/badge/arXiv-2606.08736-b31b1b?style=flat-square&logo=arxiv&logoColor=white)](https://arxiv.org/abs/2606.08736v1)\n[![HF Paper](https://img.shields.io/badge/%F0%9F%A4%97%20HF%20Papers-2606.08736-yellow?style=flat-square)](https://huggingface.co/papers/2606.08736)\n[![smithery badge](https://smithery.ai/badge/misata/misata)](https://smithery.ai/servers/misata/misata)\n[![Misata Studio](https://img.shields.io/badge/Studio-no--code%20in%20your%20browser-E89030?style=flat-square)](https://misata.studio)\n\n**Prefer no code?** Try [**Misata Studio**](https://misata.studio), the no-code synthetic data generator: design a schema on a canvas or describe your dataset in plain English, then generate it in your browser. Same engine, same integrity proof.\n\n
\n\n\n\n---\n\nMost synthetic-data tools learn from a real dataset and imitate it. Misata works the other way: you **declare the outcome you want**: \"monthly revenue rises from \\$50k to \\$200k,\" \"fraud is 3% in Q1 rising to 8% by Q4,\" \"every customer's `total_spent` equals the sum of their orders\", and Misata generates individual rows whose aggregates hit those targets **exactly**, with full referential integrity, from no source data at all.\n\nThis is *outcome-conformant generation*. The mechanism is formalised in an arXiv preprint ([2606.08736](https://arxiv.org/abs/2606.08736v1)): a closed-form method that satisfies declared aggregates to \\$0.00 error, where off-the-shelf imitation synthesisers trained on the same data miss by 74–86%. Every run can also emit an **Oracle report**, a proof bundle covering referential integrity, constraints, temporal consistency, and reproducibility.\n\nIt generates from a plain-English description, a YAML schema, or an existing database schema. No machine-learning model is required. No real data is needed.\n\nBuilt for:\n- **Known-answer testing**: declare the KPI, generate the data, then assert your dbt, Spark, or SQL transform returns exactly that number. A pipeline test with a ground truth, before any real data exists\n- **Database seeding**: fill dev and staging environments with production-like data\n- **Integration tests**: relational fixtures with FK integrity across every table\n- **Demos and prototypes**: realistic numbers, names, and distributions, no PII\n- **BI and dashboard development**: data shaped like your real domain before launch\n- **Statistical method validation**: longitudinal, grouped, and multi-site datasets that pass mixed-effects models, ICC tests, and autocorrelation checks\n\n---\n\n## Declare or mimic: two ways in\n\nMisata works in two modes, and the difference is the whole point:\n\n- **Declare (the default, no data required).** You state the schema and the outcomes you want, exact revenue curves, fraud rates, rollups, constraints, and Misata generates rows from scratch that conform to them. Use this when you do not have real data, or when you need a *known answer* to test a pipeline, dashboard, or demo against.\n- **Mimic (when you already have data).** Point `misata.mimic()` at a real CSV and get a synthetic twin that matches its distributions and correlations but contains none of the original rows, with `fidelity_report` and `privacy_report` to measure the result. Use this for privacy-safe copies of data you already hold.\n\nMost synthetic-data tools only do the second, learning from a real dataset and imitating it. Misata leads with the first: you declare the answer, then generate the data around it.\n\n---\n\n## Research\n\nMisata's exact-aggregate engine is backed by an arXiv preprint:\n\n> **Declarative Outcome-Conformant Synthesis: Exact, Closed-Form Specification Satisfaction and a Conformance Benchmark** \n> Muhammed Rasin, arXiv:2606.08736 (2026) \n> [https://arxiv.org/abs/2606.08736v1](https://arxiv.org/abs/2606.08736v1)\n\nThe paper formalises the core claim: when you declare `\"SaaS MRR from $50k in January to $200k in December\"`, Misata generates individual transactions whose monthly totals match the declared curve **to exactly $0.00 error**, not approximately, but provably, via a closed-form Gamma conditional-sum mechanism (Lukacs' characterisation). Off-the-shelf imitation synthesisers trained on the very same data miss the declared monthly aggregate by 74–86%; Misata reaches exactly 0.\n\nThe paper also introduces **SpecBench**: the first benchmark measuring conformance to analytical outcomes for cold-start relational synthesis. Misata is the reference implementation.\n\n```bibtex\n@article{rasin2026declarative,\n title = {Declarative Outcome-Conformant Synthesis: Exact, Closed-Form\n Specification Satisfaction and a Conformance Benchmark},\n author = {Rasin, Muhammed},\n year = {2026},\n url = {https://arxiv.org/abs/2606.08736v1}\n}\n```\n\n---\n\n## Install\n\n```bash\npip install misata\n```\n\nOptional extras:\n\n```bash\npip install \"misata[llm]\" # multi-provider LLM schema generation\npip install \"misata[documents]\" # PDF output via weasyprint\npip install \"misata[advanced]\" # SDV/CTGAN statistical synthesis\npip install \"misata[mcp]\" # MCP server, expose Misata to Claude, Cursor, and other AI agents\npip install \"misata[evalpack]\" # evalpacks: verified eval databases for data agents (DuckDB)\n```\n\n### Use it from a coding agent\n\nMisata ships an Agent Skill, so Claude Code and anything else that reads\n`SKILL.md` knows which entry point fits which request and what is worth\ndeclaring:\n\n```\n/plugin marketplace add rasinmuhammed/misata\n/plugin install misata@misata\n```\n\nThe skill drives the CLI, so `pip install misata` is still required. There is\nalso an MCP server (`pip install \"misata[mcp]\"`) and a Claude Desktop extension\nin [`mcpb/`](mcpb/).\n\n---\n\n## Use Misata from Claude / Cursor / Windsurf (MCP)\n\nMisata ships a built-in [Model Context Protocol](https://modelcontextprotocol.io) server with a clear division of labour: **the AI agent designs the schema, Misata guarantees the math.** Agents are good at knowing that a veterinary clinic needs a `species` column; Misata is good at making 50 000 rows where every foreign key resolves, every roll-up reconciles to the cent, and the same seed reproduces byte-identical output. The primary tool, `generate_from_schema`, accepts the agent's schema dict and returns the data **plus an integrity proof**: per-relationship orphan counts the agent can show you.\n\n**1. Install:**\n\n```bash\npip install \"misata[mcp]\"\n```\n\n**2. Add to Claude Desktop** (`~/Library/Application Support/Claude/claude_desktop_config.json`):\n\n```json\n{\n \"mcpServers\": {\n \"misata\": {\n \"command\": \"misata-mcp\"\n }\n }\n}\n```\n\nRestart Claude Desktop. Then just ask:\n\n> *\"Generate a fintech dataset with 1 000 customers, payments, and a 2% fraud rate.\"*\n\n> *\"Design a clinical-trials database (sites, patients, visits, adverse events) and generate 100k rows.\"*\n\n> *\"I need SaaS data: MRR from $50k in January, doubled by December, with a Q3 slump.\"*\n\nThe agent designs whatever tables the request needs (any domain; it isn't limited to Misata's built-ins), calls Misata, writes CSVs to disk, and reports back with previews and the verified integrity summary. See the [MCP guide](docs/guides/mcp.md) for Cursor/Windsurf/Zed setup and all six available tools.\n\nmcp-name: io.github.rasinmuhammed/misata\n\n---\n\n## Quick start\n\n```bash\nmisata generate \\\n --story \"Brazilian fintech with R$ payments, CPF verification, and 3% fraud\" \\\n --rows 1000 \\\n --output-dir ./demo_data\n\n# Writes CSVs plus:\n# ./demo_data/oracle_report.json\n```\n\n```python\nimport misata\n\n# One sentence → multi-table DataFrame dict\ntables = misata.generate(\"A SaaS company with 5k users, monthly subscriptions, and 20% churn\")\n\nprint(tables[\"users\"].head())\nprint(tables[\"subscriptions\"].head())\n```\n\n```bash\n# Or from the CLI\nmisata generate --story \"A SaaS company with 5k users and 20% churn\" --rows 5000\n```\n\n## Misata Oracle\n\nThe Oracle report is Misata's proof layer. It separates hard guarantees from advisory realism checks so generated data can be trusted in CI, demos, notebooks, and research comparisons.\n\nGuaranteed checks:\n- referential integrity across configured relationships\n- requested row-count fulfillment\n- schema validation and configured constraints\n- deterministic reproducibility when a seed is set\n\nAdvisory checks:\n- quality score and plausibility warnings\n- privacy heuristics\n- schema-vs-output fidelity score\n- locale/domain fit for countries, cities, phone prefixes, and national IDs\n- data-card metadata\n\n```python\nimport misata\n\nschema = misata.parse(\"Brazilian fintech with CPF verification\", rows=1000)\ntables = misata.generate_from_schema(schema)\noracle = misata.build_oracle_report(tables, schema, seed=schema.seed)\n\nprint(oracle[\"passed\"])\nprint(oracle[\"advisory\"][\"locale_domain_fit\"][\"locale\"])\n```\n\n---\n\n## Mimic mode: clone any CSV in one call\n\nPoint `misata.mimic()` at a real dataset and get a synthetic twin that matches every column's distributions but contains none of the original rows. No schema authoring, no config.\n\n```python\nimport pandas as pd\nimport misata\n\nreal = pd.read_csv(\"titanic.csv\")\ntwin = misata.mimic(real, rows=2000, seed=42, table_name=\"passengers\")[\"passengers\"]\n```\n\nThe profiler handles the columns that break other tools:\n\n- **Alphanumeric code columns** (Ticket `\"A/5 21171\"`, Cabin `\"C85\"`, SKUs, reference numbers) are detected by their character-class shape and reproduced structurally, same shapes in the right proportions, entirely new values, zero verbatim leak from the source. They no longer fall through to prose text generation.\n- **Floats keep their cents.** A Fare of `7.25` generates as `7.25`-shaped values. The profiler infers decimal places from the data; semantic quantization (charm pricing) never fires on mimicked columns.\n- **Distributions are fit from the data.** Skewed-positive columns get lognormal; constant columns get a uniform stub; everything else gets normal. Categorical columns with fewer than 50 values carry their real frequencies.\n\n```python\n# Verify: no verbatim rows can leak through\nshared = [c for c in real.columns if c in twin.columns]\noverlap = pd.merge(real[shared].astype(str), twin[shared].astype(str), how=\"inner\")\nassert len(overlap) == 0\n```\n\n---\n\n## Eight ways to generate data\n\n### 1. Plain English, no config required\n\n```python\ntables = misata.generate(\"A fintech startup with 10k customers, fraud rate 3%, and IBAN accounts\")\n```\n\nMisata reads the story, infers domain (fintech), scale (10 000 rows), and column semantics (fraud flag, IBAN format), no schema authoring needed.\n\nA sentence is read by a recogniser that handles a fixed set of phrasings. Anything it cannot turn into a declaration is named in a warning rather than dropped, so you always know what took effect. When a prompt needs to be exact, write it as a spec instead.\n\n### 1b. A structured spec, parsed deterministically\n\nProse that declares tables, row counts, columns and rules is not a story, and guessing at it is the wrong tool. Misata detects that shape and parses it directly, so \"exactly 4000\" means exactly 4000. No model, no inference, no rewriting.\n\n```text\nTable 1: accounts\nRows: exactly 600\nColumns:\n account_id\n company_name\n plan\n seats\n signed_up_on\nplan must only be:\nStarter\nProfessional\nEnterprise\nseats must be 1 to 120\nsigned_up_on must be 2023-01-01 to 2023-12-31\n\nTable 2: invoices\nRows: exactly 3200\nColumns:\n invoice_id\n account_id\n amount\n issued_on\naccount_id must match values from accounts table\namount must be 120 to 8500\nissued_on must be 2024-01-01 to 2024-12-31\n\nRevenue curve on invoices.amount by issued_on:\nJan 180000\nFeb 195000\nMar 210000\n```\n\n```python\ntables = misata.generate_from_schema(misata.parse(open(\"spec.txt\").read()))\n```\n\nWhat the spec guarantees:\n\n| You write | You get |\n|---|---|\n| `Rows: exactly 3200` | 3200 rows, not about 3200 |\n| `x must match values from y table` | a foreign key with zero orphans |\n| `plan must only be:` + a list | those values and no others |\n| `seats must be 1 to 120` | every row inside the bound |\n| `signed_up_on must be 2023-01-01 to 2023-12-31` | dates inside that window |\n| `Revenue curve on t.col by t.date:` | each month lands on its figure to the cent |\n\nRules can sit inside a table's block or in one section at the end; either way they attach by column name. A column ending `_on`, `_at`, `_date` or `_for` is generated as a date. Anything the parser cannot translate is listed back to you, never guessed at.\n\n### 2. YAML schema-as-code, commit it to git\n\n```bash\nmisata init # scaffolds misata.yaml in the current directory\nmisata generate # reads misata.yaml automatically\n```\n\n```yaml\n# misata.yaml\nname: my-app\nseed: 42\n\ntables:\n users:\n rows: 1000\n columns:\n user_id: { type: int, unique: true }\n email: { type: text, text_type: email }\n plan: { type: categorical, choices: [free, pro, enterprise] }\n\n orders:\n rows: 5000\n columns:\n order_id: { type: int, unique: true }\n user_id: { type: foreign_key }\n amount: { type: float, min: 5.0, max: 500.0 }\n\nrelationships:\n - \"users.user_id → orders.user_id\"\n\nconstraints:\n - name: amount_above_cost\n table: orders\n type: inequality\n column_a: amount\n operator: \">\"\n column_b: cost\n```\n\n```python\nschema = misata.load_yaml_schema(\"misata.yaml\")\ntables = misata.generate_from_schema(schema)\n```\n\n### 3. Seed an existing database directly\n\n```python\nfrom misata import schema_from_db, generate_from_schema, seed_database\n\n# Introspect the live schema: no manual column definitions\nschema = schema_from_db(\"postgresql://user:pass@localhost/myapp\")\ntables = generate_from_schema(schema)\n\n# Seed it back: insert order respects FK dependencies automatically\nreport = seed_database(tables, \"postgresql://user:pass@localhost/myapp_dev\")\n# SeedReport: seeded 6 tables, 47,300 rows in 1.2s\n```\n\n```bash\n# One-command workflow\nmisata init --db postgresql://user:pass@localhost/myapp # writes misata.yaml\nmisata generate --db-url postgresql://user:pass@localhost/myapp_dev --db-create\n```\n\nSQLAlchemy models are supported too:\n\n```python\nfrom misata import seed_from_sqlalchemy_models\nfrom myapp.models import Base\n\nreport = seed_from_sqlalchemy_models(Base, db_url=\"sqlite:///test.db\", row_count=500, create_tables=True)\n```\n\n### 4. From a dbt project's own schema.yml\n\n```bash\ncd my-dbt-project && misata dbt-seed\n```\n\nNo story, no config. Misata reads the properties YAML your project already\nhas and generates seed CSVs that satisfy it: `relationships` tests become\nforeign keys with guaranteed integrity, `accepted_values` become the exact\ncategory pools, `unique` and `not_null` become hard constraints, and\n`data_type` plus column-name semantics decide the rest. Then:\n\n```bash\ndbt build # seed + run + test — the tests you already wrote, passing on day zero\n```\n\nBoth the legacy inline test syntax and the dbt 1.9+ `arguments:` nesting are\nunderstood. Tests Misata can't translate (`dbt_utils.*`, custom generics) are\nlisted in the output rather than silently guessed at.\n\n### 5. From a Prisma schema\n\n```bash\ncd my-app && misata prisma-seed\n```\n\nReads the schema.prisma your app already maintains: `@relation` becomes foreign\nkeys with zero orphans, enums become the exact value pools, `@id` and `@unique`\nare honoured, `@@id`/`@@unique` become composite uniqueness, and optional fields\nmay be null. CSVs land in `seed-data/` ready for your seed script.\n\n### 6. Python dict schema\n\n```python\nschema = misata.from_dict_schema({\n \"customers\": {\n \"id\": {\"type\": \"integer\", \"primary_key\": True},\n \"email\": {\"type\": \"email\"},\n \"plan\": {\"type\": \"string\", \"enum\": [\"free\", \"pro\", \"enterprise\"]},\n },\n \"orders\": {\n \"id\": {\"type\": \"integer\", \"primary_key\": True},\n \"customer_id\": {\"type\": \"integer\", \"foreign_key\": {\"table\": \"customers\", \"column\": \"id\"}},\n \"amount\": {\"type\": \"float\", \"min\": 1.0, \"max\": 999.0},\n \"order_date\": {\"type\": \"date\"},\n },\n}, row_count=5_000)\n\ntables = misata.generate_from_schema(schema)\n```\n\n**Declared outcome curves**: add `__outcome_curves__` as a top-level key alongside the table definitions. Generated rows sum to every declared target exactly, to the cent:\n\n```python\nimport pandas as pd\n\nschema = misata.from_dict_schema({\n \"__outcome_curves__\": [{\n \"table\": \"orders\",\n \"column\": \"amount\",\n \"time_column\": \"order_date\",\n \"time_unit\": \"month\",\n \"value_mode\": \"absolute\",\n \"start_date\": \"2024-01-01\",\n \"avg_transaction_value\": 120.0,\n \"curve_points\": [\n {\"month\": 1, \"target_value\": 50_000.0},\n {\"month\": 6, \"target_value\": 110_000.0},\n {\"month\": 12, \"target_value\": 200_000.0},\n ],\n }],\n \"orders\": {\n \"__rows__\": 5000,\n \"order_id\": {\"type\": \"integer\", \"primary_key\": True},\n \"amount\": {\"type\": \"float\", \"min\": 5, \"max\": 500},\n \"order_date\": {\"type\": \"date\"},\n },\n}, seed=42)\n\ntables = misata.generate_from_schema(schema)\nmonthly = (\n tables[\"orders\"]\n .assign(m=pd.to_datetime(tables[\"orders\"][\"order_date\"]).dt.month)\n .groupby(\"m\")[\"amount\"].sum()\n)\nassert abs(monthly[1] - 50_000) < 0.01 # exact\nassert abs(monthly[12] - 200_000) < 0.01 # exact\n```\n\n**Exact group shares**: declare how a measure divides across a categorical column (\"Electronics is 40% of revenue, Home 25%\") with `__group_shares__`. Paired with an outcome curve on the same table and measure, the shares hold to the cent inside every declared period, and the period totals still hold; without a curve, the shares hold over the table total:\n\n```python\nschema = misata.from_dict_schema({\n \"__group_shares__\": [{\n \"table\": \"orders\",\n \"measure\": \"amount\",\n \"group_column\": \"category\",\n \"shares\": {\"Electronics\": 0.4, \"Home\": 0.25, \"Toys\": 0.2, \"Grocery\": 0.15},\n }],\n # ... same orders table and __outcome_curves__ as above,\n # plus a \"category\" enum column\n}, seed=42)\n```\n\nA period with fewer rows than positive-share groups is skipped with a warning rather than silently mangled; see LIMITATIONS.md. `story_audit` verifies the shares in the output, and evalpacks turn each period-group pair into a verified filtered-aggregation question.\n\n**Constraints and correlations**: enforce business rules and inter-column relationships directly in the dict schema:\n\n```python\nschema = misata.from_dict_schema({\n \"patients\": {\n \"__rows__\": 1000,\n \"__constraints__\": [\n # visit must be on or after enrollment: enforced at generation, not post-processing\n {\"type\": \"inequality\", \"column_a\": \"visit_date\",\n \"operator\": \">=\", \"column_b\": \"enroll_date\", \"action\": \"cap\"},\n ],\n \"__correlations__\": [\n # heavier patients tend to have higher blood pressure (r = 0.41)\n {\"col_a\": \"bmi\", \"col_b\": \"systolic_bp\", \"r\": 0.41},\n ],\n \"patient_id\": {\"type\": \"integer\", \"primary_key\": True},\n \"enroll_date\": {\"type\": \"date\"},\n \"visit_date\": {\"type\": \"date\"},\n \"bmi\": {\"type\": \"float\", \"min\": 16, \"max\": 55},\n \"systolic_bp\": {\"type\": \"float\", \"min\": 90, \"max\": 200},\n },\n})\n```\n\n`__rate_curves__` works the same way for per-period rate targets on boolean or categorical columns (fraud rates, churn flags, plan distributions).\n\n### 7. LLM-assisted generation, richer semantics, optional\n\n```python\nfrom misata import LLMSchemaGenerator\n\ngen = LLMSchemaGenerator(provider=\"groq\", model=\"llama-3.3-70b-versatile\") # free tier, fast & reliable\n# gen = LLMSchemaGenerator(provider=\"anthropic\") # Claude\n# gen = LLMSchemaGenerator(provider=\"ollama\", model=\"llama3\") # fully local, no API key\n\nschema = gen.generate_from_story(\n \"A fraud detection dataset, 2% positive rate, FICO scores, transaction velocity features\"\n)\ntables = misata.generate_from_schema(schema)\n```\n\nRequires `pip install \"misata[llm]\"` plus one of `GROQ_API_KEY`, `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, `GOOGLE_API_KEY`.\n\n> **Groq model tip:** `llama-3.3-70b-versatile` is the reliable free-tier default. Larger models (e.g. `openai/gpt-oss-120b`) can return `413 Request too large` on Groq's free tier, so use them only on a paid tier. Whatever the model returns, generation never crashes on an imperfect schema: missing relationships, malformed probabilities, and out-of-range `time_unit`s are repaired automatically.\n\n### 8. Incremental generation, grow a dataset without re-seeding\n\n```python\ntables = misata.generate(\"A fintech company with 1000 customers\", seed=1)\n\n# Add 1 000 more rows: IDs auto-offset, FK integrity maintained across both batches\ntables = misata.generate_more(tables, schema, n=1000, seed=2)\nprint(len(tables[\"customers\"])) # 2000\n```\n\n---\n\n## Realism that survives inspection\n\nSynthetic data rarely fails on the big numbers; it fails on the small tells a reviewer spots in five seconds. Misata kills each tell with a specific, deterministic mechanism. No LLM is involved; everything is seeded and reproducible.\n\n| The tell | The mechanism |\n|:--|:--|\n| `Pablo Müller, Female`: names, genders, and cultures drawn independently | **Joint identity sampling**: `(culture, gender, first, last)` is one draw from culture-keyed pools, with a measured 6% cross-culture intermix (real populations aren't endogamous). Emails derive from the final name. |\n| `appointment_date: 2022-08-29 06:36:12.995319155`: nanosecond precision, 6 AM, a Sunday | **Temporal profiles**: scheduled events snap to 15-minute grids in business hours with weekends damped; signups follow waking-hour rhythms; only machine events (logs, clicks) keep sub-second precision; birth dates are dates. |\n| Every category equally likely | **Zipf–Mandelbrot marginals**: unweighted categoricals follow the rank-frequency power law real statuses, countries, and categories follow, with the dominant value varying per column. Declared probabilities always win. |\n| `Chicago → San Diego, 145.6 km` | **Geographic facts**: distances between named cities are computed (haversine × road circuity) from 289 embedded city coordinates, and travel times follow from distances. Facts, not distributions: so the Oracle can verify them. |\n| A five-star review that reads \"disappointing\", or lorem ipsum | **Grammar microtext**: review text is generated *from* the row's rating by a seeded grammar (1★ reads angry, 5★ reads delighted), a verifiable invariant. Free-text notes come from a business-note grammar. Lorem ipsum cannot reach output. |\n| A 19-minute appointment, a price of $43.27 | **Numeric quantization**: scheduled durations snap to the slot grids calendars actually offer (15/30/45/60), retail prices end in .99/.95/.00, ages are integers. Measured quantities are left alone. |\n| An order shipped before it was placed, by a customer who had not signed up yet | **Lifecycle and causality ordering**: a row's timestamps sort along the real e-commerce/SaaS/logistics lifecycle, and a child row is shifted so it never predates its FK parent, across multi-level chains, preserving the row's own gaps. |\n| `state: cancelled` next to `city: Los Angeles`, a Tokyo row with a US ZIP, +1 phones everywhere | **Address-chain coherence**: city, state, postal format, and phone calling code all agree with the row's country (and a known city carries its exact state), across 14 countries and 8 postal formats. |\n| `is_fraud` true on half the rows, salaries in a symmetric bell, every quantity 1-5 uniform | **Statistical priors knowledge base**: recognised column names draw their real-world shape automatically: J-shaped ratings, Zipf order quantities (60% ones), lognormal salaries, .99 price endings, ~3% rare-event flags. Explicit declarations always win. |\n| An `order_total` that does not equal the sum of its line items | **Cross-table value coherence**: a line item's `unit_price` is copied from the product it references, and an entity-total column rolls up from its line-item child, never double counting a sibling table. |\n\n```python\ntables = misata.generate(\"A hospital with 300 patients, doctors and appointments\", seed=7)\n# patients: Tae-yang Ahn (Male) · Valentina Esposito (Female) · pooja.kapoor@icloud.com\n# appointments: 2023-03-08 14:00:00 · 2022-07-21 09:15:00: 15-min grid, business hours, 2% weekends\n```\n\n---\n\n## The dataset grades itself\n\nEvery coherence class above is also a detector. `story_audit` checks a generated\ndataset against the full invariant catalog: FK orphans, cross-table temporal\ncausality, roll-up agreement, status gating, count and percent bounds, rare-flag\nbase rates, age against birth date, and more. Nothing incoherent ships silently.\n\n```python\ntables = misata.generate_from_schema(schema, verify=True) # warns on any finding\n\nreport = misata.story_audit(tables, schema) # or audit explicitly\nprint(report.summary()) # \"Coherence: clean\" or a scored list of findings\n```\n\nEvery evalpack manifest embeds this verdict alongside its DuckDB answer\ncertificate, so a pack asserts both that its answers are right and that the\ndata telling the story is internally coherent.\n\n## Reproducibility and stability\n\n- **Within a version, generation is deterministic.** The same schema, seed, and\n misata version produce byte-identical tables. Evalpack manifests record the\n version, seed, and a SHA-256 of the spec for exactly this reason.\n- **Across versions, RNG streams may change** when generation improves (they\n did in 0.8.1.29 and 0.8.2). Declared outcomes still hold: aggregates,\n rates, identities, and integrity survive any upgrade; the individual rows may\n differ. Pin the version when you need bit-identical regeneration.\n- **The public API is the documented top-level surface** (`misata.generate`,\n `generate_from_schema`, `story_audit`, `coherence_audit`, `build_evalpack`,\n the schema classes, and the builders). Underscore-prefixed modules and\n functions may change without notice.\n\nWhere the library is expected to fail is documented honestly, boundary by\nboundary, in [LIMITATIONS.md](LIMITATIONS.md). Every entry there started as a\nreproduced defect or a deliberate design refusal.\n\n## Unknown domains: composed, not confabulated\n\nThe 18 built-in domains are templates. For everything else, Misata refuses to fake understanding, and refuses to give up. A compositional synthesizer derives **structure** from your sentence: plural noun phrases become tables, \"80 beekeepers\" binds a row count, and a small archetype lattice (person / asset / place / event / document) provides honest structural columns and foreign-key wiring.\n\n```python\ntables = misata.generate(\n \"A beekeeping cooperative with 12 apiaries, 80 beekeepers, hives, inspections and honey harvests\"\n)\n# beekeepers: beekeeper_id, first_name, last_name, email, joined_at, status\n# inspections: inspection_id, beekeeper_id, apiary_id, hive_id, inspection_date, status\n# → full FK integrity, profiled timestamps, Zipfian statuses: from one sentence, no LLM\n```\n\nWhat it will *not* do is invent domain semantics: unknown entities get structural columns (reference codes, statuses, dates) and the detection report says exactly that, pointing to the two upgrade paths, a schema dict, or an LLM. The same gate also prevents confabulation: a story that only weakly matches a built-in template (one incidental keyword) is composed from its own entities instead of being forced into the wrong template.\n\n---\n\n## Capsules: teach Misata a domain once\n\nA capsule is one shareable JSON file of domain vocabularies (the species, treatments, and model names a domain calls things) with provenance for every list. Intelligence is spent **once**, at creation; generation stays deterministic, offline, and free.\n\n```bash\n# Mine a capsule from example data you already have: no LLM, no key\nmisata capsule create --domain veterinary --from-csv ./samples/ -o vet.capsule.json\nmisata capsule show vet.capsule.json\n```\n\n```python\n# Vocabularies override built-in pools for matching columns\ntables = misata.generate(\"a veterinary clinic with patients and visits\",\n capsule=\"vet.capsule.json\")\n```\n\nCapsules can also be written by an LLM once and reviewed before use (`capsule_from_llm`, BYO key; Groq's free tier works), or written by hand: it's JSON. Because a capsule is a file, it's a community artifact. Share it via git, a gist, or HF datasets.\n\n---\n\n## Localisation\n\nMisata automatically detects the country context from your story and generates statistically accurate data for that locale, the right names, salary distributions, national ID formats, currencies, postcodes, and company naming conventions.\n\n```python\n# Locale is detected automatically: no extra flag needed\ntables = misata.generate(\"German SaaS company in Berlin with 2k enterprise customers\")\n# → names from de_DE Faker pool, salary ~ lognormal(μ=10.71, σ=0.5) ≈ €45k median,\n# postcodes are 5-digit, company names end in GmbH/AG/UG\n\ntables = misata.generate(\"Brazilian fintech with R$ payments and CPF verification, 50k users\")\n# → pt_BR names, salary median ~BRL 33.6k, national IDs match CPF format ###.###.###-##\n\ntables = misata.generate(\"Indian startup in Bangalore with ₹ salary bands and Aadhaar KYC\")\n# → hi_IN names, salary median ~₹350k/yr, national IDs match Aadhaar 12-digit format\n```\n\nForce or override a locale explicitly:\n\n```python\nschema = misata.parse(\"An ecommerce store with 10k orders\")\ntables = misata.generate_from_schema(schema) # defaults to en_US\n\n# CLI\nmisata generate --story \"Ecommerce store\" --locale ja_JP\n```\n\n### 15 built-in locales\n\n| Locale | Country | Currency | Salary median | National ID |\n|:--|:--|:--|--:|:--|\n| `en_US` | United States | USD / $ | $62 000 | SSN `###-##-####` |\n| `en_GB` | United Kingdom | GBP / £ | £34 000 | NIN `AA######A` |\n| `de_DE` | Germany | EUR / € | €45 000 | Steuer-IdNr |\n| `fr_FR` | France | EUR / € | €38 000 | NIR |\n| `pt_BR` | Brazil | BRL / R$ | R$33 600 | CPF `###.###.###-##` |\n| `es_ES` | Spain | EUR / € | €27 000 | NIE |\n| `hi_IN` | India | INR / ₹ | ₹350 000 | Aadhaar `####-####-####` |\n| `ja_JP` | Japan | JPY / ¥ | ¥4 400 000 | My Number |\n| `zh_CN` | China | CNY / ¥ | ¥90 000 | Resident ID |\n| `ar_SA` | Saudi Arabia | SAR | SAR 96 000 | National ID |\n| `ko_KR` | South Korea | KRW / ₩ | ₩42 000 000 | RRN |\n| `nl_NL` | Netherlands | EUR / € | €42 000 | BSN |\n| `it_IT` | Italy | EUR / € | €29 000 | Codice Fiscale |\n| `pl_PL` | Poland | PLN | PLN 72 000 | PESEL |\n| `tr_TR` | Turkey | TRY | TRY 720 000 | TC Kimlik |\n\nEach pack carries real salary distributions (median and lognormal priors), age distributions, top-ranked cities, phone-number prefixes, postcode patterns, company suffixes, and VAT rates, sourced from OECD, World Bank, ILO, and national statistics offices (2023–24 data).\n\n```python\n# Inspect a locale pack directly\npack = misata.get_locale_pack(\"de_DE\")\nprint(pack.salary_median) # 45000\nprint(pack.currency_symbol) # €\nprint(pack.top_cities[:3]) # ['Berlin', 'Hamburg', 'Munich']\nprint(pack.company_suffixes) # ['GmbH', 'AG', 'UG', 'KG', 'e.K.']\n\n# Auto-detect from a story\nlocale = misata.detect_locale(\"South Korean company in Seoul with KRW salaries\")\n# → \"ko_KR\"\n```\n\n---\n\n## Constraints\n\nEnforce business rules that survive every row of generation:\n\n```python\nfrom misata.constraints import (\n InequalityConstraint, # price > cost on every row\n ColumnRangeConstraint, # min_price <= price <= max_price\n RatioConstraint, # 70% free / 30% pro\n UniqueConstraint, # no duplicate (user_id, date) pairs\n SumConstraint, # total_hours per employee per day <= 8\n NotNullConstraint, # no nulls in required columns\n)\n\nc = InequalityConstraint(\"price\", \">\", \"cost\")\ndf = c.apply(df)\n```\n\nConstraints can also be declared in `misata.yaml`, they run at generation time, not as a post-processing step.\n\n---\n\n## Cross-table roll-ups\n\nMake parent summary columns reconcile with child rows, so the data survives a `GROUP BY ... JOIN`. A `customers.total_spent` column generated independently of that customer's actual orders is a giveaway that data is fake; a roll-up computes it from the real child rows.\n\n```python\nschema = misata.from_dict_schema({\n \"name\": \"shop\",\n \"tables\": {\n \"customers\": {\n \"rows\": 500,\n \"columns\": {\n \"customer_id\": {\"type\": \"int\", \"unique\": True},\n # total_spent = sum(orders.amount) per customer\n \"total_spent\": {\"type\": \"float\", \"rollup\": {\n \"from_table\": \"orders\", \"fk\": \"customer_id\",\n \"agg\": \"sum\", \"column\": \"amount\"}},\n # completed_spend = sum(amount) where status == \"completed\"\n \"completed_spend\": {\"type\": \"float\", \"rollup\": {\n \"from_table\": \"orders\", \"fk\": \"customer_id\", \"agg\": \"sum\",\n \"column\": \"amount\", \"where\": {\"status\": \"completed\"}}},\n },\n },\n \"orders\": {\n \"rows\": 3000,\n \"columns\": {\n \"order_id\": {\"type\": \"int\", \"unique\": True},\n \"customer_id\": {\"type\": \"foreign_key\", \"references\": \"customers.customer_id\"},\n \"amount\": {\"type\": \"float\", \"distribution\": \"lognormal\", \"mu\": 4, \"sigma\": 0.5, \"min\": 1},\n \"status\": {\"type\": \"categorical\", \"choices\": [\"completed\", \"cancelled\", \"pending\"]},\n },\n },\n },\n})\ntables = misata.generate_from_schema(schema)\n# tables[\"customers\"][\"total_spent\"] reconciles exactly with the orders table.\n```\n\nAggregations: `sum`, `count`, `mean`, `max`, `min`. When a parent column name explicitly names a child table (`num_orders`, `total_orders`), the roll-up is inferred automatically with no declaration. Roll-ups survive the `misata.yaml` round-trip and run at generation time.\n\n---\n\n## Statistical realism: data that passes method validation\n\nMost synthetic data tools generate rows independently. That works for database seeding and pipeline tests. It breaks the moment the data needs to pass a statistical method: an autocorrelation test on repeated measurements, a mixed-effects model checking whether groups differ, or an audit that catches values outside plausible bounds.\n\nMisata 0.8.1.0 adds a suite of features that close this gap. All are declared in the same plain dict schema and are reachable from MCP agents, Studio, and direct Python callers.\n\n---\n\n### Stratified distribution profiles: different distributions per subgroup\n\nA realistic A/B test dataset does not draw all users from one conversion distribution. The control group looks different from the treatment group. Use `profiles` to declare this precisely on any column:\n\n```python\nschema = misata.from_dict_schema({\n \"users\": {\n \"__rows__\": 5000,\n \"user_id\": {\"type\": \"integer\", \"primary_key\": True},\n \"cohort\": {\n \"type\": \"string\",\n \"enum\": [\"control\", \"variant_a\", \"variant_b\"],\n \"probabilities\": [0.50, 0.25, 0.25],\n },\n \"session_duration\": {\n \"type\": \"float\",\n \"distribution\": \"lognormal\",\n \"mean\": 180.0, \"std\": 90.0, # fallback for unmatched rows\n \"profiles\": [\n {\"when\": \"cohort == 'control'\", \"distribution\": \"lognormal\", \"mean\": 180.0, \"std\": 90.0},\n {\"when\": \"cohort == 'variant_a'\", \"distribution\": \"lognormal\", \"mean\": 240.0, \"std\": 100.0},\n {\"when\": \"cohort == 'variant_b'\", \"distribution\": \"lognormal\", \"mean\": 310.0, \"std\": 120.0},\n ],\n },\n }\n})\n```\n\nThe `when` expression is evaluated as a pandas query against already-generated columns in the same batch. Rows that match no profile get the column's top-level distribution. Profiles can reference any column generated before the current one in declaration order.\n\n---\n\n### Informative missingness: MAR and MNAR\n\nReal-world datasets have non-random missing values. Misata models both mechanisms:\n\n**Missing At Random (MAR):** The probability of a value being missing depends on an observed column. High-spending users are more likely to skip the optional income field.\n\n```python\n\"annual_income\": {\n \"type\": \"float\",\n \"nullable\": True,\n \"missing_if\": {\n \"predictor\": \"total_spend\",\n \"relationship\": \"higher_increases_probability\",\n \"base_rate\": 0.05,\n \"max_rate\": 0.40,\n \"mechanism\": \"MAR\",\n },\n}\n```\n\n**Missing Not At Random (MNAR):** The probability of a value being missing depends on the value itself. Very low satisfaction scores are the ones most likely to go unreported.\n\n```python\n\"satisfaction_score\": {\n \"type\": \"float\",\n \"distribution\": \"normal\", \"mean\": 7.5, \"std\": 1.8,\n \"nullable\": True,\n \"missing_if\": {\n \"predictor\": \"satisfaction_score\", # references its own column\n \"mechanism\": \"MNAR\",\n \"relationship\": \"lower_increases_probability\",\n \"base_rate\": 0.02,\n \"max_rate\": 0.50,\n },\n}\n```\n\n**Conditional nulls** (`null_when`): Null a column whenever a boolean expression is true.\n\n```python\n\"cancellation_reason\": {\n \"type\": \"string\",\n \"enum\": [\"price\", \"competitor\", \"unused\", \"other\"],\n \"nullable\": True,\n \"null_when\": \"churned == False\",\n}\n```\n\n---\n\n### Exact incidence control: precise rates, not statistical approximations\n\nA `boolean` column with `probability: 0.03` gives approximately 3% True values across many runs. If you need the dataset to contain exactly 3% (auditable against its own spec) use `exact_incidence`:\n\n```python\n\"is_fraud\": {\n \"type\": \"boolean\",\n \"exact_incidence\": {\n \"mode\": \"exact\",\n \"rate\": 0.03, # exactly floor(n * 0.03) rows are True\n },\n}\n```\n\nPer-segment exact rates work the same way:\n\n```python\n\"converted\": {\n \"type\": \"boolean\",\n \"exact_incidence\": {\n \"mode\": \"exact\",\n \"group_by\": \"cohort\",\n \"rates\": {\"control\": 0.12, \"variant_a\": 0.18, \"variant_b\": 0.24},\n },\n}\n```\n\nThe difference between \"approximately 3% fraud\" and \"exactly 3% fraud\" is the difference between a dataset that passes an audit and one that does not.\n\n---\n\n### Within-entity time-series autocorrelation: longitudinal data that passes statistical tests\n\nWithout autocorrelation, a longitudinal dataset (user sessions, IoT readings, financial time series) is statistically identical to a cross-sectional one. Every time-series test (Ljung-Box, Durbin-Watson, autocorrelation plot) will immediately detect that rows are independent and the data is synthetic.\n\nThe `time_series` spec re-writes a column to have real within-entity autocorrelation:\n\n```python\n\"daily_revenue\": {\n \"type\": \"float\",\n \"distribution\": \"lognormal\", \"mean\": 8500.0, \"std\": 3000.0,\n \"time_series\": {\n \"entity_id\": \"store_id\", # one process per store\n \"order_by\": \"day_number\",\n \"model\": \"AR1\", # AR1 | LINEAR_TREND | RANDOM_WALK | MEAN_REVERSION\n \"phi\": 0.72, # autocorrelation coefficient (0 = independent, 1 = random walk)\n \"noise_std\": 800.0,\n \"trend\": {\n \"slope_mean\": 45.0, # average daily growth per store\n \"slope_std\": 12.0, # per-store growth variability\n },\n },\n}\n```\n\nFour models are available:\n\n| Model | Use case |\n|:--|:--|\n| `AR1` | Measurements that persist between periods: revenue, active users, inventory |\n| `LINEAR_TREND` | KPIs with a declared direction: growth, decay, weight loss, skill improvement |\n| `RANDOM_WALK` | Asset prices, exchange rates, any mean-free Brownian process |\n| `MEAN_REVERSION` | Bounded metrics that pull back toward average: NPS, inventory fill rate |\n\n---\n\n### Per-entity anchored distributions: separating within-entity and between-entity variation\n\nWhen a child table's column should be anchored to its parent entity's value, use a formula in `distribution.mean`:\n\n```python\n\"stores\": {\n \"__rows__\": 50,\n \"store_id\": {\"type\": \"integer\", \"primary_key\": True},\n \"baseline_daily_revenue\": {\"type\": \"float\", \"distribution\": \"lognormal\", \"mean\": 8500.0, \"std\": 3000.0},\n},\n\"daily_sales\": {\n \"__rows__\": 18250, # 50 stores × 365 days\n \"record_id\": {\"type\": \"integer\", \"primary_key\": True},\n \"store_id\": {\"type\": \"integer\", \"foreign_key\": {\"table\": \"stores\", \"column\": \"store_id\"}},\n \"revenue\": {\n \"type\": \"float\",\n \"distribution\": \"normal\",\n \"mean\": {\"formula\": \"@stores.baseline_daily_revenue\"}, # anchored to each store's baseline\n \"std\": 800.0, # day-to-day noise\n },\n}\n```\n\nThe engine resolves the FK for every row and draws from that entity's personalised distribution. Between-store variation comes from the spread of `baseline_daily_revenue`; within-store day-to-day noise is `std: 800`. Generating all rows from one shared distribution (as every column-independent generator does) collapses between-entity and within-entity variance into a single number and fails every random-effects test.\n\n---\n\n### Hierarchical ICC cluster effects: group structure that survives statistical tests\n\nWhen rows are grouped under parent entities (stores, regions, branches), observations within the same group tend to look more alike than observations across groups. This within-group homogeneity (the intraclass correlation coefficient (ICC)) is a defining feature of grouped data. Without it, all groups look statistically identical.\n\n`__cluster_effect__` is declared on the **parent** table and applies per-entity random intercepts to columns in the **child** table:\n\n```python\n\"regions\": {\n \"__rows__\": 8,\n \"__cluster_effect__\": {\n \"affects_table\": \"stores\",\n \"affects_columns\": {\n \"avg_order_value\": {\n \"icc\": 0.22, # target intraclass correlation\n \"sd_total\": 45.0, # sd_between = sqrt(0.22) * 45 ≈ 21\n },\n \"conversion_rate\": {\n \"sd_between\": 0.04, # supply sd_between directly\n },\n },\n },\n \"region_id\": {\"type\": \"integer\", \"primary_key\": True},\n \"name\": {\"type\": \"string\", \"enum\": [\"North\", \"South\", \"East\", \"West\", \"Central\", \"NW\", \"NE\", \"SE\"]},\n}\n```\n\nOne random intercept is drawn per parent entity from N(0, sd_between) and added to every child row in that group. The marginal distribution across all rows is preserved. Typical ICC values: 0.05–0.20 for store-level retail metrics, 0.10–0.30 for educational outcomes across schools, 0.15–0.40 for branch-level banking metrics.\n\n---\n\n### Full correlation matrix: declare the complete covariance structure at once\n\nFor tables with many correlated columns, the matrix syntax is cleaner than a list of pairs:\n\n```python\n\"__correlations__\": {\n \"matrix\": {\n \"columns\": [\"session_duration\", \"pages_viewed\", \"revenue\", \"satisfaction\"],\n \"values\": {\n \"session_duration\": [1.00, 0.71, 0.55, 0.32],\n \"pages_viewed\": [0.71, 1.00, 0.48, 0.28],\n \"revenue\": [0.55, 0.48, 1.00, 0.41],\n \"satisfaction\": [0.32, 0.28, 0.41, 1.00],\n }\n }\n}\n```\n\nThe matrix is expanded into pairwise pairs and enforced via Iman-Conover rank reordering, which hits declared Pearson r values while preserving each column's marginal distribution exactly. Pairwise list syntax still works unchanged.\n\n---\n\n### State machine terminal states: process-correct categorical columns\n\nAny column that represents an entity's position in a process (customer lifecycle stage, order fulfilment state, subscription status) should follow a Markov chain, not a flat probability. `__state_machine__` generates the correct terminal distribution:\n\n```python\n\"orders\": {\n \"__state_machine__\": {\n \"state_column\": \"status\",\n \"initial_state\": \"placed\",\n \"transitions\": {\n \"placed\": {\"confirmed\": 0.95, \"cancelled\": 0.05},\n \"confirmed\": {\"shipped\": 0.92, \"cancelled\": 0.08},\n \"shipped\": {\"delivered\": 0.97, \"returned\": 0.03},\n },\n },\n ...\n}\n```\n\nStates with no outgoing transitions are terminal. The engine traverses the chain per row until a terminal state is reached. Declared transition probabilities are preserved in expectation. Works alongside exact incidence, profiles, correlations, and time series in the same table.\n\n---\n\n### Data validation: catch out-of-bounds values before they reach your pipeline\n\nAfter generation, validate against declared domain bounds before the data reaches a model or a dashboard:\n\n```python\ntables = misata.generate_from_schema(schema)\n\nreport = misata.validate_domain(tables, domain=\"financial\")\nprint(report.summary())\n# Domain validation (financial): 0 errors, 0 warnings.\n\nassert report.passed\n```\n\nBuilt-in ranges for `financial` / `fintech`: price ≥ 0, discount 0–1, rate –1 to 100, salary ≥ 0. Column matching is by substring on the lowercased column name, `\"unit_price\"` matches the `price` rule.\n\nAdd custom ranges via the `custom_ranges` dict for any column type. Declare `\"__domain__\": \"financial\"` in the dict schema to attach the domain to the `SchemaConfig` for downstream tooling.\n\n---\n\n## Export\n\n```python\n# Columnar / analytical\nmisata.to_parquet(tables, \"data/\")\nmisata.to_arrow(tables, \"data/\") # Apache Arrow IPC; requires pip install pyarrow\nmisata.to_duckdb(tables, \"data/dataset.duckdb\")\n\n# Row-oriented\nmisata.to_jsonl(tables, \"data/\")\nmisata.to_sql(tables, \"data/\", dialect=\"postgresql\") # CREATE TABLE + INSERT statements\n # dialects: ansi, postgresql, mysql\n```\n\n### Reproducible incremental rows\n\nGenerate additional rows that append cleanly to an existing dataset without ID collisions:\n\n```python\n# Day 1: generate the base dataset\nschema = misata.from_dict_schema({...}, seed=1)\nbase = misata.generate_from_schema(schema)\nfor name, df in base.items():\n df.to_csv(f\"./data/{name}.csv\", index=False)\n\n# Day 2: generate only new rows, PKs offset above existing max\nnew_rows = misata.generate_diff(\n schema,\n existing_dir=\"./data/\",\n new_rows={\"customers\": 200, \"orders\": 1500},\n output_dir=\"./data/delta/\", # optional: write delta CSVs\n)\n```\n\n`generate_diff` reads existing CSVs to find the maximum PK per table and generates new rows with PKs offset above that maximum. Use for streaming pipelines, day-over-day test fixtures, and any workflow where you need to extend a dataset without regenerating it from scratch.\n\n---\n\n## Databricks and Apache Spark\n\nGenerate realistic, referentially-correct test data straight into **Delta Lake**: no\nproduction data required. The `misata.spark` module bridges Misata's pandas output to\nSpark/Delta on Databricks (Free Edition or full), AWS Glue, EMR, or any PySpark 3.3+ cluster.\n\n```python\nimport misata\nfrom misata import spark as mspark\n\nschema = misata.from_dict_schema({\n \"customers\": {\"__rows__\": 500, \"id\": {\"type\": \"integer\", \"primary_key\": True},\n \"email\": {\"type\": \"email\"}, \"country\": {\"type\": \"string\", \"text_type\": \"country\"}},\n \"orders\": {\"__rows__\": 2000, \"id\": {\"type\": \"integer\", \"primary_key\": True},\n \"customer_id\": {\"type\": \"integer\",\n \"foreign_key\": {\"table\": \"customers\", \"column\": \"id\"}},\n \"total\": {\"type\": \"float\", \"distribution\": \"lognormal\", \"mu\": 4.5, \"sigma\": 0.9}},\n})\n\n# One call: generate all tables (FK integrity guaranteed) and write to Delta\nresult = mspark.generate_to_delta(schema, spark, catalog=\"dev\", database=\"bronze\", mode=\"overwrite\")\nprint(result.summary())\n# ✅ customers (500 rows) → dev.bronze.customers\n# ✅ orders (2,000 rows) → dev.bronze.orders\n```\n\n**What it does that `dbldatagen` can't:** multiple related tables in one call, guaranteed\nreferential integrity, realistic distributions, and *outcome conformance*, declare an exact\naggregate or rate (e.g. \"fraud is 1.8% in Jan ramping to 4.1% by Jun\") and the data conforms,\ngiving downstream pipeline tests a **known ground truth to assert against**.\n\n| Function | Purpose |\n|----------|---------|\n| `generate_to_delta(schema, spark, …)` | One-liner: generate + write all tables to Delta |\n| `to_spark(tables, spark, schema_config=…)` | Convert Misata DataFrames to Spark with an explicit, type-correct schema |\n| `write_delta(tables, spark, …)` | Write to Delta with partitioning, **liquid clustering**, table properties, or **`MERGE` upsert** |\n| `verify_delta_integrity(spark, relationships, …)` | Check FK integrity of Delta tables via Spark SQL anti-joins |\n| `from_catalog_schema(spark, database, …)` | Import an existing Unity Catalog schema (structure only) → generate matching data, FKs auto-inferred |\n| `append_to_delta(schema, spark, n_rows=…)` | Append incremental rows with non-colliding PKs |\n| `write_delta_stream(schema, spark, …)` | Stream-write 100M+ row datasets without buffering |\n\nOn **Databricks serverless / Free Edition**, install plain `misata` (PySpark is already on the\ncluster, installing `misata[spark]` would stop a serverless session). On other environments:\n`pip install misata[spark]`.\n\n**End-to-end tutorial:** a complete fraud-detection medallion pipeline (Bronze → Silver → Gold)\ntested entirely on synthetic data, with a CI-grade ground-truth assertion,\n[`examples/databricks/`](./examples/databricks/). Full API reference: [`docs/spark.md`](./docs/spark.md).\n\n---\n\n## Document generation\n\nRender one document per row from any table, useful for demo datasets that need to look real end-to-end:\n\n```python\n# Built-in templates: invoice, patient_report, transaction_receipt, user_profile\npaths = misata.generate_documents(\n tables, \"invoice\", table=\"orders\", output_dir=\"/tmp/invoices\", format=\"html\"\n)\n# format=\"pdf\" requires: pip install \"misata[documents]\"\n\n# Custom Jinja2 template\ntmpl = \"

Order #{{ order_id }}

Amount: ${{ amount }}

\"\npaths = misata.generate_documents(tables, tmpl, table=\"orders\", output_dir=\"/tmp/custom\")\n```\n\n---\n\n## Quality and privacy analysis\n\n```python\nbundle = misata.analyze_generation(tables, schema) # runs privacy, fidelity, data_card\n\nprint(bundle.fidelity.overall_score) # 0–100 statistical fidelity score vs. schema intent\nprint(bundle.fidelity.grade) # letter grade for the same score\nprint(bundle.privacy.overall_risk_score) # heuristic PII / re-identification risk\nprint(bundle.data_card.tables) # per-table row counts and metadata\n```\n\n---\n\n## Evalpacks: eval databases where the answer key cannot be wrong\n\nPublished text-to-SQL benchmarks are built by annotating question/answer pairs on top of an existing database, and that annotation step is where pervasive answer-key errors creep in. An evalpack inverts the order: the ground truth is the declared spec itself (outcome curves, rate curves, FK relationships), Misata generates a database that satisfies it, and every question shipped in the pack is then verified by executing its gold SQL against the written CSV files with DuckDB, an engine that shares no code with the generator. Questions whose observed answer does not exactly match the declared answer are dropped and recorded in the manifest. A wrong answer key is impossible by construction and double-checked by independent execution.\n\n```bash\npip install \"misata[evalpack]\"\nmisata evalpack --config misata.yaml -o ./my_pack --seed 42\n```\n\n```python\nfrom misata.evalpack import build_evalpack\n\nresult = build_evalpack(schema, \"my_pack\")\nassert result.all_verified\n```\n\nEach pack ships the tables as CSVs, `questions.jsonl`, a per-question verification certificate, a manifest with the spec hash and seed, and a standalone `verify.py` that anyone can re-run with nothing but `duckdb` installed. Use it to benchmark SQL agents, RAG-over-database systems, or any tool that claims to answer questions about data, against a database whose right answers are known before a single row exists.\n\n---\n\n## Supported domains\n\n18 built-in domain schemas, each generates a fully relational, multi-table dataset with realistic distributions, FK integrity, and domain-appropriate column semantics.\n\n| Domain | Trigger keywords | Tables generated |\n|:--|:--|:--|\n| SaaS | saas, subscription, mrr, churn | users, subscriptions, invoices |\n| Ecommerce | ecommerce, orders, store, retail | customers, products, orders, order_items |\n| Fintech | fintech, payments, banking, fraud | customers, accounts, transactions |\n| Healthcare | healthcare, patients, doctors, clinic | doctors, patients, appointments |\n| Marketplace | marketplace, sellers, buyers, listings | sellers, buyers, listings, orders |\n| Logistics | logistics, shipping, drivers, routes | drivers, vehicles, routes, shipments |\n| HR | hr, employees, payroll, workforce | departments, employees, payroll |\n| Social | social media, instagram, feed, followers | users, posts, follows, reactions |\n| Real Estate | real estate, housing, mortgage | agents, properties, transactions |\n| Pharma | pharma, clinical, trials | researchers, projects, trials, timesheets |\n| Food Delivery | food delivery, restaurant, takeout | restaurants, customers, couriers, orders, order_items |\n| EdTech | edtech, courses, students, enrollments | instructors, courses, students, enrollments, quiz_attempts |\n| Gaming | gaming, players, leaderboard, esports | players, matches, sessions, achievements |\n| CRM | crm, salesforce, deals, pipeline | companies, contacts, deals, activities |\n| Crypto / Web3 | crypto, blockchain, ethereum, defi | wallets, tokens, transactions, token_prices |\n| Insurance | insurance, policy, claims, premium | customers, policies, claims, payments |\n| Travel | travel, hotel, flights, bookings | users, hotels, flights, bookings, reviews |\n| Streaming | streaming, netflix, subscribers, watch history | subscribers, content, watch_history, ratings |\n\nNo keyword match → the compositional synthesizer builds a structural multi-table schema from your sentence's own entities (see *Unknown domains* above); stories with no entities at all fall back to a generic single table with smart column inference.\n\n---\n\n## How it works\n\n```\nstory / YAML / dict / DB introspection / MCP tool call\n ↓\n StoryParser · compositional synthesizer · locale detection · load_yaml_schema · schema_from_db\n ↓\n DetectionReport (domain, confidence, near_misses, table_preview, warnings)\n ↓\n SchemaConfig ← validate_schema() catches issues before any rows are generated\n ↓\n DataSimulator\n ├─ topological sort (FK dependency order)\n ├─ domain priors → locale priors (salary, age, monetary)\n ├─ constraint engine (inequality, range, ratio, sum, unique)\n ├─ outcome curves (monthly targets from narrative control points)\n ├─ stratified profiles (per-subgroup distributions, pandas eval)\n ├─ AR1 / time-series autocorrelation (per entity, 4 models)\n ├─ state machine (Markov terminal states)\n ├─ ICC cluster effects (per-parent-entity random intercepts)\n ├─ Iman-Conover correlation engine (pairwise + full matrix)\n ├─ MAR / MNAR missingness (predictor-scaled and value-dependent)\n ├─ exact incidence (floor(n × rate), per-group rates)\n ├─ realism core (joint identities, temporal profiles, Zipf marginals,\n │ geo facts, grammar microtext, numeric quantization)\n └─ RealisticTextGenerator (capsules + Faker locale + vocabulary assets)\n ↓\n {table_name: DataFrame}\n ↓\n validate_domain · seed_database · to_parquet · to_arrow\n to_duckdb · to_sql · to_jsonl · generate_documents · MCP CSV output\n```\n\n**Domain priors**: monetary columns get log-normal distributions. Categoricals use Zipf sampling. Blood types, country distributions, and salary bands reflect real-world statistics.\n\n**Locale priors**: salary and age distributions are overridden with country-specific lognormal/normal parameters sourced from national statistics. `\"Brazilian fintech\"` in your story means salaries are sampled from the BRL distribution, not the USD one.\n\n**Outcome curves**: natural-language narrative is parsed into exact monthly control points. Named events, quarters, and multipliers all work:\n\n```python\n# All of these produce precise, shaped outcome curves:\nmisata.generate(\"SaaS mrr from $50k in Jan to $200k in Dec, with a Q3 slump\")\nmisata.generate(\"Ecommerce orders, Black Friday spike, Christmas peak\")\nmisata.generate(\"SaaS startup, MRR 10x growth over the year\")\nmisata.generate(\"Fintech payments, strong Q4, dip in Q1\")\n```\n\n**Realism rules**: `cost` is always less than `price`. `delivered_at` is always after `shipped_at`. `hire_date` is after `date_of_birth` + 18 years and never in the future. `tenure_years` is derived on the same row from `hire_date`. Email addresses derive from first and last name columns, names agree with declared genders, route distances agree with their cities, and review text agrees with its star rating.\n\n---\n\n## What makes Misata different\n\nComparison reflects each tool's documented, out-of-the-box behavior as of late 2025; all\nof these are capable libraries built for different goals, and a \"No\" means \"not a built-in\nfeature,\" not \"impossible.\"\n\n| | Faker | Synth | syda | SDV | **Misata** |\n|:--|:--:|:--:|:--:|:--:|:--:|\n| No config, one line to multi-table data | No | No | No | No | **Yes** |\n| Story auto-detects locale + country stats | No | No | No | No | **Yes** |\n| 18 built-in domain schemas (SaaS → streaming) | No | No | No | No | **Yes** |\n| Narrative curves (Q4 push, Black Friday, 10×) | No | No | No | No | **Yes** |\n| Unknown domains composed from the sentence itself | No | No | No | No | **Yes** |\n| Coherent identities (name ↔ gender ↔ email agree) | No | No | No | No | **Yes** |\n| Review text provably matches its star rating | No | No | No | No | **Yes** |\n| Real city distances on route tables | No | No | No | No | **Yes** |\n| Shareable domain vocabulary capsules | No | No | No | No | **Yes** |\n| Mimic mode: clone distributions from a CSV | No | No | No | **Yes** | **Yes** |\n| Pairwise + full-matrix correlation (Iman-Conover) | No | No | No | **Yes** | **Yes** |\n| Geospatial columns (lat, lng, postal_code) | No | No | No | No | **Yes** |\n| Anomaly injection (per-column outlier rate) | No | No | No | No | **Yes** |\n| MCP server: usable from Claude / Cursor | No | No | No | No | **Yes** |\n| YAML schema committed to git | No | **Yes** | **Yes** | No | **Yes** |\n| JSON Schema validation + editor auto-complete | No | No | No | No | **Yes** |\n| DB introspection → generate → re-seed | No | **Yes** | No | Limited | **Yes** |\n| Direct DB seeding (Postgres / MySQL / SQLite) | No | No | ", "bytes": 60000, "sha": "68acff79edd40909dd6a30fed957e2c42724a6622e34b2d8c111b9271f64b287", "repo_slug": "rasinmuhammed/misata", "fonte": "repo", "truncated": false, "api": "https://agentalog.com/api/listings/mcp_io_github_rasinmuhammed_misata_f6f0b2e3/readme" }