{ "markdown": "

\n \"Metaxy\n

\n\n

Metaxy

\n\n

\n \"PyPI\n \"Python\n \"PyPI\n \"CI\"\n \"codecov\"\n \"JOSS\n \"Ruff\"\n \"Ty\"\n \"prek\"\n

\n\nMetaxy is a metadata layer for **multimodal Data and ML pipelines**. Metaxy tracks lineage and versioning across complex computational graphs for multimodal datasets. Metaxy can cache every single sample and scale to handle millions of them.\n\nMetaxy manages **metadata** while **data** typically lives elsewhere:\n\n```\n┌─────────────────────────────────┐ ┌─────────────────────────┐\n│ Metadata (Metaxy) │ │ Data (e.g., S3) │\n├──────┬──────────┬──────┬────────┤ │ │\n│ ID │ path │ size │version │ │ 📦 s3://my-bucket/ │\n├──────┼──────────┼──────┼────────┤ │ │\n│ img1 │ s3://... │ 2.1M │a3fdsf │ ────────>│ ├─ img1.jpg │\n│ img2 │ s3://... │ 1.8M │b7e123 │ ────────>│ ├─ img2.jpg │\n└──────┴──────────┴──────┴────────┘ └─────────────────────────┘\n```\n\nThe feature that makes Metaxy stand out is the ability to track **partial data dependencies** and detect **prunable updates** — updates that don't trigger change propagation through certain paths in the dependency graph because they modify fields that aren't dependencies of those downstream features. For example, updating audio upstream of a face recognition step allows pruning the face recognition branch since it only depends on video frames. This problem is specific to multimodal pipelines and doesn't typically emerge in traditional data engineering.\n\nMetaxy's goal is to provide a standard instrument for any kind of multimodal (or purely tabular) **incremental** pipelines, standardizing dependency specification, versioning, partial data dependencies, and manipulations over metadata. Or, in short, to be a universal glue for incremental data pipelines.\n\nMetaxy is very reliable and is fanatically tested across all supported Python versions and platforms [^1].\n\n## Documentation\n\nRead the [docs](https://docs.metaxy.io) to learn more.\n\nMetaxy is published in the [Journal of Open Source Software](https://joss.theoj.org/papers/10.21105/joss.10449).\n\n## Installation\n\nInstall Metaxy from [PyPI](https://pypi.org/project/metaxy/):\n\n```shell\nuv add metaxy\n```\n\n## Using Metaxy\n\nMetaxy is highly pluggable and generally can be used with any kind of incremental pipelines, storage, metadata storage, and dataframe libraries.\n\nMetaxy provides integrations with popular tools such as [Dagster](https://docs.metaxy.io/stable/integrations/orchestration/dagster), [Ray](https://docs.metaxy.io/stable/integrations/compute/ray), [ClickHouse](https://docs.metaxy.io/stable/integrations/metadata-stores/databases/clickhouse), [DeltaLake](https://docs.metaxy.io/stable/integrations/metadata-stores/storage/delta/), [SQLModel](https://docs.metaxy.io/stable/integrations/plugins/sqlmodel/).\n\nThe full list can be found [here](https://docs.metaxy.io/stable/integrations).\n\n## Blog Posts\n\n- [Announcement post](https://anam.ai/blog/metaxy) by Anam\n\n- [Dagster + Metaxy](https://dagster.io/blog/building-real-time-interactive-avatars-with-metaxy) by Dagster labs\n\n- [Docling + Slurm + Metaxy](https://georgheiler.com/2026/02/22/metaxy-dagster-slurm-multimodal/) by ASCII\n\n## Contributing\n\nSee [CONTRIBUTING.md](CONTRIBUTING.md).\n\n\n\n\n[^1]: The CLI is not tested on Windows yet.\n", "bytes": 5118, "sha": "016aaae8bd72c67a198ccbdeb8e918a6c249be0c6afb81f2a744b378733a5cbd", "repo_slug": "anam-org/metaxy", "fonte": "repo", "truncated": false, "api": "https://agentalog.com/api/listings/skl_anam_org_metaxy_claude_skills_hypothesis_e4550b49/readme" }