{
  "markdown": "# Iterable Data\n\n<img src=\"docs/static/img/logo.svg\" alt=\"Iterable Data\" width=\"72\" height=\"72\" />\n\n<!-- mcp-name: io.github.datenoio/iterabledata -->\n\nIterable Data is a Python library for reading and writing data files row by row in a consistent, iterator-based interface. It provides a unified API for working with various data formats (CSV, JSON, Parquet, XML, etc.) similar to `csv.DictReader` but supporting many more formats.\n\nThis library simplifies data processing and conversion between formats while preserving complex nested data structures (unlike pandas DataFrames which require flattening).\n\n## Features\n\n- **Unified API**: Single interface for reading/writing multiple data formats\n- **Automatic Format Detection**: Detects file type and compression from filename or content (magic numbers and heuristics)\n- **Format Capability Reporting**: Programmatically query format capabilities (read/write/bulk/totals/streaming/tables)\n- **Support for Compression**: Works seamlessly with compressed files\n- **Preserves Nested Data**: Handles complex nested structures as Python dictionaries\n- **DuckDB Integration**: Optional DuckDB engine for high-performance queries with pushdown optimizations\n- **Pipeline Processing**: Built-in pipeline support for data transformation\n- **Encoding Detection**: Automatic encoding and delimiter detection for text files\n- **Bulk Operations**: Efficient batch reading and writing\n- **Native Batch Conversion**: Opt-in columnar-to-columnar transfers with projection, row-range, and batch-size selection\n- **Bounded Columnar I/O**: Shared row/bulk cursors and configurable Parquet row groups keep large reads and writes bounded\n- **Codec Performance Profiles**: Choose `fast`, `balanced`, or `max` compression settings with effective-setting diagnostics\n- **Table Listing**: Discover available tables, sheets, and datasets in multi-table formats\n- **Context Manager Support**: Use `with` statements for automatic resource cleanup\n- **DataFrame Bridges**: Convert iterable data to Pandas, Polars, and Dask DataFrames with one-liner methods\n- **Cloud Storage Support**: Direct access to S3, GCS, and Azure Blob Storage via URI schemes\n- **Database Engine Support**: Read-only access to SQL and NoSQL databases (PostgreSQL, ClickHouse, MySQL, MongoDB, Elasticsearch, etc.) as iterable data sources\n- **Atomic Writes**: Production-safe file writing with temporary files and atomic renames\n- **Bulk File Conversion**: Convert multiple files at once using glob patterns or directories\n- **Progress Tracking and Metrics**: Built-in progress bars, callbacks, and structured metrics objects\n- **Error Handling Controls**: Configurable error policies and structured error logging; malformed input raises typed errors by default instead of reading as empty datasets\n- **Security Hardening**: XXE-safe XML parsing, AST-whitelisted filter expressions, and explicit pickle trust acknowledgement\n- **Performance Regression Gate**: CI-enforced baselines for representative read/convert workloads\n- **Container Formats**: Stream records from TAR archives without extracting members to disk\n- **Type Hints and Type Safety**: Complete type annotations with typed helper functions for dataclasses and Pydantic models\n- **Lakehouse Tables**: Read and write Delta Lake, Iceberg, and DuckLake; read Hudi; experimental Apache Paimon tables plus Row/Mosaic file formats\n\n## Supported File Types\n\n### Core Formats\n\n- **JSON** - Standard JSON files\n- **JSONL/NDJSON** - JSON Lines format (one JSON object per line)\n- **JSON-LD** - JSON for Linking Data (RDF format)\n- **CSV/TSV** - Comma and tab-separated values\n- **Annotated CSV** - CSV with type annotations and metadata\n- **CSVW** - CSV on the Web (with metadata)\n- **PSV/SSV** - Pipe and semicolon-separated values\n- **LTSV** - Labeled Tab-Separated Values\n- **FWF** - Fixed Width Format\n- **XML** - XML files with configurable tag parsing\n- **ZIP XML** - XML files within ZIP archives\n- **HTML** - HTML files with table extraction\n\n### Binary Formats\n\n- **BSON** - Binary JSON format\n- **MessagePack** - Efficient binary serialization\n- **CBOR** - Concise Binary Object Representation\n- **UBJSON** - Universal Binary JSON\n- **SMILE** - Binary JSON variant\n- **Bencode** - BitTorrent encoding format\n- **Avro** - Apache Avro binary format (read & write)\n- **Pickle** - Python pickle format (untrusted input is unsafe; pass `trust=True` to acknowledge)\n\n### Columnar & Analytics Formats\n\n- **Parquet** - Apache Parquet columnar format\n- **ORC** - Optimized Row Columnar format\n- **Arrow/Feather** - Apache Arrow columnar format\n- **GeoParquet** - GeoParquet metadata-aware Parquet profile with geometry/CRS preservation\n- **Lance** - Modern columnar format optimized for ML and vector search\n- **Vortex** - Modern columnar format with fast random access\n- **Paimon Row** - Apache Paimon row format for O(1) row-number access\n- **Paimon Mosaic** - Apache Paimon columnar-bucket format for wide tables\n- **Paimon** - Apache Paimon warehouse/catalog tables\n- **Delta Lake** - Delta Lake format (read & write)\n- **Iceberg** - Apache Iceberg format (read & write)\n- **DuckLake** - DuckLake lakehouse tables (read & write)\n- **Hudi** - Apache Hudi format (read; writes deferred)\n\n### Database Formats\n\n- **SQLite** - SQLite database files\n- **DBF** - dBase/FoxPro database files\n- **MySQL Dump** - MySQL dump files\n- **PostgreSQL Copy** - PostgreSQL COPY format\n- **DuckDB** - DuckDB database files\n\n### Statistical Formats\n\n- **SAS** - SAS data files\n- **Stata** - Stata data files\n- **SPSS** - SPSS data files\n- **R Data** - R RDS and RData files\n- **fst** - R fst columnar on-disk frames (`fst` extra; experimental)\n- **PX** - PC-Axis format\n- **ARFF** - Attribute-Relation File Format (Weka format)\n- **LIBSVM** - Sparse labeled feature vectors (read & write)\n- **NumPy** - `.npy` / `.npz` array rows (read & write; `npy` extra)\n\n### Scientific Formats\n\n- **NetCDF** - Network Common Data Form for scientific data\n- **CDF** - NASA Common Data Format (space science)\n- **HDF5** - Hierarchical Data Format\n- **Zarr** - Chunked array stores (`zarr` extra; experimental)\n- **XYZ** - Molecular/point coordinate tables\n- **CIF** - Crystallographic Information File (`atom_site` loops; experimental)\n- **PDB** - Protein Data Bank ATOM/HETATM records\n- **MATLAB MAT** - MATLAB `.mat` variables (`mat` extra; experimental)\n- **SEG-Y** - Seismic traces (`geophysical` extra; experimental)\n- **GRIB2** - Meteorological messages (`geophysical` extra; experimental)\n- **miniSEED** - Seismological waveform windows (`geophysical` extra; experimental)\n\n### Geospatial Formats\n\n- **GeoJSON** - Geographic JSON format\n- **GeoJSON Text Sequence** - RFC 8142 line-delimited GeoJSON Features (`.geojsonl`, `.geojsons`); streaming-friendly\n- **GeoPackage** - OGC GeoPackage format\n- **GML** - Geography Markup Language\n- **KML** - Keyhole Markup Language\n- **KMZ** - KML Zipped (ZIP archive containing KML)\n- **GPX** - GPS Exchange Format (waypoints, routes, tracks)\n- **Shapefile** - ESRI Shapefile format\n- **File Geodatabase** - ESRI FileGDB layers via Fiona (`geospatial` extra; experimental)\n- **MapInfo MIF** - MapInfo Interchange Format (`geospatial` extra; experimental)\n- **Esri ASCII Grid** - Raster grids as cell or row records (`.asc`)\n- **ArcInfo E00** - Interchange exports (experimental subset)\n- **LAS** - LiDAR point clouds (`lidar` extra; experimental)\n- **BAG** - Bathymetric Attributed Grid (`hdf5` extra; experimental)\n- **CZML** - Cesium CZML document packets\n- **FlatGeobuf** - Streaming geospatial features with optional spatial-index filtering\n- **MVT/PBF** - Mapbox Vector Tiles\n- **TopoJSON** - Topology-preserving GeoJSON extension\n\n### RDF & Semantic Formats\n\n- **JSON-LD** - JSON for Linking Data\n- **RDF/XML** - RDF in XML format\n- **Turtle** - Terse RDF Triple Language\n- **N-Triples** - Line-based RDF format\n- **N-Quads** - N-Triples with context\n- **TriG** - RDF Triple Graph format\n- **N3** - Notation3 RDF format\n- **TriX** - XML Triple RDF format\n- **HDT** - Header-Dictionary-Triples compact RDF (`rdf` extra; experimental)\n\n### Feed Formats\n\n- **Atom** - Atom Syndication Format\n- **RSS** - Rich Site Summary feed format\n\n### Network Formats\n\n- **PCAP** - Packet Capture format\n- **PCAPNG** - PCAP Next Generation format\n\n### Log & Event Formats\n\n- **Apache Log** - Apache access/error logs\n- **CEF** - Common Event Format\n- **GELF** - Graylog Extended Log Format\n- **WARC** - Web ARChive format\n- **CDX** - Web archive index format\n- **ILP** - InfluxDB Line Protocol\n- **HTML** - HTML files with table extraction\n\n### Email Formats\n\n- **EML** - Email message format\n- **MBOX** - Mailbox format\n- **MHTML** - MIME HTML format\n\n### Configuration Formats\n\n- **INI** - INI configuration files\n- **TOML** - Tom's Obvious Minimal Language\n- **YAML** - YAML Ain't Markup Language\n- **HOCON** - Human-Optimized Config Object Notation\n- **EDN** - Extensible Data Notation\n\n### Office Formats\n\n- **XLS/XLSX** - Microsoft Excel files\n- **XLSB** - Excel Binary format\n- **ODS** - OpenDocument Spreadsheet\n- **Microsoft Access** - Access `.mdb` / `.accdb` tables (`access` extra; experimental)\n- **Lotus 1-2-3** - Legacy WK1 / `.123` spreadsheets (experimental)\n\n### Business & Exchange Formats\n\n- **EDI** - X12 / EDIFACT segment streams (experimental)\n- **IATI** - Aid-transparency activity XML (`xml` extra; experimental)\n\n### CAD Formats\n\n- **DXF** - AutoCAD Drawing Exchange Format\n\n### Graph Formats\n\n- **GraphML** - Graph Markup Language\n- **GEXF** - Graph Exchange XML Format\n- **DOT** - GraphViz DOT format\n\n### Sequence & Alignment Formats\n\n- **FASTA** - Sequence format (protein/nucleotide)\n- **FASTQ** - Sequence with quality format\n- **SAM** - Sequence Alignment/Map (text)\n- **BAM** - Binary SAM format\n- **Genomic VCF/BCF** - Variant Call Format for genomic data (distinct from vCard `.vcf`; requires `bio` extra)\n- **CRAM** - Reference-compressed sequence alignments (requires `alignment` extra and an explicit reference when needed)\n- **BED** - BED3–BED12 genomic intervals\n- **GFF3/GTF** - Genomic feature annotations with coordinate and attribute preservation\n\n### Streaming & Big Data Formats\n\n- **Kafka** - Apache Kafka format\n- **Pulsar** - Apache Pulsar format\n- **Flink** - Apache Flink format\n- **Beam** - Apache Beam format\n- **RecordIO** - RecordIO format\n- **SequenceFile** - Hadoop SequenceFile\n- **TFRecord** - TensorFlow Record format\n- **WebDataset** - TAR shards grouped into ML sample dicts (`format=\"webdataset\"`; experimental)\n\n### Protocol & Serialization Formats\n\n- **Protocol Buffers** - Google Protocol Buffers\n- **Cap'n Proto** - Cap'n Proto serialization\n- **FlatBuffers** - FlatBuffers serialization\n- **FlexBuffers** - FlexBuffers format\n- **Thrift** - Apache Thrift format\n- **ASN.1** - ASN.1 encoding format\n- **Ion** - Amazon Ion format\n- **OTLP JSON/Protobuf** - OpenTelemetry traces, logs, and metrics export profiles (requires `otlp` extra)\n\n### Other Formats\n\n- **TAR** - Multi-file archive container (read-only; streams members without extracting to disk)\n- **vCard (VCF)** - Electronic business cards (RFC 6350); not genomic Variant Call Format\n- **iCal** - iCalendar format\n- **LDIF** - LDAP Data Interchange Format\n- **TXT** - Plain text files\n\nSee the [formats documentation](https://datenoio.github.io/iterabledata/formats/) (or `docs/docs/formats/` in this repo) for per-format parameters, record shapes, and extras.\n\n## Supported Compression Codecs\n\n- **GZip** (.gz)\n- **BZip2** (.bz2)\n- **LZMA** (.xz, .lzma)\n- **LZ4** (.lz4)\n- **ZIP** (.zip)\n- **Brotli** (.br)\n- **ZStandard** (.zst, .zstd)\n- **Snappy** (.snappy, .sz) — streaming decompression for framed files\n- **LZO** (.lzo, .lzop) — streaming decompression with legacy blob fallback\n- **7z** (.7z; requires `py7zr` via `iterabledata[compression]`)\n\n## Requirements\n\nPython 3.10+\n\n## Installation\n\n```bash\npip install iterabledata\n```\n\nThe PyPI package is **iterabledata**. Import **iterable**:\n\n```python\nfrom iterable import open_iterable\n```\n\nOr install from source:\n\n```bash\ngit clone https://github.com/datenoio/iterabledata.git\ncd iterabledata\npip install .\n```\n\n### Optional Dependencies\n\nIterableData supports optional extras for additional features:\n\n```bash\n# AI-powered documentation generation\npip install iterabledata[ai]\n\n# Database ingestion (PostgreSQL, ClickHouse, MongoDB, MySQL, Elasticsearch, etc.)\npip install iterabledata[db]\n\n# RDF formats (TriG, N3, TriX)\npip install iterabledata[rdf]\n\n# Excel Binary (XLSB)\npip install iterabledata[xlsb]\n\n# Graph formats (GraphML, GEXF, DOT)\npip install iterabledata[graph]\n\n# Alignment formats (BAM, SAM, CRAM)\npip install iterabledata[alignment]\n\n# Genomic formats (VCF/BCF, CRAM, BED, GFF3/GTF via pysam and bio readers)\npip install iterabledata[bio]\n\n# Zarr chunked array stores\npip install iterabledata[zarr]\n\n# GeoParquet, FlatGeobuf, FileGDB, MapInfo MIF\npip install iterabledata[parquet,geospatial]\n\n# LiDAR LAS point clouds\npip install iterabledata[lidar]\n\n# MATLAB .mat files\npip install iterabledata[mat]\n\n# SEG-Y, GRIB2, miniSEED\npip install iterabledata[geophysical]\n\n# Microsoft Access (.mdb/.accdb)\npip install iterabledata[access]\n\n# R fst frames (requires a suitable fst/rfst binding)\npip install iterabledata[fst]\n\n# OpenTelemetry JSON and Protobuf export profiles\npip install iterabledata[otlp]\n\n# Lakehouse table formats (Delta, Iceberg, Lance, Hudi, DuckLake)\npip install iterabledata[lakehouse]\n\n# Apache Paimon (tables + Row + Mosaic file formats; separate from lakehouse)\npip install iterabledata[paimon]\n# Or individually:\n# pip install iterabledata[paimon-table]\n# pip install iterabledata[paimon-row]\n# pip install iterabledata[paimon-mosaic]\n# pip install iterabledata[ducklake]\n\n# Individual format extras (one per format family), for example:\npip install iterabledata[avro]     # Apache Avro\npip install iterabledata[npy]      # NumPy .npy/.npz\npip install iterabledata[ods]      # OpenDocument spreadsheets\npip install iterabledata[rdata]    # R RData/RDS\npip install iterabledata[ics]      # iCalendar\n# Also available: ubj, vcf, capnp, thrift, fbs, edn, hocon, der, bencode, ldif, hdf5, xml, rdf\n\n# All optional dependencies\npip install iterabledata[all]\n```\n\n**AI Features** (`[ai]`): Enables AI-powered documentation generation using OpenAI, OpenRouter, Ollama, LMStudio, or Perplexity.\n\n**Database Engines** (`[db]`): Enables read-only database access as iterable data sources. Supports PostgreSQL, ClickHouse, MySQL/MariaDB, Microsoft SQL Server, SQLite, MongoDB, and Elasticsearch/OpenSearch. Includes convenience groups:\n\n- `[db-sql]`: SQL databases only (PostgreSQL, ClickHouse, MySQL, MSSQL)\n- `[db-nosql]`: NoSQL databases only (MongoDB, Elasticsearch)\n\n**Genomic formats** (`[bio]`): Enables genomic VCF/BCF, CRAM, BED, GFF3, and GTF support. Alignment formats use `pysam` and may require a reference file.\n\n**Geospatial / scientific extras**: `[geospatial]` covers FileGDB and MapInfo MIF (plus existing GeoPackage/Shapefile stack). `[lidar]`, `[mat]`, and `[geophysical]` enable LAS, MATLAB MAT, and SEG-Y/GRIB2/miniSEED respectively. Many structure formats (XYZ, CIF, PDB, ASCII Grid, CZML, EDI, WebDataset, Lotus WK1) need no extra.\n\n**Lakehouse** (`[lakehouse]`): Delta Lake, Apache Iceberg, Lance, Apache Hudi, and DuckLake. Delta, Iceberg, and DuckLake support bounded writes; Hudi is read-only for now.\n\n**Paimon** (`[paimon]`): Apache Paimon warehouse tables plus Row and Mosaic file formats. Install `[paimon-table]`, `[paimon-row]`, or `[paimon-mosaic]` individually if you only need one surface. DuckLake alone is also available as `[ducklake]`.\nSee the [API documentation](https://datenoio.github.io/iterabledata/) for details on these features.\n\nFor AI agents and LLM tooling, see **[llms.txt](llms.txt)** (short index), **[llms-full.txt](llms-full.txt)** (copy-paste recipes), the portable skill **[skills/iterabledata/SKILL.md](skills/iterabledata/SKILL.md)**, and [CONTRIBUTING.md](CONTRIBUTING.md).\n\n## AI Quick Start\n\nGenerate dataset documentation with a **local** LLM (no API key) via LM Studio or Ollama, or use OpenAI:\n\n```python\nfrom iterable.ai import doc\n\n# Local (LM Studio on http://localhost:1234/v1)\ndocumentation = doc.generate(\n    \"data.csv\",\n    provider=\"lmstudio\",\n    base_url=\"http://localhost:1234/v1\",\n    format=\"markdown\",\n)\n\n# Or analyze structure + docs in one call\nfrom iterable.ops import inspect\n\nanalysis = inspect.analyze(\"data.csv\", autodoc=True, autodoc_provider=\"openai\")\nprint(analysis[\"documentation\"])\n```\n\nNeed structured, machine-readable output? Use `generate_blocks()` to get independent documentation blocks (`general`, `schema`, `quality`, `examples`, `statistics`, `agent_skill`; plus opt-in `codebook`) plus the assembled markdown:\n\n```python\nfrom iterable.ai import doc\n\nresult = doc.generate_blocks(\n    \"data.csv\",\n    provider=\"openai\",\n    context={\"title\": \"Sales 2025\", \"description\": \"Monthly sales export\"},\n    progress=lambda event: print(event.stage, event.detail),\n)\nprint(result[\"blocks\"][\"schema\"][\"data\"])     # structured JSON\nprint(result[\"blocks\"][\"agent_skill\"][\"markdown\"])  # portable agent skill (YAML + Markdown)\nprint(result[\"full_document_markdown\"])        # assembled markdown\n```\n\nThe `agent_skill` block emits a portable skill document (YAML frontmatter + Markdown) that AI agents can load for dataset-specific load/query/safety guidance.\n\nInstall AI support: `pip install iterabledata[ai]`. See [examples/ai/](examples/ai/) and [docs/docs/api/ai.md](docs/docs/api/ai.md).\n\n### Nested schema and stats (opt-in)\n\n```python\nfrom iterable.ops import schema, stats\n\nsch = schema.infer(\"nested.jsonl\", flatten_nested=True)\nprint(sch[\"fields\"][\"capital_city.lat\"][\"type\"])\n\nsummary = stats.compute(\"nested.jsonl\", flatten_nested=True)\nprint(summary[\"capital_city.lat\"][\"mean\"])\n```\n\nFor multi-table workbooks/databases, `iterable.ai.table_profile.profile_selected_table()`\nprofiles one named sheet/table under row/time budgets (nested flattening enabled).\n\n### Format catalog (agents)\n\n```python\nfrom iterable.catalog import describe_format\n\ninfo = describe_format(\"xml\")\nprint(info[\"example_args\"])  # {'tagname': 'item'}\n```\n\nFull export: `dev/formats.json` or `export_catalog(format=\"json\")`. See [docs/docs/api/catalog.md](docs/docs/api/catalog.md).\n\n## Quick Start\n\n### Basic Reading\n\n```python\nfrom iterable import open_iterable\n\nwith open_iterable(\"data.csv.gz\") as source:\n    for row in source:\n        print(row)\n```\n\n### Writing Data\n\n```python\nfrom iterable import open_iterable\n\nwith open_iterable(\"output.jsonl.zst\", mode=\"w\") as dest:\n    for item in my_data:\n        dest.write(item)\n```\n\n## Usage Examples\n\n### Reading Compressed CSV Files\n\n```python\nfrom iterable import open_iterable\n\nwith open_iterable(\"data.csv.xz\") as source:\n    n = 0\n    for row in source:\n        n += 1\n        if n % 1000 == 0:\n            print(f\"Processed {n} rows\")\n```\n\n### Reading Different Formats\n\n```python\nfrom iterable import open_iterable\n\nwith open_iterable(\"data.jsonl\") as source:\n    for row in source:\n        print(row)\n\nwith open_iterable(\"data.parquet\") as source:\n    for row in source:\n        print(row)\n\nwith open_iterable(\"data.xml\", iterableargs={\"tagname\": \"item\"}) as source:\n    for row in source:\n        print(row)\n\nwith open_iterable(\"data.xlsx\") as source:\n    for row in source:\n        print(row)\n\n# Read GeoJSON Text Sequence (streaming, one feature per line)\nwith open_iterable('features.geojsonl') as source:\n    for feature in source:\n        print(feature['properties'], feature['geometry'])\n\n# Stream records from a TAR archive (members detected by filename)\nwith open_iterable('dataset.tar.gz', iterableargs={'members': '*.csv'}) as source:\n    for row in source:\n        print(row['_member'], row)\n\n# Read genomic VCF (requires pip install iterabledata[bio])\nwith open_iterable('variants.vcf') as source:\n    for variant in source:\n        print(variant['chrom'], variant['pos'], variant['ref'], variant['alt'])\n\n# Read genomic intervals (BED, GFF3, or GTF)\nwith open_iterable('genes.gff3') as source:\n    for feature in source:\n        print(feature['seqid'], feature['type'], feature['start'], feature['end'])\n\n# Read a Zarr array (requires pip install iterabledata[zarr])\nwith open_iterable('signals.zarr', iterableargs={'array': 'values'}) as source:\n    for row in source:\n        print(row['value'])\n\n# Read GeoParquet or FlatGeobuf (requires parquet/geospatial extras)\nwith open_iterable('roads.geoparquet') as source:\n    for feature in source:\n        print(feature.get('geometry'), feature.get('properties'))\n\n# Read an OTLP JSON export (requires pip install iterabledata[otlp])\nwith open_iterable('telemetry.otlp.json') as source:\n    for item in source:\n        print(item['signal'], item['record'])\n```\n\n### Reading from Databases\n\n```python\nfrom iterable import open_iterable\n\n# Read from PostgreSQL database\nwith open_iterable(\n    'postgresql://user:password@localhost:5432/mydb',\n    engine='postgres',\n    iterableargs={'query': 'users'}\n) as source:\n    for row in source:\n        print(row)\n\n# Read specific columns with filtering\nwith open_iterable(\n    'postgresql://localhost/mydb',\n    engine='postgres',\n    iterableargs={\n        'query': 'users',\n        'columns': ['id', 'name', 'email'],\n        'filter': 'active = TRUE'\n    }\n) as source:\n    for row in source:\n        print(row)\n\n# Read from ClickHouse database\nwith open_iterable(\n    'clickhouse://user:password@localhost:9000/analytics',\n    engine='clickhouse',\n    iterableargs={'query': 'events', 'settings': {'max_threads': 4}}\n) as source:\n    for row in source:\n        print(row)\n\n# Convert database to file\nfrom iterable.convert import convert\nconvert(\n    fromfile='postgresql://localhost/mydb',\n    tofile='users.parquet',\n    iterableargs={'engine': 'postgres', 'query': 'users'}\n)\n\n# Convert ClickHouse to Parquet\nconvert(\n    fromfile='clickhouse://localhost:9000/analytics',\n    tofile='events.parquet',\n    iterableargs={'engine': 'clickhouse', 'query': 'events'}\n)\n```\n\n### Format Detection and Encoding\n\n```python\nfrom iterable import open_iterable\nfrom iterable.helpers.detect import detect_file_type, detect_file_type_from_content\nfrom iterable.helpers.utils import detect_encoding, detect_delimiter\n\n# Detect file type and compression (uses filename extension)\nresult = detect_file_type('data.csv.gz')\nprint(f\"Type: {result['datatype']}, Codec: {result['codec']}\")\n\n# Content-based detection (for files without extensions or streams)\nwith open('data.unknown', 'rb') as f:\n    detection_result = detect_file_type_from_content(f)\n    if detection_result:\n        format_id, confidence, method = detection_result\n        print(f\"Detected format: {format_id} (confidence: {confidence:.2f}, method: {method})\")\n\n# open_iterable() automatically uses content-based detection as fallback\n# Works with files without extensions, streams, or incorrect extensions\nwith open_iterable('data.unknown') as source:  # Detects from content\n    for row in source:\n        print(row)\n\n# Detect encoding for CSV files\nencoding_info = detect_encoding('data.csv')\nprint(f\"Encoding: {encoding_info['encoding']}, Confidence: {encoding_info['confidence']}\")\n\n# Detect delimiter for CSV files\ndelimiter = detect_delimiter('data.csv', encoding=encoding_info['encoding'])\n\n# Open with detected settings\nsource = open_iterable('data.csv', iterableargs={\n    'encoding': encoding_info['encoding'],\n    'delimiter': delimiter\n})\n```\n\n### Error Handling\n\nIterableData provides a comprehensive exception hierarchy and configurable error handling:\n\n```python\nfrom iterable import open_iterable\nfrom iterable.exceptions import (\n    FormatDetectionError,\n    FormatNotSupportedError,\n    FormatParseError,\n    ReadError,\n    CodecError,\n    IterableDataError,\n)\n\n# Basic exception handling\ntry:\n    with open_iterable('data.unknown') as source:\n        for row in source:\n            process(row)\nexcept FormatDetectionError as e:\n    print(f\"Could not detect format: {e.reason}\")\n    # Try with explicit format or check file content\nexcept FormatNotSupportedError as e:\n    print(f\"Format '{e.format_id}' not supported: {e.reason}\")\n    # Install missing dependencies or use different format\nexcept FormatParseError as e:\n    print(f\"Failed to parse {e.format_id} format\")\n    if e.position:\n        print(f\"Error at position: {e.position}\")\nexcept ReadError as e:\n    print(f\"Read failed: {e}\")\nexcept IterableDataError as e:\n    print(f\"Library error: {e}\")\nexcept CodecError as e:\n    print(f\"Compression error with {e.codec_name}: {e.message}\")\n    # Check file integrity or try different codec\nexcept Exception as e:\n    print(f\"Unexpected error: {e}\")\n```\n\n**Configurable Error Policies**: Control how malformed records are handled:\n\n```python\n# Skip malformed records and continue processing\nwith open_iterable(\n    'data.csv',\n    iterableargs={'on_error': 'skip', 'error_log': 'errors.log'}\n) as src:\n    for row in src:\n        process(row)  # Only processes valid rows\n\n# Warn on errors but continue processing\nwith open_iterable(\n    'data.jsonl',\n    iterableargs={'on_error': 'warn', 'error_log': 'errors.log'}\n) as src:\n    for row in src:\n        process(row)  # Warnings logged, processing continues\n\n# Default: raise exceptions immediately (existing behavior)\nwith open_iterable('data.csv', iterableargs={'on_error': 'raise'}) as src:\n    for row in src:\n        process(row)\n```\n\n**No silent empty reads**: Under the default policy (`on_error='raise'`), a malformed non-empty file raises `FormatParseError` rather than yielding zero records. Use `on_error='skip'` or `'warn'` to tolerate bad records explicitly.\n\n**Pickle safety**: Unpickling executes arbitrary code. Reading pickle files emits a warning unless you pass `trust=True`:\n\n```python\nwith open_iterable('data.pickle', iterableargs={'trust': True}) as source:\n    for row in source:\n        process(row)\n```\n\n**Error Logging**: Structured JSON logs with context (filename, row number, byte offset, error message, original line).\n\nSee [Exception Hierarchy documentation](docs/docs/api/exceptions.md) for complete exception reference.\n\n### Querying Format Capabilities\n\n```python\nfrom iterable.helpers.capabilities import (\n    get_format_capabilities,\n    get_capability,\n    list_all_capabilities\n)\n\n# Get all capabilities for a format\ncaps = get_format_capabilities(\"csv\")\nprint(f\"CSV readable: {caps['readable']}\")\nprint(f\"CSV writable: {caps['writable']}\")\nprint(f\"CSV supports totals: {caps['totals']}\")\nprint(f\"CSV supports tables: {caps['tables']}\")\n\n# Query a specific capability\nis_writable = get_capability(\"json\", \"writable\")\nhas_totals = get_capability(\"parquet\", \"totals\")\nsupports_tables = get_capability(\"xlsx\", \"tables\")\n\n# List capabilities for all formats\nall_caps = list_all_capabilities()\nfor format_id, capabilities in all_caps.items():\n    if capabilities.get(\"tables\"):\n        print(f\"{format_id} supports multiple tables\")\n```\n\n### Format Conversion\n\n```python\nfrom iterable import open_iterable\nfrom iterable.convert import convert\n\n# Simple format conversion\nconvert('input.jsonl.gz', 'output.parquet')\n\n# Convert with options\nconvert(\n    'input.csv.xz',\n    'output.jsonl.zst',\n    iterableargs={'delimiter': ';', 'encoding': 'utf-8'},\n    batch_size=10000\n)\n\n# Convert and flatten nested structures\nconvert(\n    'input.jsonl',\n    'output.csv',\n    is_flatten=True,\n    batch_size=50000\n)\n```\n\n### Atomic Writes for Production Safety\n\nUse atomic writes to ensure output files are never left in a partially written state:\n\n```python\nfrom iterable.convert import convert\nfrom iterable.pipeline import pipeline\n\n# Convert with atomic writes (production-safe)\nresult = convert('input.csv', 'output.parquet', atomic=True)\n# Output file only appears when conversion completes successfully\n\n# Atomic writes in pipelines\npipeline(\n    source=source,\n    destination=destination,\n    process_func=transform_func,\n    atomic=True  # Ensures destination file is only created on success\n)\n```\n\n**Benefits**: Prevents data corruption from crashes, interruptions, or mid-process failures. Original files are preserved on failure.\n\n### Bulk File Conversion\n\nConvert multiple files at once using glob patterns, directories, or file lists:\n\n```python\nfrom iterable.convert import bulk_convert\n\n# Convert all CSV files matching glob pattern\nresult = bulk_convert('data/raw/*.csv.gz', 'data/processed/', to_ext='parquet')\n\n# Convert with custom filename pattern\nresult = bulk_convert('data/*.csv', 'output/', pattern='{name}.parquet')\n\n# Convert entire directory\nresult = bulk_convert('data/raw/', 'data/processed/', to_ext='parquet')\n\n# Access results\nprint(f\"Converted {result.successful_files}/{result.total_files} files\")\nprint(f\"Total rows: {result.total_rows_out}\")\nprint(f\"Throughput: {result.throughput:.0f} rows/second\")\n\n# Check individual file results\nfor file_result in result.file_results:\n    if file_result.success:\n        print(f\"✓ {file_result.source_file}: {file_result.result.rows_out} rows\")\n    else:\n        print(f\"✗ {file_result.source_file}: {file_result.error}\")\n```\n\n**Features**: Error resilience (continues if one file fails), aggregated metrics, flexible output naming with placeholders (`{name}`, `{stem}`, `{ext}`).\n\n### Progress Tracking and Metrics\n\nTrack conversion and pipeline progress with callbacks, progress bars, and structured metrics:\n\n```python\nfrom iterable.convert import convert\nfrom iterable.pipeline import pipeline\n\n# Progress callback for conversions\ndef progress_cb(stats):\n    print(f\"Progress: {stats['rows_read']} rows read, \"\n          f\"{stats['rows_written']} rows written, \"\n          f\"{stats.get('elapsed', 0):.2f}s elapsed\")\n\n# Convert with progress tracking\nresult = convert(\n    'input.csv',\n    'output.parquet',\n    progress=progress_cb,\n    show_progress=True  # Also shows tqdm progress bar\n)\n\n# Access conversion metrics\nprint(f\"Converted {result.rows_out} rows in {result.elapsed_seconds:.2f}s\")\nprint(f\"Read {result.bytes_read} bytes, wrote {result.bytes_written} bytes\")\n\n# Pipeline with progress and metrics\nresult = pipeline(\n    source=source,\n    destination=destination,\n    process_func=transform_func,\n    progress=progress_cb  # Progress callback\n)\n\n# Access pipeline metrics (supports both attribute and dict access)\nprint(f\"Processed {result.rows_processed} rows\")\nprint(f\"Throughput: {result.throughput:.0f} rows/second\")\nprint(f\"Exceptions: {result.exceptions}\")\n# Backward compatible: result['rec_count'] also works\n```\n\n**Features**: Real-time progress callbacks, automatic progress bars with `tqdm`, structured metrics objects (`ConversionResult`, `PipelineResult`).\n\n### Using Pipeline for Data Processing\n\n```python\nfrom iterable import open_iterable\nfrom iterable.pipeline import pipeline\n\nsource = open_iterable('input.parquet')\ndestination = open_iterable('output.jsonl.xz', mode='w')\n\ndef transform_record(record, state):\n    \"\"\"Transform each record\"\"\"\n    # Add processing logic\n    out = {}\n    for key in ['name', 'email', 'age']:\n        if key in record:\n            out[key] = record[key]\n    return out\n\ndef progress_callback(stats, state):\n    \"\"\"Called every trigger_on records\"\"\"\n    print(f\"Processed {stats['rec_count']} records, \"\n          f\"Duration: {stats.get('duration', 0):.2f}s\")\n\ndef final_callback(stats, state):\n    \"\"\"Called when processing completes\"\"\"\n    print(f\"Total records: {stats['rec_count']}\")\n    print(f\"Total time: {stats['duration']:.2f}s\")\n\nresult = pipeline(\n    source=source,\n    destination=destination,\n    process_func=transform_record,\n    trigger_func=progress_callback,\n    trigger_on=1000,\n    final_func=final_callback,\n    start_state={},\n    atomic=True  # Use atomic writes for production safety\n)\n\n# Access pipeline metrics\nprint(f\"Throughput: {result.throughput:.0f} rows/second\")\n\nsource.close()\ndestination.close()\n```\n\n### Manual Format and Codec Usage\n\n```python\nfrom iterable.datatypes.jsonl import JSONLinesIterable\nfrom iterable.datatypes.bsonf import BSONIterable\nfrom iterable.codecs.gzipcodec import GZIPCodec\nfrom iterable.codecs.lzmacodec import LZMACodec\n\n# Read gzipped JSONL\nread_codec = GZIPCodec('input.jsonl.gz', mode='r', open_it=True)\nreader = JSONLinesIterable(codec=read_codec)\n\n# Write LZMA compressed BSON\nwrite_codec = LZMACodec('output.bson.xz', mode='wb', open_it=False)\nwriter = BSONIterable(codec=write_codec, mode='w')\n\nfor row in reader:\n    writer.write(row)\n\nreader.close()\nwriter.close()\n```\n\n### Cloud Storage Support\n\nRead and write data directly from cloud object storage (S3, GCS, Azure):\n\n```python\nfrom iterable import open_iterable\n\n# Read from S3\nwith open_iterable('s3://my-bucket/data/events.csv') as source:\n    for row in source:\n        print(row)\n\n# Read compressed file from GCS\nwith open_iterable('gs://my-bucket/data/events.jsonl.gz') as source:\n    for row in source:\n        process(row)\n\n# Write to Azure Blob Storage\nwith open_iterable(\n    'az://my-container/output/results.jsonl',\n    mode='w',\n    iterableargs={'storage_options': {'connection_string': '...'}}\n) as dest:\n    dest.write({'name': 'Alice', 'age': 30})\n    dest.write({'name': 'Bob', 'age': 25})\n```\n\n**Supported Providers**:\n\n- Amazon S3: `s3://` and `s3a://` schemes\n- Google Cloud Storage: `gs://` and `gcs://` schemes\n- Azure Blob Storage: `az://`, `abfs://`, and `abfss://` schemes\n\n**Installation**: `pip install iterabledata[cloud]`\n\n**Note**: DuckDB engine does not support cloud storage URIs; use `engine='internal'` (default).\n\n### Using DuckDB Engine with Pushdown Optimizations\n\nThe DuckDB engine provides high-performance querying with advanced optimizations:\n\n```python\nfrom iterable import open_iterable\n\n# Basic DuckDB usage\nsource = open_iterable('data.csv.gz', engine='duckdb')\ntotal = source.totals()  # Fast counting\nfor row in source:\n    print(row)\nsource.close()\n\n# Column projection pushdown (only read specified columns)\nwith open_iterable(\n    'data.csv',\n    engine='duckdb',\n    iterableargs={'columns': ['name', 'age']}  # Reduces I/O and memory\n) as src:\n    for row in src:\n        process(row)\n\n# Filter pushdown (filter at database level)\nwith open_iterable(\n    'data.csv',\n    engine='duckdb',\n    iterableargs={'filter': \"age > 18 AND status = 'active'\"}\n) as src:\n    for row in src:\n        process(row)\n\n# Combined column projection and filtering\nwith open_iterable(\n    'data.parquet',\n    engine='duckdb',\n    iterableargs={\n        'columns': ['name', 'age', 'email'],\n        'filter': 'age > 18'\n    }\n) as src:\n    for row in src:\n        process(row)\n\n# Direct SQL query support\nwith open_iterable(\n    'data.parquet',\n    engine='duckdb',\n    iterableargs={\n        'query': 'SELECT name, age FROM read_parquet(\\'data.parquet\\') WHERE age > 18 ORDER BY age DESC LIMIT 100'\n    }\n) as src:\n    for row in src:\n        process(row)\n```\n\n**Supported Formats**: CSV, JSONL, NDJSON, JSON, Parquet  \n**Supported Codecs**: GZIP, ZStandard (.zst)  \n**Benefits**: Reduced I/O, lower memory usage, faster processing through database-level optimizations\n\n### Bulk Operations\n\n```python\nfrom iterable import open_iterable\n\nsource = open_iterable('input.jsonl')\ndestination = open_iterable('output.parquet', mode='w')\n\n# Read and write in batches for better performance\nbatch = []\nfor row in source:\n    batch.append(row)\n    if len(batch) >= 10000:\n        destination.write_bulk(batch)\n        batch = []\n\n# Write remaining records\nif batch:\n    destination.write_bulk(batch)\n\nsource.close()\ndestination.close()\n```\n\n### Performance-oriented Conversion\n\nUse native batches when both endpoints are Parquet or Arrow/Feather. The\nselection is pushed into the columnar reader when supported; otherwise the\nconversion safely falls back to the regular row/bulk loop:\n\n```python\nfrom iterable.convert import BatchSelection, convert\n\nconvert(\n    'events.parquet',\n    'events-copy.parquet',\n    use_native_batch=True,\n    selection=BatchSelection(columns=('event_id', 'created_at'), batch_size=8192),\n    toiterableargs={'row_group_size': 32768},\n)\n```\n\nFor general compression workloads, use the balanced profile. Choose `fast`\nfor CPU-bound pipelines or `max` for archival output:\n\n```python\nwith open_iterable('events.jsonl.zst', mode='w', codecargs={'profile': 'fast'}) as dest:\n    dest.write_bulk(records)\n```\n\nSee [native batches](docs/docs/api/native-batches.md), [codec profiles](docs/docs/api/codecs.md),\nand the [performance guide](docs/docs/getting-started/performance.md) for selection,\nmemory, row-group, and fallback behavior.\n\n### Working with Excel Files\n\n```python\nfrom iterable import open_iterable\nfrom iterable.ai.fileinfo import open_table\n\n# Read Excel file (specify sheet or page)\nxls_file = open_iterable('data.xlsx', iterableargs={'page': 0})\n\nfor row in xls_file:\n    print(row)\nxls_file.close()\n\n# Open a named sheet (uses page index under the hood)\nsheet = open_table('data.xlsx', 'Sheet2')\nfor row in sheet:\n    print(row)\nsheet.close()\n```\n\n### XML Processing\n\n```python\nfrom iterable import open_iterable\n\n# Parse XML with specific tag name\nxml_file = open_iterable(\n    'data.xml',\n    iterableargs={\n        'tagname': 'book',\n        'prefix_strip': True  # Strip XML namespace prefixes\n    }\n)\n\nfor item in xml_file:\n    print(item)\nxml_file.close()\n```\n\n### DataFrame Bridges\n\nConvert iterable data to Pandas, Polars, or Dask DataFrames:\n\n```python\nfrom iterable import open_iterable\n\n# Convert to Pandas DataFrame\nwith open_iterable('data.csv.gz') as source:\n    df = source.to_pandas()\n    print(df.head())\n\n# Chunked processing for large files\nwith open_iterable('large_data.csv') as source:\n    for df_chunk in source.to_pandas(chunksize=100_000):\n        # Process each chunk\n        result = df_chunk.groupby('category').sum()\n        process_chunk(result)\n\n# Convert to Polars DataFrame\nwith open_iterable('data.csv.gz') as source:\n    df = source.to_polars()\n    print(df.head())\n\n# Convert to Dask DataFrame (single file)\nwith open_iterable('data.csv.gz') as source:\n    ddf = source.to_dask()\n    result = ddf.groupby('category').sum().compute()\n\n# Multi-file Dask DataFrame (automatic format detection)\nfrom iterable.helpers.bridges import to_dask\n\nddf = to_dask(['file1.csv', 'file2.jsonl', 'file3.parquet'])\nresult = ddf.groupby('category').sum().compute()\n```\n\n**Note**: DataFrame bridges require optional dependencies. Install with:\n\n```bash\npip install iterabledata[dataframes]  # All DataFrame libraries\n# Or individually:\npip install pandas\npip install polars\npip install \"dask[dataframe]\"\n```\n\n### Type Hints and Type Safety\n\nIterableData includes complete type annotations and typed helper functions for modern Python development:\n\n```python\nfrom iterable import open_iterable\nfrom iterable.helpers.typed import as_dataclasses, as_pydantic\nfrom dataclasses import dataclass\nfrom pydantic import BaseModel\n\n# Type aliases for better code readability\nfrom iterable.types import Row, IterableArgs, CodecArgs\n\n# Convert to dataclasses for type safety\n@dataclass\nclass Person:\n    name: str\n    age: int\n    email: str | None = None\n\nwith open_iterable('people.csv') as source:\n    for person in as_dataclasses(source, Person):\n        # Full IDE autocomplete and type checking\n        print(person.name, person.age)\n\n# Convert to Pydantic models with validation\nclass PersonModel(BaseModel):\n    name: str\n    age: int\n    email: str | None = None\n\nwith open_iterable('people.jsonl') as source:\n    for person in as_pydantic(source, PersonModel, validate=True):\n        # Automatic schema validation\n        print(person.name, person.age)\n        # Access as Pydantic model with all features\n```\n\n**Benefits**:\n\n- Complete type annotations across the public API\n- `py.typed` marker file enables mypy, pyright, and other type checkers\n- Typed helpers provide IDE autocomplete and type safety\n- Pydantic validation catches schema issues early\n\n**Installation**: `pip install iterabledata[pydantic]` for Pydantic support\n\n### Advanced: Converting Compressed XML to Parquet\n\n```python\nfrom iterable.datatypes.xml import XMLIterable\nfrom iterable.datatypes.parquet import ParquetIterable\nfrom iterable.codecs.bz2codec import BZIP2Codec\n\n# Read compressed XML\nread_codec = BZIP2Codec('data.xml.bz2', mode='r')\nreader = XMLIterable(codec=read_codec, tagname='page')\n\n# Write to Parquet with schema adaptation\nwriter = ParquetIterable(\n    'output.parquet',\n    mode='w',\n    use_pandas=False,\n    adapt_schema=True,\n    batch_size=10000\n)\n\nbatch = []\nfor row in reader:\n    batch.append(row)\n    if len(batch) >= 10000:\n        writer.write_bulk(batch)\n        batch = []\n\nif batch:\n    writer.write_bulk(batch)\n\nreader.close()\nwriter.close()\n```\n\n## API Reference\n\n### Main Functions\n\n#### `open_iterable(filename, mode='r', engine='internal', codecargs={}, iterableargs={})`\n\nOpens a file and returns an iterable object.\n\n**Parameters:**\n\n- `filename` (str): Path to the file (supports local files and cloud storage URIs: `s3://`, `gs://`, `az://`)\n- `mode` (str): File mode ('r' for read, 'w' for write)\n- `engine` (str): Processing engine ('internal' or 'duckdb')\n- `codecargs` (dict): Arguments for codec initialization\n- `iterableargs` (dict): Arguments for iterable initialization\n  - `columns` (list[str]): For DuckDB engine, only read specified columns (pushdown optimization)\n  - `filter` (str | callable): For DuckDB engine, filter rows at database level (SQL string or Python callable)\n  - `query` (str): For DuckDB engine, execute custom SQL query (read-only)\n  - `on_error` (str): Error policy ('raise', 'skip', or 'warn')\n  - `error_log` (str | file-like): Path or file object for structured error logging\n  - `storage_options` (dict): Cloud storage authentication options\n\n**Returns:** Iterable object for the detected file type\n\n#### `detect_file_type(filename)`\n\nDetects file type and compression codec from filename.\n\n**Returns:** Dictionary with `success`, `datatype`, and `codec` keys\n\n#### `convert(fromfile, tofile, iterableargs={}, toiterableargs={}, scan_limit=1000, batch_size=50000, silent=True, is_flatten=False, use_totals=False, progress=None, show_progress=False, atomic=False, use_native_batch=False, selection=None, strict_native=False) -> ConversionResult`\n\nConverts data between formats.\n\n**Parameters:**\n\n- `fromfile` (str): Source file path\n- `tofile` (str): Destination file path\n- `iterableargs` (dict): Options for reading source file\n- `toiterableargs` (dict): Options for writing destination file\n- `scan_limit` (int): Number of records to scan for schema detection\n- `batch_size` (int): Batch size for bulk operations\n- `silent` (bool): Suppress progress output\n- `is_flatten` (bool): Flatten nested structures\n- `use_totals` (bool): Use total count for progress tracking (if available)\n- `progress` (callable): Optional callback function receiving progress stats dictionary\n- `show_progress` (bool): Display progress bar using tqdm (if available)\n- `atomic` (bool): Write to temporary file and atomically rename on success\n- `use_native_batch` (bool): Request native columnar batch transfer when both endpoints support it\n- `selection` (`BatchSelection` or dict): Optional columns, row range, slice, table, or backend predicate selection\n- `strict_native` (bool): Raise instead of falling back when native batching or the requested selection is unsupported\n\n**Returns:** `ConversionResult` object with:\n\n- `rows_in` (int): Total rows read\n- `rows_out` (int): Total rows written\n- `elapsed_seconds` (float): Conversion time\n- `bytes_read` (int | None): Bytes read (if available)\n- `bytes_written` (int | None): Bytes written (if available)\n- `errors` (list[Exception]): List of errors encountered\n\n#### `bulk_convert(source, destination, pattern=None, to_ext=None, **kwargs) -> BulkConversionResult`\n\nConvert multiple files at once using glob patterns, directories, or file lists.\n\n**Parameters:**\n\n- `source` (str): Glob pattern, directory path, or file path\n- `destination` (str): Output directory or filename pattern\n- `pattern` (str): Filename pattern with placeholders (`{name}`, `{stem}`, `{ext}`)\n- `to_ext` (str): Replace file extension (e.g., `'parquet'`)\n- `**kwargs`: All parameters from `convert()` function\n\n**Returns:** `BulkConversionResult` object with:\n\n- `total_files` (int): Total files processed\n- `successful_files` (int): Files successfully converted\n- `failed_files` (int): Files that failed\n- `total_rows_in` (int): Total rows read across all files\n- `total_rows_out` (int): Total rows written across all files\n- `total_elapsed_seconds` (float): Total conversion time\n- `file_results` (list[FileConversionResult]): Per-file results\n- `errors` (list[Exception]): All errors encountered\n- `throughput` (float | None): Rows per second\n\n#### `pipeline(source, destination, process_func, trigger_func=None, trigger_on=1000, final_func=None, reset_iterables=True, skip_nulls=True, start_state=None, debug=False, batch_size=1000, progress=None, atomic=False) -> PipelineResult`\n\nExecute a data processing pipeline.\n\n**Parameters:**\n\n- `source` (BaseIterable): Source iterable to read from\n- `destination` (BaseIterable | None): Destination iterable to write to\n- `process_func` (callable): Function to process each record\n- `trigger_func` (callable | None): Function called periodically during processing\n- `trigger_on` (int): Number of records between trigger function calls\n- `final_func` (callable | None): Function called after processing completes\n- `reset_iterables` (bool): Reset iterables before processing\n- `skip_nulls` (bool): Skip None results from process_func\n- `start_state` (dict | None): Initial state dictionary\n- `debug` (bool): Raise exceptions instead of catching them\n- `batch_size` (int): Number of records to batch before writing\n- `progress` (callable | None): Optional callback function for progress updates\n- `atomic` (bool): Use atomic writes if destination is a file\n\n**Returns:** `PipelineResult` object with:\n\n- `rows_processed` (int): Total rows processed\n- `elapsed_seconds` (float): Processing time\n- `throughput` (float | None): Rows per second\n- `exceptions` (int): Number of exceptions encountered\n- `nulls` (int): Number of null results\n- Supports both attribute access (`result.rows_processed`) and dictionary access (`result['rec_count']`) for backward compatibility\n\n### Iterable Methods\n\nAll iterable objects support:\n\n- `read(skip_empty=True) -> Row` - Read single record\n- `read_bulk(num=DEFAULT_BULK_NUMBER) -> list[Row]` - Read multiple records\n- `write(record)` - Write single record\n- `write_bulk(records)` - Write multiple records\n- `reset()` - Reset iterator to beginning\n- `close()` - Close file handles\n- `to_pandas(chunksize=None)` - Convert to pandas DataFrame (optional chunked processing)\n- `to_polars(chunksize=None)` - Convert to Polars DataFrame (optional chunked processing)\n- `to_dask(chunksize=1000000)` - Convert to Dask DataFrame\n- `list_tables(filename=None) -> list[str] | None` - List available tables/sheets/datasets\n- `has_tables() -> bool` - Check if format supports multiple tables\n\n### Helper Functions\n\n#### `as_dataclasses(iterable, dataclass_type, skip_empty=True) -> Iterator[T]`\n\nConvert dict-based rows from an iterable into dataclass instances.\n\n**Parameters:**\n\n- `iterable` (BaseIterable): The iterable to read rows from\n- `dataclass_type` (type[T]): The dataclass type to convert rows to\n- `skip_empty` (bool): Whether to skip empty rows\n\n**Returns:** Iterator of dataclass instances\n\n#### `as_pydantic(iterable, model_type, skip_empty=True, validate=True) -> Iterator[T]`\n\nConvert dict-based rows from an iterable into Pydantic model instances.\n\n**Parameters:**\n\n- `iterable` (BaseIterable): The iterable to read rows from\n- `model_type` (type[T]): The Pydantic model type to convert rows to\n- `skip_empty` (bool): Whether to skip empty rows\n- `validate` (bool): Whether to validate rows against the model schema\n\n**Returns:** Iterator of Pydantic model instances\n\n**Raises:** `ImportError` if pydantic is not installed\n\n#### `to_dask(files, chunksize=1000000, **iterableargs) -> DaskDataFrame`\n\nConvert multiple files to a unified Dask DataFrame with automatic format detection.\n\n**Parameters:**\n\n- `files` (str | list[str]): Single file path or list of file paths\n- `chunksize` (int): Number of rows per partition\n- `**iterableargs`: Additional arguments to pass to `open_iterable()` for each file\n\n**Returns:** Dask DataFrame containing data from all files\n\n**Raises:** `ImportError` if dask or pandas is not installed\n\n## Engines\n\n### Internal Engine (Default)\n\nThe internal engine uses pure Python implementations for all formats. It supports all file types and compression codecs.\n\n### DuckDB Engine\n\nThe DuckDB engine provides high-performance querying capabilities for supported formats:\n\n- **Formats**: CSV, JSONL, NDJSON, JSON\n- **Codecs**: GZIP, ZStandard (.zst)\n- **Features**: Fast querying, totals counting, SQL-like operations\n\nUse `engine='duckdb'` when opening files:\n\n```python\nsource = open_iterable('data.csv.gz', engine='duckdb')\n```\n\n## Examples Directory\n\nSee the [examples](examples/) directory for more complete examples:\n\n- `simplewiki/` - Processing Wikipedia XML dumps\n\n## More Examples and Tests\n\nSee the [tests](tests/) directory for comprehensive usage examples and test cases.\n\n**Contributors**: run the full suite with `pytest --verbose`. The performance regression gate is opt-in:\n\n```bash\npytest tests/test_performance_regression.py -m performance --no-cov\n```\n\nRegenerate baselines intentionally after legitimate performance changes:\n\n```bash\npytest tests/test_performance_regression.py -m performance --no-cov --update-baselines\n```\n\nSee [AGENTS.md](AGENTS.md) for development conventions.\n\n## AI Integration Guides\n\nIterableData can be integrated with AI platforms and frameworks for intelligent data processing:\n\n- **[AI Frameworks](docs/integrations/AI_FRAMEWORKS.md)** - Integration with LangChain, CrewAI, and AutoGen\n  - Tool creation for data reading and format conversion\n  - Schema inference and data quality analysis\n  - Multi-agent workflows for data processing\n- **[OpenAI](docs/integrations/OPENAI.md)** - Direct OpenAI API integration (GPT-4, GPT-3.5, etc.)\n  - Function calling and Assistants API\n  - Structured outputs for consistent results\n  - Natural language data analysis and transformation\n- **[Claude](docs/integrations/CLAUDE.md)** - Anthropic Claude AI integration\n  - Claude API integration with tools support\n  - Intelligent data analysis and schema inference\n  - Format conversion with AI guidance\n  - Data quality assessment and documentation\n- **[Gemini](docs/integrations/GEMINI.md)** - Google Gemini AI integration\n  - Natural language data analysis\n  - Intelligent format conversion with AI guidance\n  - Schema documentation and data quality assessment\n  - Function calling integration\n\nThese guides provide patterns, examples, and best practices for combining IterableData's unified data interface with AI capabilities.\n\n## Related Projects\n\nThis library is part of the [Dateno](https://dateno.io) open-source project. It is also used in:\n\n- [undatum](https://github.com/datacoon/undatum) - Command line data processing tool\n- [datacrafter](https://github.com/apicrafter/datacrafter) - Data processing ETL engine\n\n## License\n\nIterableData is part of the [Dateno](https://dateno.io) open-source project.\n\n- **Code** is licensed under the [MIT License](LICENSE). Copyright © 2022–2026 Dateno.\n- **Documentation and data** (including this README, the docs site, and test fixtures) are licensed under [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/).\n\nSee the [license page](https://datenoio.github.io/iterabledata/license) for attribution guidance.\n\n## Contributing\n\nContributions are welcome! Please feel free to submit pull requests or open issues.\n\n## Changelog\n\nSee [CHANGELOG.md](CHANGELOG.md) for detailed version history.\n\n### Version 1.0.23 (2026-08-18)\n\n- **MCP Registry**: Shorten `server.json` description to the 100-character registry limit\n\n### Version 1.0.22 (2026-08-18)\n\n- **Coding agents**: Canonical `open_iterable` / `convert` imports, `llms-full.txt`, portable skill, cookbook\n- **Discovery**: MCP `server.json`, `/.well-known/llms.txt`, prompt-eval; OIDC publish to the MCP Registry on tagged releases\n- **CI**: Windows file-lock and CSV newline fixes; optional extras skip cleanly in the base install\n\n### Version 1.0.21 (2026-08-10)\n\n- **Nested schema/stats**: Opt-in `flatten_nested=True` on `schema.infer()` / `stats.compute()` for dotted paths like `capital_city.lat`\n- **Bounded table profiling**: `iterable.ai.table_profile.profile_selected_table()` with row/time budgets for multi-table sources\n- **Excel/SQLite hardening**: Correct named-sheet opens, skip blank header rows, recover bad XLSX dimensions; SQLite prefers read-only opens and quotes table names\n\n### Version 1.0.20 (2026-08-07)\n\n- **Agent skill block (`agent_skill`)**: Default `generate_blocks()` now includes a portable agent-skill document (YAML frontmatter + Markdown) with dataset facts, workflow, and safety guidance\n- **Safer usage examples**: Examples / legacy autodoc prompts require `python`/`r`/`sql`, SQL against table `dataset`, and read-only constraints\n- **Schema examples**: Nested provider `example` values are coerced to strings for structured-output validation\n\n### Version 1.0.18 (2026-07-22)\n\n- **Open-data formats (experimental)**: FileGDB, MapInfo MIF, ASCII Grid, E00, LAS, BAG, CZML, XYZ, CIF, PDB, MATLAB MAT, SEG-Y, GRIB2, miniSEED, EDI, Access MDB, Lotus 1-2-3, WebDataset, R fst, HDT, IATI\n- **Extras**: `[lidar]`, `[mat]`, `[geophysical]`, `[access]`, `[fst]` (plus existing `[geospatial]`, `[hdf5]`, `[xml]`, `[rdf]`)\n- **New lakehouse formats**: Apache Paimon Row/Mosaic files and Paimon catalog tables; DuckLake (`ducklake` / `lakehouse` extras)\n- **Lakehouse writes**: Bounded create/append/overwrite for Delta Lake and Iceberg; DuckLake and Paimon table writes; Hudi writes deferred\n- **Extras (lakehouse)**: `[paimon]`, `[paimon-table]`, `[paimon-row]`, `[paimon-mosaic]`, and `[ducklake]`; DuckLake folded into `[lakehouse]`\n- **Docs**: All format stub pages expanded; formats index and sidebars updated; Delta/Iceberg write docs corrected\n\n### Version 1.0.17 (2026-07-16)\n\n- **Performance**: Bounded Parquet/Arrow/Lance batching, shared row/bulk cursors, cached conversion totals, and opt-in native columnar batch conversion.\n- **Compression**: `fast`, `balanced`, and `max` codec profiles with effective-setting diagnostics.\n- **Format support**: Zarr, GeoParquet, FlatGeobuf, CRAM, BED, GFF3/GTF, and OTLP JSON/Protobuf profiles.\n- **Repository quality**: Versioned capability metadata, distribution-content checks, fixture isolation, optional-family CI coverage, and OIDC release guidance.\n\n### Version 1.0.16 (2026-07-15)\n\n- **New formats**: GeoJSON Text Sequence (`geojsonseq`), TAR container (`tar`), genomic VCF/BCF (`genomic_vcf`, `bio` extra)\n- **Security**: XXE-safe XML parsing, AST-whitelisted filter expressions, pickle `trust=True` acknowledgement\n- **Streaming**: Snappy/LZO streaming codecs; lazy/batch readers for Shapefile, Arrow, Lance, Delta, Iceberg\n- **Error policy**: Malformed input raises typed errors by default; `open_iterable()` surfaces `IterableDataError` subclasses\n- **Quality**: Performance regression gate in CI; refactored core entry points; expanded test resilience\n\n### Version 1.0.15 (2026-07-04)\n\n- **Bare install importability**: `import iterable` no longer requires optional BSON or Pydantic dependencies\n",
  "bytes": 55621,
  "sha": "1fcec436fdc434723bf1003e3bbf73e951dd5dad3bd83c2b4f7aea8a3c295d31",
  "repo_slug": "datenoio/iterabledata",
  "fonte": "repo",
  "truncated": false,
  "api": "https://agentalog.com/api/listings/mcp_io_github_datenoio_iterabledata_543f2de0/readme"
}