{
  "markdown": "# agents\n\nAI agent tooling for data engineering workflows. Includes an [MCP server](./astro-airflow-mcp/) for Airflow, a [CLI tool (`af`)](./astro-airflow-mcp/README.md#airflow-cli-tool) for interacting with Airflow from your terminal, and [skills](#skills) that extend AI coding agents with specialized capabilities for working with Airflow and data warehouses. Works with [Claude Code](https://docs.anthropic.com/en/docs/claude-code), [Cursor](https://cursor.com), and other agentic coding tools.\n\nBuilt by [Astronomer](https://www.astronomer.io/). [Apache 2.0 licensed](https://github.com/astronomer/agents/blob/main/LICENSE) and compatible with open-source Apache Airflow.\n\n## Table of Contents\n\n<!-- START doctoc generated TOC please keep comment here to allow auto update -->\n<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->\n\n- [Installation](#installation)\n  - [Quick Start](#quick-start)\n  - [Compatibility](#compatibility)\n  - [Claude Code](#claude-code)\n  - [Cursor](#cursor)\n  - [Other MCP Clients](#other-mcp-clients)\n- [Features](#features)\n  - [MCP Server](#mcp-server)\n  - [Skills](#skills)\n- [Why Astro?](#why-astro)\n  - [User Journeys](#user-journeys)\n  - [Airflow CLI (`af`)](#airflow-cli-af)\n- [Configuration](#configuration)\n  - [Warehouse Connections](#warehouse-connections)\n  - [Airflow](#airflow)\n- [Usage](#usage)\n  - [Getting Started](#getting-started)\n- [Development](#development)\n  - [Local Development Setup](#local-development-setup)\n  - [Adding Skills](#adding-skills)\n- [Troubleshooting](#troubleshooting)\n  - [Common Issues](#common-issues)\n- [Contributing](#contributing)\n- [Roadmap](#roadmap)\n- [License](#license)\n\n<!-- END doctoc generated TOC please keep comment here to allow auto update -->\n\n## Installation\n\n### Quick Start\n\n```bash\nnpx skills add astronomer/agents --skill '*'\n```\n\nThis installs all Astronomer skills into your project via [skills.sh](https://skills.sh). You'll be prompted to select which agents to install to. To also select skills individually, omit the `--skill` flag.\n\n> [!IMPORTANT]\n> **Claude Code users:** We recommend using the plugin instead (see [Claude Code](#claude-code) section below) for better integration with MCP servers and hooks.\n\n### Compatibility\n\n**Skills:** Works with [25+ AI coding agents](https://github.com/vercel-labs/add-skill?tab=readme-ov-file#available-agents) including Claude Code, Cursor, VS Code (GitHub Copilot), Windsurf, Cline, and more.\n\n**MCP Server:** Works with any [MCP-compatible client](https://modelcontextprotocol.io/clients) including Claude Desktop, VS Code, and others.\n\n> [!NOTE]\n> **Open-source Airflow users:** The MCP server works with any Airflow 2.x/3.x REST API. Set `AIRFLOW_API_URL` to your self-hosted instance. Skills are tool-agnostic and work with any Airflow deployment.\n\n### Claude Code\n\n```bash\n# Add the marketplace and install the plugin\nclaude plugin marketplace add astronomer/agents\nclaude plugin install astronomer-data@astronomer\n\n# Upgrading from the old plugin name? Uninstall first:\n# claude plugin uninstall data@astronomer && claude plugin marketplace update && claude plugin install astronomer-data@astronomer\n```\n\nThe plugin includes the Airflow MCP server that runs via `uvx` from PyPI. Data warehouse queries are handled by the `analyzing-data` skill using a background Jupyter kernel.\n\n### Cursor\n\nCursor supports both MCP servers and skills.\n\n**MCP Server** - Click to install:\n\n<a href=\"https://cursor.com/en-US/install-mcp?name=astro-airflow-mcp&config=eyJjb21tYW5kIjoidXZ4IiwiYXJncyI6WyJhc3Ryby1haXJmbG93LW1jcCIsIi0tdHJhbnNwb3J0Iiwic3RkaW8iXX0\"><img src=\"https://cursor.com/deeplink/mcp-install-dark.svg\" alt=\"Add Airflow MCP to Cursor\" height=\"32\"></a>\n\n**Skills** - Install to your project:\n\n```bash\nnpx skills add astronomer/agents --skill '*' -a cursor\n```\n\nThis installs skills to `.cursor/skills/` in your project.\n\n<details>\n<summary>Manual MCP configuration</summary>\n\nAdd to `~/.cursor/mcp.json`:\n\n```json\n{\n  \"mcpServers\": {\n    \"airflow\": {\n      \"command\": \"uvx\",\n      \"args\": [\"astro-airflow-mcp\", \"--transport\", \"stdio\"]\n    }\n  }\n}\n```\n\n</details>\n\n<details>\n<summary>Enable hooks (session management)</summary>\n\nCreate `.cursor/hooks.json` in your project:\n\n```json\n{\n  \"version\": 1,\n  \"hooks\": {\n    \"stop\": [\n      {\n        \"command\": \"uv run $CURSOR_PROJECT_DIR/.cursor/skills/analyzing-data/scripts/cli.py stop\",\n        \"timeout\": 10\n      }\n    ]\n  }\n}\n```\n\n**What these hooks do:**\n- `stop`: Cleans up kernel when session ends\n\n</details>\n\n### Other MCP Clients\n\nFor any MCP-compatible client (Claude Desktop, VS Code, etc.):\n\n```bash\n# Airflow MCP\nuvx astro-airflow-mcp --transport stdio\n\n# With remote Airflow\nAIRFLOW_API_URL=https://your-airflow.example.com \\\nAIRFLOW_USERNAME=admin \\\nAIRFLOW_PASSWORD=admin \\\nuvx astro-airflow-mcp --transport stdio\n```\n\n## Features\n\nThe `astronomer-data` plugin bundles an MCP server and skills into a single installable package.\n\n### MCP Server\n\n| Server | Description |\n|--------|-------------|\n| **[Airflow](https://github.com/astronomer/agents/tree/main/astro-airflow-mcp)** | Full Airflow REST API integration via [astro-airflow-mcp](https://github.com/astronomer/agents/tree/main/astro-airflow-mcp): DAG management, triggering, task logs, system health |\n\n### Skills\n\n#### Data Discovery & Analysis\n\n| Skill | Description |\n|-------|-------------|\n| [warehouse-init](./skills/warehouse-init/) | Initialize schema discovery - generates `.astro/warehouse.md` for instant lookups |\n| [analyzing-data](./skills/analyzing-data/) | SQL-based analysis to answer business questions (uses background Jupyter kernel) |\n| [checking-freshness](./skills/checking-freshness/) | Check how current your data is |\n| [profiling-tables](./skills/profiling-tables/) | Comprehensive table profiling and quality assessment |\n\n#### Data Lineage\n\n| Skill | Description |\n|-------|-------------|\n| [tracing-downstream-lineage](./skills/tracing-downstream-lineage/) | Analyze what breaks if you change something |\n| [tracing-upstream-lineage](./skills/tracing-upstream-lineage/) | Trace where data comes from |\n| [annotating-task-lineage](./skills/annotating-task-lineage/) | Add manual lineage to tasks using inlets/outlets |\n| [creating-openlineage-extractors](./skills/creating-openlineage-extractors/) | Build custom OpenLineage extractors for operators |\n\n#### DAG Development\n\n| Skill | Description |\n|-------|-------------|\n| [airflow](./skills/airflow/) | Main entrypoint - routes to specialized Airflow skills |\n| [setting-up-astro-project](./skills/setting-up-astro-project/) (Astro) | Initialize and configure new Astro/Airflow projects |\n| [managing-astro-local-env](./skills/managing-astro-local-env/) (Astro) | Manage local Airflow environment (start, stop, logs, troubleshoot) |\n| [authoring-dags](./skills/authoring-dags/) | Create and validate Airflow DAGs with best practices |\n| [blueprint](./skills/blueprint/) | Compose DAGs from YAML using reusable templates with Pydantic validation ([airflow-blueprint](https://github.com/astronomer/blueprint)) |\n| [testing-dags](./skills/testing-dags/) | Test and debug Airflow DAGs locally |\n| [debugging-dags](./skills/debugging-dags/) | Deep failure diagnosis and root cause analysis |\n| [deploying-airflow](./skills/deploying-airflow/) | Deploy Airflow DAGs and projects (Astro, Docker Compose, Kubernetes) |\n| [airflow-hitl](./skills/airflow-hitl/) | Human-in-the-loop workflows: approval gates, form input, branching (Airflow 3.1+) |\n\n#### dbt Integration\n\n| Skill | Description |\n|-------|-------------|\n| [cosmos-dbt-core](./skills/cosmos-dbt-core/) | Run dbt Core projects as Airflow DAGs using [Astronomer Cosmos](https://github.com/astronomer/astronomer-cosmos) |\n| [cosmos-dbt-fusion](./skills/cosmos-dbt-fusion/) | Run dbt Fusion projects with Cosmos (Snowflake/Databricks only) |\n\n#### Migration\n\n| Skill | Description |\n|-------|-------------|\n| [migrating-airflow-2-to-3](./skills/migrating-airflow-2-to-3/) | Migrate DAGs from Airflow 2.x to 3.x |\n\n## Why Astro?\n\nAstro is Astronomer's managed Airflow platform. It's optional, but a good fit if you want managed deployments, built-in alerting, and centralized observability across environments. If you run open-source Airflow, everything in this repo still applies—you'll just configure your own Airflow URL and infrastructure.\n\n### User Journeys\n\n#### Data Analysis Flow\n\n```mermaid\nflowchart LR\n    init[\"/astronomer-data:warehouse-init\"] --> analyzing[\"/astronomer-data:analyzing-data\"]\n    analyzing --> profiling[\"/astronomer-data:profiling-tables\"]\n    analyzing --> freshness[\"/astronomer-data:checking-freshness\"]\n```\n\n1. **Initialize** (`/astronomer-data:warehouse-init`) - One-time setup to generate `warehouse.md` with schema metadata\n2. **Analyze** (`/astronomer-data:analyzing-data`) - Answer business questions with SQL\n3. **Profile** (`/astronomer-data:profiling-tables`) - Deep dive into specific tables for statistics and quality\n4. **Check freshness** (`/astronomer-data:checking-freshness`) - Verify data is up to date before using\n\n#### DAG Development Flow\n\nFor open-source Airflow, use Docker Compose for local dev and the Helm chart for production (see `deploying-airflow`) instead of Astro setup skills.\n\n```mermaid\nflowchart LR\n    setup[\"/astronomer-data:setting-up-astro-project\"] --> authoring[\"/astronomer-data:authoring-dags\"]\n    setup --> env[\"/astronomer-data:managing-astro-local-env\"]\n    authoring --> testing[\"/astronomer-data:testing-dags\"]\n    testing --> debugging[\"/astronomer-data:debugging-dags\"]\n```\n\n1. **Setup** (`/astronomer-data:setting-up-astro-project`) - Initialize project structure and dependencies\n2. **Environment** (`/astronomer-data:managing-astro-local-env`) - Start/stop local Airflow for development\n3. **Author** (`/astronomer-data:authoring-dags`) - Write DAG code following best practices\n4. **Test** (`/astronomer-data:testing-dags`) - Run DAGs and fix issues iteratively\n5. **Debug** (`/astronomer-data:debugging-dags`) - Deep investigation for complex failures\n\n### Airflow CLI (`af`)\n\nThe `af` command-line tool lets you interact with Airflow directly from your terminal. Install it with:\n\n```bash\nuvx --from astro-airflow-mcp af --help\n```\n\nFor frequent use, add an alias to your shell config (`~/.bashrc` or `~/.zshrc`):\n\n```bash\nalias af='uvx --from astro-airflow-mcp af'\n```\n\nThen use it for quick operations like `af health`, `af dags list`, or `af runs trigger <dag_id>`.\n\nSee the [full CLI documentation](./astro-airflow-mcp/README.md#airflow-cli-tool) for all commands and instance management.\n\n> **Telemetry:** The `af` CLI collects anonymous usage telemetry to help improve the tool. Only the command name is collected (e.g., `dags list`), never the arguments or their values. Opt out with `af telemetry disable`.\n\n## Configuration\n\n### Warehouse Connections\n\nConfigure data warehouse connections at `~/.astro/agents/warehouse.yml`:\n\n```yaml\nmy_warehouse:\n  type: snowflake\n  account: ${SNOWFLAKE_ACCOUNT}\n  user: ${SNOWFLAKE_USER}\n  auth_type: private_key\n  private_key_path: ~/.ssh/snowflake_key.p8\n  private_key_passphrase: ${SNOWFLAKE_PRIVATE_KEY_PASSPHRASE}\n  warehouse: COMPUTE_WH\n  role: ANALYST\n  query_tag: claude-code\n  databases:\n    - ANALYTICS\n    - RAW\n```\n\n> [!IMPORTANT]\n> **How the `databases` list works:**\n> - **Optional for most connectors** (`snowflake`, `postgres`, `bigquery`) but **required for `sqlalchemy`**\n> - **For schema discovery** (`/astronomer-data:warehouse-init`): Determines which databases are scanned and included in the generated `.astro/warehouse.md`. Only databases listed here will be discovered. If omitted, no schema discovery will occur.\n> - **For query execution** (`/astronomer-data:analyzing-data`): The **first database** in the list becomes the default database context for the connection, but does NOT restrict which databases you can query. You can still access any database you have permissions for using fully-qualified table names (e.g., `OTHER_DB.SCHEMA.TABLE`).\n>\n> **Example:** If you configure `databases: [ANALYTICS, RAW]`:\n> - `ANALYTICS` becomes the default database for queries\n> - You can still query `PROD` with `SELECT * FROM PROD.PUBLIC.USERS`\n> - Only `ANALYTICS` and `RAW` will appear in warehouse schema documentation (run `/astronomer-data:warehouse-init --refresh` after adding `PROD` to include it)\n\n> [!NOTE]\n> The `account` field requires your Snowflake **account identifier** (e.g., `orgname-accountname` or `xy12345.us-east-1`), not your account name. Find this in your Snowflake console under Admin > Accounts.\n\nStore credentials in `~/.astro/agents/.env`:\n\n```bash\nSNOWFLAKE_ACCOUNT=myorg-myaccount  # Use your Snowflake account identifier (format: orgname-accountname or accountname.region)\nSNOWFLAKE_USER=myuser\nSNOWFLAKE_PRIVATE_KEY_PASSPHRASE=your-passphrase-here  # Only required if using an encrypted private key\n```\n\n**Supported databases:**\n\n| Type | Package | Description |\n|------|---------|-------------|\n| `snowflake` | Built-in | Snowflake Data Cloud |\n| `postgres` | Built-in | PostgreSQL |\n| `bigquery` | Built-in | Google BigQuery |\n| `sqlalchemy` | Any SQLAlchemy driver | Auto-detects packages for 25+ databases (see below) |\n\n<details>\n<summary>Auto-detected SQLAlchemy databases</summary>\n\nThe connector automatically installs the correct driver packages for:\n\n| Database | Dialect URL |\n|----------|-------------|\n| PostgreSQL | `postgresql://` or `postgres://` |\n| MySQL | `mysql://` or `mysql+pymysql://` |\n| MariaDB | `mariadb://` |\n| SQLite | `sqlite:///` |\n| SQL Server | `mssql+pyodbc://` |\n| Oracle | `oracle://` |\n| Redshift | `redshift://` |\n| Snowflake | `snowflake://` |\n| BigQuery | `bigquery://` |\n| DuckDB | `duckdb:///` |\n| Trino | `trino://` |\n| ClickHouse | `clickhouse://` |\n| CockroachDB | `cockroachdb://` |\n| Databricks | `databricks://` |\n| Amazon Athena | `awsathena://` |\n| Cloud Spanner | `spanner://` |\n| Teradata | `teradata://` |\n| Vertica | `vertica://` |\n| SAP HANA | `hana://` |\n| IBM Db2 | `db2://` |\n\nFor unlisted databases, install the driver manually and use standard SQLAlchemy URLs.\n\n</details>\n\n<details>\n<summary>Example configurations</summary>\n\n```yaml\n# PostgreSQL\nmy_postgres:\n  type: postgres\n  host: localhost\n  port: 5432\n  user: analyst\n  password: ${POSTGRES_PASSWORD}\n  database: analytics\n  application_name: claude-code\n\n# BigQuery\nmy_bigquery:\n  type: bigquery\n  project: my-gcp-project\n  credentials_path: ~/.config/gcloud/service_account.json\n  location: US\n  labels:\n    team: data-eng\n    env: prod\n\n# SQLAlchemy (any supported database)\nmy_duckdb:\n  type: sqlalchemy\n  url: duckdb:///path/to/analytics.duckdb\n  databases: [main]\n\n# SQLAlchemy with connect_args (passed to the DBAPI driver)\nmy_pg_sqlalchemy:\n  type: sqlalchemy\n  url: postgresql://${PG_USER}:${PG_PASSWORD}@localhost/analytics\n  databases: [analytics]\n  connect_args:\n    application_name: claude-code\n\n# Redshift (via SQLAlchemy)\nmy_redshift:\n  type: sqlalchemy\n  url: redshift+redshift_connector://${REDSHIFT_USER}:${REDSHIFT_PASSWORD}@${REDSHIFT_HOST}:5439/${REDSHIFT_DATABASE}\n  databases: [my_database]\n```\n\n</details>\n\n### Airflow\n\nThe Airflow MCP auto-discovers your project when you run Claude Code from an Airflow project directory (contains `airflow.cfg` or `dags/` folder).\n\nFor remote instances, set environment variables:\n\n| Variable | Description |\n|----------|-------------|\n| `AIRFLOW_API_URL` | Airflow webserver URL |\n| `AIRFLOW_USERNAME` | Username |\n| `AIRFLOW_PASSWORD` | Password |\n| `AIRFLOW_AUTH_TOKEN` | Bearer token (alternative to username/password) |\n\n## Usage\n\nSkills are invoked automatically based on what you ask. You can also invoke them directly with `/astronomer-data:<skill-name>`.\n\n### Getting Started\n\n1. **Initialize your warehouse** (recommended first step):\n   ```\n   /astronomer-data:warehouse-init\n   ```\n   This generates `.astro/warehouse.md` with schema metadata for faster queries.\n\n2. **Ask questions naturally**:\n   - \"What tables contain customer data?\"\n   - \"Show me revenue trends by product\"\n   - \"Create a DAG that loads data from S3 to Snowflake daily\"\n   - \"Why did my etl_pipeline DAG fail yesterday?\"\n\n## Development\n\nSee [CLAUDE.md](./CLAUDE.md) for plugin development guidelines.\n\n### Local Development Setup\n\n```bash\n# Clone the repo\ngit clone https://github.com/astronomer/agents.git\ncd agents\n\n# Test with local plugin\nclaude --plugin-dir .\n\n# Or install from local marketplace\nclaude plugin marketplace add .\nclaude plugin install astronomer-data@astronomer\n```\n\n### Adding Skills\n\nCreate a new skill in `skills/<name>/SKILL.md` with YAML frontmatter:\n\n```yaml\n---\nname: my-skill\ndescription: When to invoke this skill\n---\n\n# Skill instructions here...\n```\n\nAfter adding skills, reinstall the plugin:\n```bash\nclaude plugin uninstall astronomer-data@astronomer && claude plugin marketplace update && claude plugin install astronomer-data@astronomer\n```\n\n## Troubleshooting\n\n### Common Issues\n\n| Issue | Solution |\n|-------|----------|\n| Skills not appearing | Reinstall plugin: `claude plugin uninstall astronomer-data@astronomer && claude plugin marketplace update && claude plugin install astronomer-data@astronomer` |\n| Installed as `data@astronomer` (old name) | Uninstall old name and reinstall: `claude plugin uninstall data@astronomer && claude plugin marketplace update && claude plugin install astronomer-data@astronomer` |\n| Warehouse connection errors | Check credentials in `~/.astro/agents/.env` and connection config in `warehouse.yml` |\n| Airflow not detected | Ensure you're running from a directory with `airflow.cfg` or a `dags/` folder |\n\n## Contributing\n\nContributions welcome! Please read our [Code of Conduct](./CODE_OF_CONDUCT.md) and [Contributing Guide](./CONTRIBUTING.md) before getting started.\n\n## Roadmap\n\nSkills we're likely to build:\n\n**DAG Operations**\n- CI/CD pipelines for DAG deployment\n- Performance optimization and tuning\n- Monitoring and alerting setup\n- Data quality and validation workflows\n\n**Astronomer Open Source**\n- [DAG Factory](https://github.com/astronomer/dag-factory) - Generate DAGs from YAML\n- Other open source projects we maintain\n\n**Conference Learnings**\n- Reviewing talks from Airflow Summit, Coalesce, Data Council, and other conferences to extract reusable skills and patterns\n\n**Broader Data Practitioner Skills**\n- Churn prediction, data modeling, ML training, and other workflows that span DE/DS/analytics roles\n\n**Don't see a skill you want? [Open an issue](https://github.com/astronomer/agents/issues) or submit a PR!**\n\n## License\n\nApache 2.0\n\n---\n\nMade with :heart: by Astronomer\n<img referrerpolicy=\"no-referrer-when-downgrade\" src=\"https://static.scarf.sh/a.png?x-pxid=f1c1d270-334e-45ec-b711-77385036cff9\" />\n",
  "bytes": 18778,
  "sha": "9517ffa1311b0d7ae6616b62de72bf9275316c3925e4d09493d21918a237cde3",
  "repo_slug": "astronomer/agents",
  "fonte": "repo",
  "truncated": false,
  "api": "https://agentalog.com/api/listings/plg_astronomer_agents_data_engineering_a5e6fda6/readme"
}