{
  "markdown": "# PySpark MCP Server\n\nSQL migration assistance, AWS Glue job *template* generation, and Spark code\noptimization — as an MCP server.\n\n> **Not the live-Spark `pyspark-mcp` package.** This project is SQL → PySpark /\n> Glue *source generation*, published as [`pyspark-tools`](https://pypi.org/project/pyspark-tools/).\n> [SemyonSinchenko/pyspark-mcp](https://pypi.org/project/pyspark-mcp/) introspects a\n> running SparkSession. A deprecated `pyspark-mcp` console script remains here so\n> old configs keep working; it prints a warning, then starts this server.\n\n[![CI Pipeline](https://github.com/AnnasMazhar/pyspark_mcp/actions/workflows/pr-validation.yml/badge.svg)](https://github.com/AnnasMazhar/pyspark_mcp/actions/workflows/pr-validation.yml)\n[![Python 3.11+](https://img.shields.io/badge/python-3.11+-blue.svg)](https://www.python.org/downloads/)\n[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)\n\n## What It Does\n\n- **SQL Dialect Transpilation** — Convert between PostgreSQL, Oracle, Redshift, MySQL, Snowflake, and Spark SQL using [SQLGlot](https://github.com/tobymao/sqlglot)\n- **PySpark DataFrame API Generation** — Generate DataFrame API *source text* from SQL, with optimization hints\n- **AWS Glue templates** — Job script strings, DynamicFrame conversions, Data Catalog definitions, S3 layout advice\n- **Batch Processing** — Walk SQL files/directories and emit converted modules\n- **Code Review & Optimization** — Pattern-based review of existing PySpark source\n- **Pattern Detection** — Find duplicated snippets and suggest utilities\n\n## What It Doesn't Do\n\n- Recursive CTEs → provides Spark SQL equivalent + guidance (PySpark has no native recursive CTE support)\n- MERGE/PIVOT/CONNECT BY → transpiles to Spark SQL, provides DataFrame API guidance\n- Perfect 1:1 DataFrame API transpilation for all SQL — complex queries get Spark SQL + recommendations\n- It does **not** start a SparkSession, submit Glue jobs, or execute SQL\n- `optimize(mode=\"code\")` returns **suggestions**; it does not rewrite your code\n- `glue_s3` is a **path heuristic** (no AWS call, no measured speedups)\n- It does **not** replace [SemyonSinchenko/pyspark-mcp](https://pypi.org/project/pyspark-mcp/) for live catalog/plans\n\n## Why this vs calling sqlglot yourself\n\nSQLGlot already transpiles dialects. This MCP adds three things around that kernel: DataFrame-API pretty-printing with join/window/cast mappings that the conversion tests lock, Glue job *boilerplate strings* (bookmarks, DynamicFrames, catalog tables) so an agent can emit a file instead of assembling one, and a 14-tool FastMCP surface so an LLM picks `convert` / `mode=sql` instead of wiring sqlglot itself. If you only need `sqlglot.transpile(...)`, use sqlglot.\n\n## Quick Start\n\n```bash\npip install pyspark-tools\npyspark-tools\n```\n\nZero-clone alternative: `uvx pyspark-tools`. `run_server.py` is a development convenience that inserts `sys.path` and prints startup banners. Prefer `pyspark-tools` in configs and production.\n\n## Try it\n\n```bash\npip install pyspark-tools\npython -c \"from pathlib import Path; from pyspark_tools.sql_converter import SQLToPySparkConverter as C; from pyspark_tools.consolidated_tools import glue_job; c,s,o=C(),Path('examples'),Path('examples/out'); [(o/f'{n}.py').write_text(c.convert_sql_to_pyspark((s/f'{n}.sql').read_text(), dialect=d).pyspark_code) for n,d in [('postgres_orders','postgres'),('oracle_decode','oracle')]]; (o/'orders_etl_glue.py').write_text(glue_job(mode='template', job_name='orders_etl', sql_query=(s/'postgres_orders.sql').read_text())['template'])\"\n```\n\nWrites the same files as `examples/out/`. MCP stdio CLI: `pyspark-tools`.\n\n## Example: SQL → PySpark\n\n```sql\nSELECT o.customer_id, c.name, SUM(o.amount) AS total\nFROM orders o\nJOIN customers c ON o.customer_id = c.id\nWHERE o.status = 'paid'\nGROUP BY o.customer_id, c.name\n```\n\nCall `convert` with `mode=sql`. Captured converter output (`dialect=spark`):\n\n```python\nfrom pyspark.sql import SparkSession\nfrom pyspark.sql.functions import (\n    col, lit, when, count, sum as spark_sum, avg, min, max, countDistinct,\n    coalesce, concat, datediff, date_add, to_date,\n    row_number, rank, lag, lead,\n)\nfrom pyspark.sql.window import Window\n\n# Generated from SPARK SQL\nspark = SparkSession.builder.appName('SQLToPySpark').getOrCreate()\n\n# Load table: customers\ncustomers_df = spark.table('customers')\n# Load table: orders\norders_df = spark.table('orders')\n\n# Main query\nresult_df = (orders_df.alias('o')\n    .join(customers_df.alias('c'), (col('o.customer_id') == col('c.id')), 'inner')\n    .filter((col('o.status') == lit('paid')))\n    .groupBy(col('o.customer_id'), col('c.name'))\n    .select(col('o.customer_id'), col('c.name'), (spark_sum(col('o.amount'))).alias('total')))\n```\n\nExact output depends on dialect detection and fallbacks; conversion tests in `tests/test_sql_conversion_fixes.py` pin the important constructs. Notebook-style `import *` / `show()` is opt-in via `style=\"notebook\"` on the converter.\n\n## MCP Configuration\n\n### Claude Desktop\n\nmacOS: `~/Library/Application Support/Claude/claude_desktop_config.json`\n\nLinux: `~/.config/Claude/claude_desktop_config.json`\n\n```json\n{\n  \"mcpServers\": {\n    \"pyspark\": {\n      \"command\": \"pyspark-tools\",\n      \"args\": []\n    }\n  }\n}\n```\n\n### Hermes Agent\n\nAdd to `~/.hermes/config.yaml`:\n\n```yaml\nmcp:\n  servers:\n    pyspark:\n      command: pyspark-tools\n      enabled_tools: all\n```\n\n### Docker\n\nThe image is **stdio only** (FastMCP over stdin/stdout). There is no HTTP server\non port 8000. `docker compose up` is for local tests, not a health-checkable\nweb service.\n\n```bash\ndocker compose --profile test run --rm pyspark-tools-test\n```\n\n## Tools\n\nThree primary tools. The other eleven routers stay registered this minor\nversion but are **deprecated** — prefer `convert`, `glue_job`, and `review`.\n\n### `convert` — SQL → PySpark (including `mode=batch_dir`)\n```python\nconvert(mode=\"sql\", sql_query=\"SELECT id FROM users\", dialect=\"postgres\")\nconvert(mode=\"batch_dir\", directory_path=\"etl/\", output_dir=\"out\")\n```\n\n### `glue_job` — Glue 5.0 job *template* strings\n```python\nglue_job(mode=\"template\", job_name=\"orders_etl\", sql_query=\"SELECT * FROM orders\")\n```\n\n### `review` — code review, patterns, duplicates\n```python\nreview(mode=\"code\", code=\"df = spark.table('t')\\ndf.collect()\")\n```\n\n**Legacy / deprecated:** `analyze`, `optimize`, `glue_schema`, `glue_s3`,\n`glue_data`, `refactor`, `search`, `context`, `batch_status`, `s3_source`,\n`analytics`. Still callable; do not advertise to new agents.\n\n## Security\n\nThis MCP can **read local files** (SQL, TXT, PDF) and, if the `[aws]` extra is installed, **list/read S3 with the host's default AWS credentials**. File tools only allow paths under the process working directory (or an explicit `base_path` / `FileHandler(base_directory=...)`). That is not a sandbox.\n\nRun the server under a restricted OS account. Do not point it at secrets directories. Do not attach AWS credentials with write access unless you intend S3 reads via `s3_source` / `glue_s3`. Optional extras:\n\n```bash\npip install \"pyspark-tools[aws]\"    # boto3 for S3/Glue catalog helpers\npip install \"pyspark-tools[spark]\"  # pyspark — not required at runtime; generated code only\n```\n\n## Development\n\n```bash\npython -m venv .venv\nsource .venv/bin/activate\npip install -e \".[dev]\"\n\n# Test\npytest tests/ -v --cov=pyspark_tools\n\n# Format\nblack pyspark_tools tests\nisort pyspark_tools tests\n\n# Lint\nflake8 pyspark_tools tests\n```\n\nRequires **Python 3.11+** (matches the CI matrix).\n\n## Architecture\n\n```\npyspark_tools/\n├── server.py              # FastMCP server + helper implementations\n├── consolidated_tools.py  # 14 @app.tool() routers\n├── sql_converter.py       # SQLGlot-based transpilation + DataFrame API generation\n├── aws_glue_integration.py # Glue job templates, DynamicFrame, Data Catalog\n├── advanced_optimizer.py  # Performance analysis + optimization suggestions\n├── batch_processor.py     # Concurrent file processing\n├── code_reviewer.py       # PySpark code review patterns\n├── duplicate_detector.py  # Code deduplication\n├── data_source_analyzer.py # Data source analysis (optional boto3)\n└── file_utils.py          # File I/O with allow-root checks\n```\n\n## License\n\nMIT — see [LICENSE](LICENSE).\n\n---\n`mcp-name: io.github.AnnasMazhar/pyspark-mcp`\n",
  "bytes": 8355,
  "sha": "a2b057a9c5fffa7eed894e5187bb78ef32378a712635ffaf2c8bf6171d44d2a3",
  "repo_slug": "annasmazhar/pyspark_mcp",
  "fonte": "repo",
  "truncated": false,
  "api": "https://agentalog.com/api/listings/mcp_io_github_annasmazhar_pyspark_mcp_d5079cbc/readme"
}