// the find
pyspark-ai/pyspark-ai
English SDK for Apache Spark
pyspark-ai lets you write English instructions to generate PySpark transformations, plots, and UDFs via an LLM backend (defaulting to GPT-4). It patches DataFrame with an `.ai` accessor so you can call `df.ai.transform('what are the top sellers per category')` and get back a working Spark query. It's aimed at data analysts who know what they want but don't want to write Spark SQL.
The monkey-patching approach via `.activate()` is genuinely clever — it slots into existing PySpark workflows without changing how you create or pass DataFrames. The file-based caching layer means repeated identical queries don't re-hit the LLM, which matters when you're iterating interactively. The optional vector similarity search for improving transform accuracy shows real thinking about the failure mode where column name mismatches produce wrong queries. LangChain abstraction means you can swap in Azure OpenAI or any other provider without touching the core logic.
Last commit was June 2024 and the project looks stalled — no Databricks-sponsored activity and GPT-4 is still the recommended model by name in the README, which is now two model generations behind. The LLM-generated code runs via `exec()` in a Python executor with no sandboxing, so a confused or adversarial prompt could run arbitrary code in your Spark driver. The dependency on LangChain adds significant version friction — LangChain's API has broken compatibility repeatedly and the pinned versions in poetry.lock will conflict with anything modern. No schema awareness at query time means the agent can hallucinate column names on wide DataFrames.