// the find
Nike-Inc/koheesio
Python framework for building efficient data pipelines. It promotes modularity and collaboration, enabling the creation of complex pipelines from simple, reusable components.
Koheesio is Nike's internal PySpark/Pydantic framework for structuring ETL pipelines as composable, typed Steps. It's aimed at data engineers building Databricks/Spark pipelines who want software-engineering discipline (type safety, testability, structured logging) applied to their data code. Think of it as an opinionated wrapper that adds Pydantic validation and a common interface on top of PySpark transforms.
Pydantic-based Step model gives you free input validation and structured configuration — no more dict-passing soup between pipeline stages. The SCD (slowly changing dimension) writer and Delta integration are production-grade; the scd.py implementation handles Type 1/2 merges that most teams end up writing from scratch. Test utilities and a `DummyStep` pattern make unit testing pipeline logic genuinely easy without needing a full Spark session. Integrations cover enough of the real enterprise stack (Snowflake, Tableau Hyper, Box, SFTP, Kafka) that you might not need to bolt on extra glue.
Heavily Databricks-oriented despite claiming framework-neutrality — the most useful writers and readers are in `koheesio.integrations.spark.databricks`, and Spark Connect support is described as 'partial' outside Databricks until PySpark 4.0. At 814 stars and 44 forks, the community is thin; this looks like Nike's internal tooling that was open-sourced rather than built for broad adoption, and the contributor base reflects that. No support for streaming-first architectures — async is an afterthought (`asyncio/http.py` exists but the framework is structured around batch Steps). The `Context` component is a glorified dict with YAML loading, which sounds useful but in practice becomes the same ambient config-object antipattern it was meant to avoid.