// the find
dotnet/spark
.NET for Apache® Spark™ makes Apache Spark™ easily accessible to .NET developers.
.NET for Apache Spark is a C#/F# binding layer that lets you write Spark jobs without touching Scala or Python. It works by running a worker process on each Spark executor that communicates with the JVM over a socket, using Apache Arrow for columnar data and Pyrolite (pickle protocol) for row-major data. The target is .NET shops that want to use Spark without context-switching to the JVM ecosystem.
Using Apache Arrow for bulk columnar transfers is the right call — it avoids per-row serialization costs for DataFrame operations and matches how PySpark does it. F# support is a genuine differentiator; functional-style data pipelines in F# read better than the equivalent Scala in many cases. Deployment coverage is thorough: HDInsight, Databricks on both clouds, and EMR all have documented paths, which matters when your org already has one of these provisioned. The TPC-H benchmarks in the repo give you a concrete performance baseline to test against rather than vague claims.
The JVM bridge is load-bearing infrastructure that adds a failure mode Spark-native setups don't have — you're now operating a socket server on every executor. .NET UDFs cross this boundary with serialization overhead on every call, so the performance penalty for UDF-heavy workloads is real and the docs understate it. The project has 2094 stars for a Microsoft-backed repo, which signals limited adoption; the community is thin and the SPIP for native Spark support (filed in 2019) has gone nowhere, meaning this stays a workaround indefinitely. Latest supported version caps at Spark 3.5 with no mention of 4.0, and the release cadence suggests catching up to Spark releases will always lag.