// the find
sfu-db/connector-x
Fastest library to load data from DB to DataFrames in Rust and Python
ConnectorX is a Rust library with Python bindings that loads data from SQL databases directly into DataFrames (Pandas, Arrow, Polars) using a zero-copy architecture. The core idea: pre-allocate the destination buffer using row count and schema info, then write directly into it from parallel query partitions without intermediate copies. It's for data engineers and ML practitioners who routinely pull large result sets from databases into Python and have hit pandas+sqlalchemy performance walls.
The zero-copy architecture is the real thing — it issues a COUNT before loading to pre-allocate memory, then streams each partition directly into the target buffer. This is why the 3x memory advantage over pandas is reproducible, not a benchmark trick. Parallel partitioning on a numeric column is simple to use and works well for wide, evenly-distributed tables. The Arrow/Polars path is the right default for modern pipelines: you get an Arrow RecordBatch without touching Python objects per row. The federated query feature (joining across two databases in one call) is genuinely useful for ETL work where data lives in multiple systems.
Partition column must be numeric and non-nullable — this is a hard limitation that rules out UUID, timestamp, or string-keyed tables without workarounds. The COUNT + MIN/MAX pre-queries add round trips that hurt on slow connections or when the optimizer doesn't handle the subquery well; there's no way to supply row count hints. ODBC support is still WIP, so anything not on the explicit source list requires forking or waiting. The federated query feature is marked experimental with no clear path to stable, and the cross-database SQL rewriting has obvious limits around dialect differences and pushdown correctness.