// the find
hi-primus/optimus
:truck: Agile Data Preparation Workflows made easy with Pandas, Dask, cuDF, Dask-cuDF, Vaex and PySpark
Optimus is a Python data wrangling library that wraps Pandas, Dask, cuDF, Vaex, and Spark behind a single API, so you can write one transformation chain and theoretically scale it from a laptop to a GPU cluster. It targets data scientists who want a chainable, method-rich API without learning five different backends. Think of it as a compatibility shim with opinionated ergonomics on top.
The unified API is the real value — swapping `Optimus('pandas')` for `Optimus('spark')` and having the same `.cols.lower()` chain work is genuinely useful for teams that need to prototype locally and run at scale. The `.cols` and `.rows` accessor pattern is clean and chainable, making multi-step transformations readable without intermediate variables. The string/date/URL function library (100+ operations) covers the messy real-world cleaning work that pandas makes verbose. Multiple format support (CSV, JSON, Parquet, Avro, Excel, JDBC) from one `.load.*` interface reduces boilerplate.
The project requires Python 3.7 or 3.8 per the README — both are years past end-of-life, which is a hard blocker for any serious production adoption in 2024+. Vaex itself is effectively abandoned upstream, making that backend a liability. Last push was December 2024 but the activity looks sparse, and the 1.5k stars for a library with this scope suggests it never got real traction against dbt, Polars, or just using Spark directly. The abstraction leaks constantly in practice — when cuDF or Spark behavior diverges from Pandas, you end up debugging the internals of whichever backend misbehaved, not the Optimus layer.