finds.dev

hi-primus/optimus

Agile Data Preparation Workflows made easy with Pandas, Dask, cuDF, Dask-cuDF, Vaex and PySpark

View on GitHubHomepage ↗

★ 1,535PythonApache-2.0updated Dec 2024

// the take

Optimus is a Python data wrangling library that wraps Pandas, Dask, cuDF, Vaex, and Spark behind a single API, so you can write one transformation chain and theoretically scale it from a laptop to a GPU cluster. It targets data scientists who want a chainable, method-rich API without learning five different backends. Think of it as a compatibility shim with opinionated ergonomics on top.

The unified API is the real value — swapping `Optimus('pandas')` for `Optimus('spark')` and having the same `.cols.lower()` chain work is genuinely useful for teams that need to prototype locally and run at scale. The `.cols` and `.rows` accessor pattern is clean and chainable, making multi-step transformations readable without intermediate variables. The string/date/URL function library (100+ operations) covers the messy real-world cleaning work that pandas makes verbose. Multiple format support (CSV, JSON, Parquet, Avro, Excel, JDBC) from one `.load.*` interface reduces boilerplate.

The project requires Python 3.7 or 3.8 per the README — both are years past end-of-life, which is a hard blocker for any serious production adoption in 2024+. Vaex itself is effectively abandoned upstream, making that backend a liability. Last push was December 2024 but the activity looks sparse, and the 1.5k stars for a library with this scope suggests it never got real traction against dbt, Polars, or just using Spark directly. The abstraction leaks constantly in practice — when cuDF or Spark behavior diverges from Pandas, you end up debugging the internals of whichever backend misbehaved, not the Optimus layer.

We read the README, the directory tree and a key source file in June 2026. Nobody pays to be listed. How we judge →

// at a glance

Stars
1,535
Forks
232
Language
Python
License
Apache-2.0
Last commit
2 Dec 2024

// nearby

If optimus isn't quite it.

All repos →