// the find
mars-project/mars
Mars is a tensor-based unified framework for large-scale data computation which scales numpy, pandas, scikit-learn and Python functions.
Mars is a distributed computation framework that mirrors the APIs of numpy, pandas, and scikit-learn so you write mostly the same code but get chunked, parallel execution across cores or a cluster. It targets people already comfortable with the PyData stack who have outgrown a single machine but don't want to rewrite everything in Spark or learn Dask's lazy graph model.
The API fidelity is the real selling point — `mt.tensor`, `md.DataFrame`, and `mars.learn` are close enough to their originals that porting existing scripts is often a search-and-replace plus an `.execute()` call, which is a much lower migration cost than Dask's partial pandas subset. Eager mode is a nice touch for debugging: you can flip to immediate execution during development and back to lazy/graph mode for production runs. It has two genuinely different deployment stories — bare-metal supervisor/worker processes or riding on an existing Ray cluster — so it's not locked into one infra pattern. The benchmark numbers in the README (3-4x on a laptop for tensor/dataframe ops) are at least plausible given the chunking approach, not just marketing.
Last push was January 2024 — this project looks dead or close to it, which is the biggest issue for anyone considering adopting it today over Dask or Ray Data, both of which are actively maintained and have far larger communities to absorb bugs. Pandas-compatibility claims from projects like this always have gaps in practice (multi-index operations, certain groupby/merge edge cases), and with no recent commits there's no reason to expect unimplemented corners get filled in. The integration list (TensorFlow, PyTorch, XGBoost, LightGBM, joblib, statsmodels) is broad on paper but the README gives no sense of how deep each integration actually goes versus being thin wrappers. If you hit a wall with an unsupported operation or a scaling bug, there's no active maintainer pipeline to fix it — you're on your own with a project that already stopped moving two years ago.