// the find
mosecorg/mosec
A high-performance ML model serving framework, offers dynamic batching and CPU/GPU pipelines to fully exploit your compute machine
Mosec is a Python-facing ML model serving framework with a Rust core handling the HTTP layer, dynamic batching, and inter-process communication. It's aimed at teams who want to take a trained PyTorch/JAX/TF model and expose it as a production API without writing a lot of glue code. The Python interface is deliberately thin — you subclass Worker, implement forward(), and the Rust side handles the rest.
The Rust HTTP and task coordination layer is a legitimate architectural choice, not marketing — it means you get async I/O and low per-request overhead without touching asyncio in your model code. Dynamic batching is configurable per-stage with both a size cap and a timeout, which is the right abstraction for GPU workloads where you need to amortize kernel launch costs. The pipeline model (chain workers into stages with IPC between them) maps cleanly onto the CPU-preprocess → GPU-infer → CPU-postprocess pattern that actually shows up in production. Prometheus metrics are built in and expose batch size histograms, which is exactly what you need to tune max_batch_size in practice.
Under 1000 stars after being around since 2021 suggests it hasn't broken through beyond the TensorChord ecosystem — you're betting on a framework with a small community and limited third-party resources if you hit an edge case. The multi-stage IPC story requires serialization between every stage boundary, and the shared-memory workaround (RedisShmIPCMixin) adds an operational dependency on Redis for something that should be local. There's no built-in model versioning, A/B routing, or canary deployment story — you'd have to bolt that on yourself or put it in front of a load balancer. The typed worker mixin for request validation generates OpenAPI docs, but it's a custom type annotation system rather than standard Pydantic, so it won't play well with existing validation tooling.