// the find
pyper-dev/pyper
Concurrent Python made simple
Pyper is a pipeline abstraction for Python that lets you chain functions across threads, processes, and asyncio tasks using a single decorator and the pipe operator. It handles the queue plumbing between stages so you don't have to. Aimed at data engineers who want concurrent ETL without writing their own thread pool management.
The unified API across all three concurrency models (asyncio, threading, multiprocessing) is genuinely useful — you can mix sync and async functions in the same pipeline without adapter boilerplate. Zero dependencies is a real advantage in environments where dependency sprawl is a problem. The lazy, queue-based execution means you don't materialize the whole dataset before processing starts, which matters for large streams. The automatic propagation of exceptions up through the pipeline avoids the classic silent-failure trap of manual thread workers.
Last commit was February 2025 and the project has 31 forks — the community is tiny and bus factor is effectively one. There's no built-in backpressure control; if a downstream stage is slow, the queue between stages will grow unbounded and eat your memory. Multiprocessing with `workers=20` means 20 *processes*, each with its own memory space — the README breezes past the serialization cost of passing data through multiprocessing queues, which will hurt if your objects aren't trivially picklable. There's also no observability: no metrics, no way to see queue depth or worker throughput without adding your own instrumentation.