// the find
elixir-broadway/broadway
Concurrent and multi-stage data ingestion and data processing with Elixir
Broadway is a framework from Dashbit for building concurrent, multi-stage data ingestion pipelines in Elixir, built on top of GenStage. It's for teams already on the BEAM who need to consume from SQS, Kafka, PubSub, or RabbitMQ with back-pressure, batching, and acknowledgement handling done for them instead of hand-rolled.
Back-pressure and fault tolerance come from GenStage and OTP supervision, not custom retry logic bolted on top — that's a real structural advantage over most queue-consumer libraries in other languages. The producer/processor/batcher split with per-stage concurrency config is genuinely simple: the SQS example is 20 lines and reads like config, not a callback maze. It's had a stable v1.0 API since 2019 with Dashbit backing (same team as Flow and Nx), so this isn't a library that's going to churn its interface under you. Built-in testing support and automatic acknowledgement at pipeline end remove two things people usually get wrong when building this themselves.
It's Elixir-only — no story here if the rest of your stack isn't BEAM, so this only fits teams already committed to the ecosystem. The producers for SQS, Kafka, PubSub, and RabbitMQ all live in separate repos (broadway_sqs, etc.), so 'batteries included' isn't quite true — you're pulling in and trusting the maintenance of additional packages beyond this core one. Metrics and rate-limiting are listed as built-in features but the README gives zero detail on either; you have to go to hexdocs to find out what telemetry events actually exist. There's no guidance in the README on dead-letter or poison-message handling beyond 'custom failure handling' as a bullet point — that's the part that actually bites people in production and it's left as an exercise.