// the find
digitalocean/firebolt
Golang framework for streaming ETL, observability data pipeline, and event processing apps
A Go framework from DigitalOcean for building Kafka-based streaming pipelines as a tree of config-wired nodes — logging/observability pipelines, straight-through ETL, event processing. It's for teams who want a Kafka-in, Kafka/Elasticsearch-out pipeline with ops concerns handled, not for anything needing joins, windowing, or stateful aggregation.
The config-driven node tree with a type registry is a genuinely clean pattern: nodes implement one of three small interfaces (sync/fanout/async), get wired via YAML, and stay independently testable since they don't know about each other. The operational stuff you'd normally bolt on yourself is built in — per-node Prometheus metrics with zero extra code, a dead-letter Kafka topic for failed events, and outage recovery that replays the gap in parallel (rate-limited) while continuing to process realtime data. Zookeeper leader election is included for the common case of needing exactly one instance to run some task in a cluster. Test setup is solid — unit and integration suites, generated mocks, CI with coverage and go report card badges.
Hard CGo dependency on librdkafka for both compiling and running — you're apt-installing a C library before `go build` even works, which complicates cross-compilation and multi-stage Docker builds. Out of the box you only get two sinks (Kafka producer, Elasticsearch) and one source (Kafka consumer); anything else — Postgres, HTTP, SQS — is on you to write and register. Last push was June 2024, over two years ago as of now, with no visible maintenance since, so any librdkafka or dependency CVEs land on you to patch yourself. It's explicitly single-pass/stateless by design — no grouping, windowing, or sorting — which is a legitimate scope choice but rules it out the moment you need any real stream aggregation.