finds.dev

digitalocean/firebolt

Golang framework for streaming ETL, observability data pipeline, and event processing apps

View on GitHub

★ 720GoNOASSERTIONupdated Jun 2024

// the take

A Go framework from DigitalOcean for building Kafka-based streaming pipelines as a tree of config-wired nodes — logging/observability pipelines, straight-through ETL, event processing. It's for teams who want a Kafka-in, Kafka/Elasticsearch-out pipeline with ops concerns handled, not for anything needing joins, windowing, or stateful aggregation.

The config-driven node tree with a type registry is a genuinely clean pattern: nodes implement one of three small interfaces (sync/fanout/async), get wired via YAML, and stay independently testable since they don't know about each other. The operational stuff you'd normally bolt on yourself is built in — per-node Prometheus metrics with zero extra code, a dead-letter Kafka topic for failed events, and outage recovery that replays the gap in parallel (rate-limited) while continuing to process realtime data. Zookeeper leader election is included for the common case of needing exactly one instance to run some task in a cluster. Test setup is solid — unit and integration suites, generated mocks, CI with coverage and go report card badges.

Hard CGo dependency on librdkafka for both compiling and running — you're apt-installing a C library before `go build` even works, which complicates cross-compilation and multi-stage Docker builds. Out of the box you only get two sinks (Kafka producer, Elasticsearch) and one source (Kafka consumer); anything else — Postgres, HTTP, SQS — is on you to write and register. Last push was June 2024, over two years ago as of now, with no visible maintenance since, so any librdkafka or dependency CVEs land on you to patch yourself. It's explicitly single-pass/stateless by design — no grouping, windowing, or sorting — which is a legitimate scope choice but rules it out the moment you need any real stream aggregation.

We read the README, the directory tree and a key source file in October 2026. Nobody pays to be listed. How we judge →

// at a glance

Stars
720
Forks
50
Language
Go
License
NOASSERTION
Last commit
14 Jun 2024

// nearby

If firebolt isn't quite it.

All repos →