// the find
zerodha/dungbeetle
A distributed job server built specifically for queuing and executing heavy SQL read jobs asynchronously. Separate out reporting layer from apps. MySQL, Postgres, ClickHouse.
DungBeetle is a Go binary that queues SQL read jobs (think: report generation) via Redis/AMQP and writes results into ephemeral tables in a separate MySQL/Postgres results database. It's from Zerodha, the Indian fintech firm, so it's been battle-tested against real reporting load at scale. The target user is a backend engineer whose app is getting hammered by user-initiated reports against a large OLAP or operational database.
1. The results-as-separate-tables pattern is genuinely clever — downstream clients can sort/filter/paginate results without touching the source DB at all. 2. SQL tasks are defined as plain `.sql` files with goyesql annotations, so non-Go engineers can add or modify reports without touching application code. 3. Queue and concurrency are first-class: you can run high-priority and low-priority worker pools independently on the same broker, and job routing is done at the task definition level via `-- queue:` tags. 4. Single static binary with no runtime dependencies beyond the broker — straightforward to deploy as a sidecar or a dedicated service.
1. Results backend is limited to MySQL and Postgres — no Redis, S3, or anything schemaless, so you're paying Postgres write overhead for every report result even if the consumer just downloads a CSV once. 2. The TTL field is documented as not supported by the default SQL backend, which means result cleanup is your problem; there's no built-in expiry mechanism for the ephemeral result tables. 3. MySQL query cancellation doesn't actually stop the DB-side query — the server abandons the connection but MySQL keeps running it, so misbehaving long queries on MySQL will eat DB resources regardless of what the job server does. 4. The test file is `server_test.go.bak` — it was renamed out of the build, meaning CI is not actually running integration tests against the server logic.