// the find
tansu-io/tansu
Apache Kafka® compatible broker with S3, PostgreSQL, SQLite, Apache Iceberg and Delta Lake
Tansu is a stateless Apache Kafka-compatible broker written in Rust that delegates all persistence to a pluggable backend — PostgreSQL, SQLite, S3, or memory. The pitch is that you get Kafka wire compatibility without running Kafka, and schema-validated topics (Avro/Protobuf/JSON) are automatically materialized as Parquet files for direct query via DuckDB or Spark. Aimed at teams that need Kafka's producer/consumer ecosystem but not its operational weight.
The stateless broker design is the real idea here — state lives entirely in the storage backend, so the broker process itself is disposable. Restart it, run multiple instances, no ZooKeeper or KRaft ceremony. The data lake integration is genuinely useful: if your topic has a schema, you get Parquet files in S3 or Iceberg tables for free, which means your streaming data is directly queryable without a separate ETL step. The fuzz targets for protocol decode show they're taking wire compatibility seriously — Kafka's binary protocol is the kind of thing that breaks in subtle ways and fuzz testing is the right call. Single statically linked binary with broker, proxy, and CLI is exactly the right deployment model for something trying to displace a complex JVM stack.
The README admits 'all brokers are node 111' — there's no real cluster mode. You get one broker, and if you need HA, that's entirely the storage backend's problem. For S3 this means leaning on conditional writes, which works but caps your write throughput in a way real Kafka doesn't. No published benchmarks is a red flag for something competing with Kafka, which can sustain millions of messages per second; PostgreSQL as a write-ahead log is going to hit a ceiling well before that. Kafka API compatibility is notoriously hard to achieve fully — the message directory shows the right types but implementation gaps (especially around consumer group rebalancing edge cases and exactly-once semantics) tend to surface only under production load with specific client libraries. The 1794 stars with only 68 forks suggests this is being watched and evaluated, not adopted yet.