// the find
pinterest/secor
Secor is a service implementing Kafka log persistence
Secor is a Pinterest-built Kafka consumer that persists messages to S3, GCS, Azure Blob, and Swift. It handles exactly-once delivery to object storage by tracking offsets in ZooKeeper, then batching and uploading local files once size or time thresholds are met. It's for teams running Kafka at scale who want durable, queryable archives without standing up something like Kafka Connect.
The exactly-once guarantee is real and well-thought-out — offset tracking through ZooKeeper with a two-phase approach prevents both data loss and duplication even when S3 eventual consistency was still a thing. Output format support is genuinely broad: Parquet, Avro, ORC, Thrift, Protobuf, plain delimited text — you can land data in a format Hive or Spark can read directly without a transform step. The upload policy configuration (size-based and time-based, independently tunable) is practical; most teams need both knobs. Production lineage is solid — Airbnb, Yelp, Zalando, Strava all ran this at real scale.
ZooKeeper as the coordination layer is a meaningful operational burden in 2026 — Kafka itself dropped the ZK dependency in 2.8, but Secor still requires it, so you're maintaining a ZK cluster just for this consumer. The last substantive commit activity suggests the project is in maintenance mode at best; Pinterest appears to have moved on internally. Kubernetes support exists but feels bolted on (separate README, Helm chart lives in deploys/ rather than a proper chart repo). There's no support for Kafka's native consumer group protocol for offset storage, which is the standard approach everywhere else.