// the find
dpkp/kafka-python
Python client for Apache Kafka
kafka-python is a pure-Python Kafka client covering consumer, producer, and admin operations. Version 3.0 rewrote the protocol layer to generate encode/decode classes directly from Kafka's own JSON schema definitions, which keeps it honest against broker changes. It's for Python shops that need Kafka integration without pulling in a JVM or a Rust extension.
Protocol stack auto-generated from Kafka's own schema JSON means less drift between client and broker behavior over time. No mandatory C/Cython/Rust core — installs cleanly anywhere Python runs, including stripped-down containers. SASL coverage is genuinely broad: PLAIN, SCRAM, GSSAPI, OAuth, MSK IAM, and even SSPI for Windows. The admin CLI (`kafka-python admin`) is a real JVM-free replacement for the bin/ scripts, which is useful when you're debugging in environments where the Kafka distribution isn't installed.
No async-native API — KafkaConsumer and KafkaProducer are synchronous, so you end up blocking a thread per consumer in any async Python service, which is awkward with asyncio. The repo had a long maintenance gap before the 3.x revival; there's accumulated ecosystem debt around things like Schema Registry integration (nothing built-in) and Avro/Protobuf deserialization. Throughput ceiling is noticeably lower than confluent-kafka-python, which wraps librdkafka — for high-volume pipelines this matters and the README's own `crc32c` tip is a workaround, not a solution. The `kafka/protocol/old/` directory still ships in the tree, which signals incomplete cleanup and raises questions about what's actually load-bearing.