// the find
joowani/kq
Kafka-based Job Queue for Python
KQ wraps Kafka topics as a job queue, letting you enqueue arbitrary Python function calls and execute them asynchronously with a worker process. It's for teams already running Kafka who want task-queue semantics without adding Redis or RabbitMQ. Thin abstraction — essentially serializes the callable + args with dill and shoves it into a topic.
Uses Kafka's native consumer groups for worker scaling and partition-level ordering with zero extra coordination logic. The API is deliberately minimal — Queue and Worker are the entire surface area, which means less to learn and less to break. Supports per-job timeouts and custom serializers if dill isn't what you want. CI setup is clean with CodeQL scanning and codecov.
Dead since February 2022 — Kafka client ecosystem has moved on and kafka-python itself has had maintenance gaps. No retry logic or dead-letter queue built in; a failed job is just logged and dropped unless you wire that yourself. Serializing callables with dill is fragile — function signature changes between enqueue and execution will silently produce wrong behavior or crashes. No priority queue support at all; Kafka topics are append-only so you can't jump a job to the front without topic-level workarounds.