finds.dev

joowani/kq

Kafka-based Job Queue for Python

View on GitHubHomepage ↗

★ 573PythonMITupdated Feb 2022

// the take

KQ wraps Kafka topics as a job queue, letting you enqueue arbitrary Python function calls and execute them asynchronously with a worker process. It's for teams already running Kafka who want task-queue semantics without adding Redis or RabbitMQ. Thin abstraction — essentially serializes the callable + args with dill and shoves it into a topic.

Uses Kafka's native consumer groups for worker scaling and partition-level ordering with zero extra coordination logic. The API is deliberately minimal — Queue and Worker are the entire surface area, which means less to learn and less to break. Supports per-job timeouts and custom serializers if dill isn't what you want. CI setup is clean with CodeQL scanning and codecov.

Dead since February 2022 — Kafka client ecosystem has moved on and kafka-python itself has had maintenance gaps. No retry logic or dead-letter queue built in; a failed job is just logged and dropped unless you wire that yourself. Serializing callables with dill is fragile — function signature changes between enqueue and execution will silently produce wrong behavior or crashes. No priority queue support at all; Kafka topics are append-only so you can't jump a job to the front without topic-level workarounds.

We read the README, the directory tree and a key source file in June 2026. Nobody pays to be listed. How we judge →

// at a glance

Stars
573
Forks
23
Language
Python
License
MIT
Last commit
4 Feb 2022

// nearby

If kq isn't quite it.

All repos →