// the find
Netflix/genie
Distributed Big Data Orchestration Service
Genie is Netflix's internal system for abstracting away the complexity of submitting and running Big Data jobs (Spark, Hive, Presto, etc.) across multiple clusters. Users describe what they want to run; Genie resolves which cluster, which binary versions, and which configs to use, then executes and monitors the job. It's a real production system that Netflix has run at scale for years.
The agent/server split is well thought out — the agent runs as a CLI process on the worker node, communicating back to the server via gRPC, which means job execution is decoupled from server availability. The state machine driving job execution (see job-execution-state-machine.dot) is explicit and documented, not implicit if-else chains buried in a service class. The cluster/command/application resolution model is genuinely flexible — you can define routing rules that pick different clusters based on job tags or user without touching code. Test coverage uses Spock/Groovy specs with good separation between unit and integration concerns.
The Travis CI badges in the README link to a service that effectively shut down for open source in 2021 — a sign that the project's public-facing maintenance is on autopilot. The dependency on Spring State Machine was dropped in favor of a custom implementation, but the documentation hasn't caught up and still references the old architecture in places. Getting this running locally is genuinely painful — it requires Zookeeper, the right Hadoop/Spark binaries registered, and a working gRPC setup before you can submit a single job. There is no meaningful path for a team outside Netflix to adopt this without significant platform investment; it assumes you already have the surrounding Netflix OSS stack.