// the find
JudgmentLabs/judgeval
The Continuous-Improvement Stack for Agents. Our environment data and evals power agent improvement and monitoring.
Judgeval is an agent observability and evaluation SDK built on OpenTelemetry. You instrument your agent with a decorator, define LLM-based judges to score behaviors, and run those judges against live traffic or historical traces. Aimed at teams shipping production agents who need something between printf debugging and a full APM platform.
OpenTelemetry as the tracing foundation is the right call — it means your spans flow into existing Datadog/Grafana setups without fighting a proprietary format. The 'behaviors' model (structured, labeled outputs from judges that accumulate over time) is a more useful abstraction than raw pass/fail scores. Auto-instrumentation for the major providers (OpenAI, Anthropic, Google GenAI) means you get token usage and latency without wiring it by hand. The offline test runner lets you replay production traces against a new judge or prompt version, which is the actual workflow you need before shipping a change.
The core product is a thin client that ships data to a hosted backend — JUDGMENT_API_KEY and JUDGMENT_ORG_ID are required from the first line of the quickstart, so 'open source' here means 'open source SDK for a closed platform.' There's no self-hosted server option documented, which is a dealbreaker for anyone with data residency requirements. The 'agent judges' concept requires you to write good evaluation prompts, and there's no guidance or tooling for calibrating judge reliability — you can easily end up with a judge that scores inconsistently at 70% accuracy and doesn't know it. The LangGraph integration directory is present but empty in the tree, which suggests the framework coverage is less complete than the feature list implies.