// the find
truera/trulens
Evaluation and Tracking for LLM Experiments and AI Agents
TruLens is an evaluation and observability framework for LLM applications, built around OpenTelemetry tracing and LLM-as-judge feedback functions. It targets teams who have shipped a RAG or agent and now need to know if it actually works — not just vibes-check it. Backed by Snowflake/TruEra, which explains the strong Cortex and Snowflake integrations throughout.
The OTEL foundation is the right call — it means traces are exportable to Jaeger, Datadog, or anything OTLP-compatible without being locked into TruLens's own dashboard forever. The seven agentic evaluators (LogicalConsistency, ToolSelection, PlanAdherence, etc.) are specific enough to be useful, unlike the usual vague 'faithfulness' catch-all. The Selector API for targeting arbitrary span attributes gives you precise control over what gets fed into each metric, which most eval frameworks get wrong by hardcoding input/output only. Provider breadth is solid — OpenAI, Bedrock, Gemini, LiteLLM, and Cortex all ship as first-class packages rather than afterthoughts.
The package split (trulens-core, trulens-providers-*, trulens-apps-*) sounds modular but in practice you'll spend 20 minutes figuring out which combination you need before writing a line of eval code. The LLM-as-judge approach inherits all the usual problems — scores are noisy, model-dependent, and expensive; there's no built-in calibration against human labels to tell you whether your groundedness score means anything. The Snowflake Cortex integration feels like the primary commercial use case, and non-Snowflake users will notice the bias in the examples and docs. Migration history is messy — there's a trulens_eval deprecation layer, a grit migration script, and a DEPRECATION.md, which signals the API has broken on users at least once already.