finds.dev

cvs-health/uqlm

UQLM: Uncertainty Quantification for Language Models, is a Python package for UQ-based LLM hallucination detection

View on GitHubHomepage ↗

★ 1,166PythonApache-2.0updated Jun 2026

// the take

UQLM is a Python library from CVS Health for detecting LLM hallucinations using uncertainty quantification — sampling multiple responses and measuring consistency, using token probabilities, or routing through judge LLMs. It covers the full spectrum from cheap white-box token-probability methods to expensive black-box multi-sampling approaches, plus claim-level scoring for long text. Aimed at ML engineers who need to ship LLM applications with some confidence signal about when the model is making things up.

Published in JMLR and TMLR with papers backing the specific scorer implementations — this isn't vibes-based, it's grounded in the literature. The scorer taxonomy (black-box vs white-box vs judge vs ensemble) is genuinely useful and maps directly to the latency/cost tradeoffs you'll face in production. LangChain integration means you can swap in any supported model without changing the scoring code. The ensemble tuner that learns weights from ground truth is a practical touch — most UQ libraries stop at off-the-shelf scorers and leave calibration as your problem.

Everything is async-first, which is fine until you try to integrate it into a sync codebase or a notebook where async is a pain. The LangChain dependency is both the strength and the trap — you're now tied to LangChain's API churn, and if LangChain deprecates a model interface you'll be waiting on UQLM to catch up. Black-box methods that require 5+ LLM calls per query will multiply your inference costs in a way that's hard to explain to a budget owner. No streaming support mentioned, so you're waiting for the full response before scoring starts, which hurts perceived latency.

We read the README, the directory tree and a key source file in June 2026. Nobody pays to be listed. How we judge →

// at a glance

Stars
1,166
Forks
127
Language
Python
License
Apache-2.0
Last commit
8 Jun 2026

// nearby

If uqlm isn't quite it.

All repos →