finds.dev

DocAILab/XRAG

XRAG: eXamining the Core - Benchmarking Foundational Component Modules in Advanced Retrieval-Augmented Generation

View on GitHubHomepage ↗

★ 597PythonApache-2.0updated Oct 2026

// the take

XRAG is a benchmarking toolkit for RAG pipelines: it lets you swap in different retrievers, embedding models, LLMs and chunking strategies, then score the combination with both classic IR metrics (F1, MRR, NDCG) and LLM-judged ones (faithfulness, relevancy). It's aimed at researchers and RAG builders who want to A/B test pipeline components rather than hand-roll eval scripts, and it backs a published ICDE 2026 paper.

The orchestrator abstraction (sequential, conditional, iterative, parallel, hybrid) is actually implemented as real pipelines — self-rag, adaptive-rag and SIM-rag are separate modules with their own prompts and retrievers, not just config flags. It covers both ends of evaluation: cheap traditional metrics for fast iteration and DeepEval/LLM-based metrics for the things traditional metrics miss (hallucination, contextual precision). It also doubles as a small service: `xrag-cli api` spins up a FastAPI server with /query and /health, and there's a Vue-based WebUI for walking through dataset → index → strategy → eval without touching config.toml by hand.

Several advertised eval metrics (Semantic Perplexity, Entropy, Auto-J, Prometheus, late chunking) are still unchecked roadmap items, not shipped — check before assuming a given metric works. The install instructions pin old, narrow version ranges (requests==2.27.1, urllib3==1.25.11, jiwer<4.0.0) alongside a separate `pip install jury --no-deps` workaround, which is a sign the dependency tree is fragile and will fight you in a modern environment. config.toml holds the API key in plaintext next to model settings, with no mention of secrets handling for the API server mode. The API server is single-engine/single-process with no auth on /query, fine for local experimentation but not something to expose beyond a research sandbox.

We read the README, the directory tree and a key source file in October 2026. Nobody pays to be listed. How we judge →

search the same ground:

// at a glance

Stars
597
Forks
17
Language
Python
License
Apache-2.0
Last commit
19 Oct 2026

// nearby

If XRAG isn't quite it.

All repos →