// the find
raga-ai-hub/RagaAI-Catalyst
Python SDK for Agent AI Observability, Monitoring and Evaluation Framework. Includes features like agent, llm and tools tracing, debugging multi-agentic system, self-hosted dashboard and advanced analytics with timeline and execution graph view
RagaAI Catalyst is a Python SDK for observing, tracing, and evaluating LLM and multi-agent applications. It covers the full LLMOps stack: traces, evals, guardrails, prompt management, red-teaming, and synthetic data generation, all tied to a hosted dashboard. It targets teams running production AI agents who need more visibility than print statements provide.
Auto-instrumentation covers the frameworks people actually use (LangChain, LlamaIndex, CrewAI, LangGraph, smolagents, OpenAI Agents SDK) with minimal boilerplate — one init_tracing() call and you get spans. The agentic tracer captures per-span cost and token counts at the tool level, not just the top-level call, which is where the money actually goes. Red-teaming is built-in and supports custom detectors alongside the standard ones, so you can add domain-specific checks without forking anything. The decorator pattern (@trace_llm, @trace_tool, @trace_agent) gives you structured attribution without requiring a specific framework.
Everything funnels through the RagaAI hosted backend — there's a self-hosted dashboard mentioned in the description but the README goes straight to cloud credentials with no self-host setup path documented, which is a deal-breaker for regulated environments. The evaluation metrics are hardcoded to OpenAI/GPT-4o-mini in every code example, with no obvious path to run them against a local or open-source judge. Last push was February 2026 and the repo contains a redteaming_old.py and llm_generator_old.py sitting in the production package directory, which is not confidence-inspiring. The test suite requires live credentials to run anything meaningful, making CI on forks effectively impossible.