// the find
coleam00/mcp-crawl4ai-rag
Web Crawling and RAG Capabilities for AI Agents and AI Coding Assistants
An MCP server that pairs Crawl4AI's web scraping with pgvector-backed RAG, so AI agents can crawl documentation sites and query the results semantically. It's aimed at AI coding assistants that need to fetch and search external docs at runtime. The optional Neo4j hallucination-detection layer is the most novel piece — it parses a repo's AST into a graph and validates AI-generated code against it.
The feature flag system (USE_HYBRID_SEARCH, USE_CONTEXTUAL_EMBEDDINGS, etc.) lets you trade cost for quality incrementally instead of committing to the full stack upfront. Hybrid search — vector + keyword in parallel with result merging — is the right default for technical content where exact method names matter. The agentic RAG mode that extracts and separately indexes code blocks ≥300 chars is a genuinely useful split: prose and code have different retrieval patterns and deserve different tables. Docker-first setup with a working Dockerfile means you can have this running in a Cursor/Claude workflow inside 20 minutes.
Hard-wired to OpenAI for embeddings — there's no adapter layer, so swapping to a local model means editing source code, not flipping a config flag. The author explicitly says the knowledge graph feature isn't Docker-compatible yet, which breaks the 'use Docker for everything' recommendation in the same README. Supabase is required even for local use; there's no path to a plain Postgres instance without modifying the Supabase client calls. The project is self-described as a testbed for a larger system (Archon), meaning issues and PRs are low priority and the API surface will shift.