// the find
illuin-tech/colpali
The code used to train and run inference with the ColVision models, e.g. ColPali, ColQwen2, and ColSmol.
ColPali is a document retrieval system that skips OCR entirely — instead of extracting text, it embeds document pages as images using vision-language models and retrieves them via ColBERT-style multi-vector late interaction. The idea is that a VLM already understands charts, tables, and layout, so why throw that away before indexing. It's for anyone building RAG pipelines over PDFs where traditional text extraction loses important structure.
The core insight holds up: treating pages as images sidesteps a whole class of OCR failures (multi-column layouts, embedded charts, scanned docs). The model zoo is genuinely useful — ColQwen2.5 at 89.4 on ViDoRe with Apache 2.0 license is production-viable without Gemma's restrictive terms. Token pooling with a pool factor of 3 cuts vectors by two-thirds while losing less than 3% retrieval quality, which matters a lot when you're storing embeddings per page patch. The Triton MaxSim kernel via `late-interaction-kernels` is a concrete engineering win — doubling max batch size on H100 without touching throughput is not a rounding error.
Multi-vector storage costs are real and the README barely mentions them: each page becomes hundreds of patch embeddings, and naive storage in Qdrant or pgvector gets expensive fast at scale. Inference requires a full VLM loaded in GPU memory — there's no CPU-friendly path and no ONNX export for the ColQwen models, so you're locked to GPU infra for serving. The dynamic resolution support in ColQwen2 is good but the MPS (Apple Silicon) incompatibility with torch 2.6 is a known regression that's not fixed upstream, just worked around by pinning to 2.5.1 — fragile for anyone using Apple hardware seriously. Training configs use a YAML+Python hybrid through `configue` that isn't well-documented outside the repo, making custom fine-tuning harder than it should be.