// the find
mbzuai-oryx/GeoChat
[CVPR 2024 🔥] GeoChat, the first grounded Large Vision Language Model for Remote Sensing
GeoChat fine-tunes LLaVA-1.5/Vicuna on a 318k-pair instruction dataset to do grounded VLM tasks on remote-sensing imagery: image/region captioning, VQA, scene classification, and referring object detection with rotated bounding boxes for overhead shots. It's for researchers who want a training and eval starting point for domain-specific grounded VLMs, not anyone looking for a model they can just call.
Ships the full pipeline, not just inference code — pretrain, LoRA/QLoRA finetune scripts, DeepSpeed ZeRO-2/3 configs, and an eval suite across 7 benchmarks with greedy decoding for reproducibility. The grounding output format (text interleaved with rotated bounding boxes, plus referring-expression and region-caption tasks) is a real point of difference from a generic VLM demo, since rotated boxes actually matter for overhead imagery. Built directly on the LLaVA-1.5/Vicuna stack, so anyone already familiar with that codebase will recognize the structure immediately.
Compiled .pyc files for four different Python versions are checked into git under geochat/model/language_model/mpt/__pycache__ — nobody set up a .gitignore before pushing, which is a bad sign for repo hygiene generally. The documented training requirement is 3x A100 40GB for ~25 hours with no quantized or consumer-GPU path spelled out; the QLoRA script exists but isn't documented. The instruction dataset ships as multiple split zip files on HuggingFace that you manually cat together before unzipping — an easy step to get wrong with no validation. No commits since Nov 2024, no CI, no tests: this is a frozen paper artifact, so expect to fix transformers/flash-attn version conflicts yourself rather than filing an issue and getting a fix.