// the find
abacaj/fine-tune-mistral
Fine-tune mistral-7B on 3090s, a100s, h100s
A minimal full fine-tuning script for Mistral-7B using PyTorch FSDP across multiple GPUs. This is a research artifact — code the author used to produce one specific HuggingFace model — not a general-purpose training framework. Useful as a concrete starting point if you want to understand what a bare-minimum distributed training loop looks like without abstraction layers.
Uses FSDP natively rather than wrapping everything in a trainer class, so you can actually see what's happening. Includes a multipack sampler (packing multiple short sequences per batch) which meaningfully improves GPU utilization compared to naive padding. Tiny codebase — the entire training logic is readable in one sitting. Sample data is included so you can validate the pipeline runs before swapping in your own.
Last commit October 2023 — predates significant Mistral model updates and tooling improvements, so deps may have drifted. No checkpointing strategy beyond what FSDP provides by default, meaning a crash mid-run loses everything. Tips section is hand-wavy ('use enough data', 'experiment with epochs') rather than prescriptive. No evaluation loop built in — the README tells you to add one, then leaves you to figure out how.