// the find
microsoft/LoRA
Code for loralib, an implementation of "LoRA: Low-Rank Adaptation of Large Language Models"
The original Microsoft research repo that introduced LoRA, a technique for fine-tuning large language models by injecting trainable low-rank matrices into frozen weights. It ships a small `loralib` package with drop-in replacements for `nn.Linear`, `nn.Embedding`, and `nn.Conv2d`, plus reference implementations for GPT-2, RoBERTa, and DeBERTa. This is the paper artifact — read it to understand the idea, but don't use it as your production fine-tuning library.
The core idea is proven: 0.8M trainable parameters matching full fine-tuning of 125M on GLUE is a meaningful result with real benchmark numbers attached. The `model.eval()` weight-merging trick is clever — LoRA adapters fold into the base weights at inference time, so there's literally zero latency cost at serving. The `MergedLinear` layer for fused QKV projections handles the annoying real-world case where attention Q, K, V share a single linear. Bias training via the `bias=` flag is a small but useful knob that most re-implementations quietly drop.
This repo is effectively archived — last meaningful commit activity was 2022-2023, and Hugging Face PEFT has absorbed everything useful here and kept it maintained. The example code in `examples/NLU/` is a vendored snapshot of an old HuggingFace Transformers version, which means security patches and API changes from upstream never landed. There's no support for `nn.MultiheadAttention` directly, so adapting modern architectures requires manual surgery. No support for QLoRA (quantized base weights), which is now the baseline expectation for anyone fine-tuning on consumer hardware.