LoRA
LoRA (low-rank adaptation) fine-tunes a model by training small adapter matrices instead of all its weights.
LoRA, short for low-rank adaptation, is a fine-tuning method that freezes a pretrained model's original weights and trains a small pair of low-rank matrices alongside them instead. It adds these adapter matrices at each targeted layer, typically the attention projections. During inference, their output is summed with the frozen layer's output, so the base model itself never changes.
A 7 billion parameter model has billions of weights. A LoRA adapter at rank 8 adds roughly 4 million to 8 million trainable parameters, well under 1 percent of the total. That drops fine-tuning VRAM from the 80 GB or more a full fine-tune takes down to what fits on a single 24 GB card. The optimizer only tracks state for the small adapter, not for every original weight.
For anyone specing hardware for fine-tuning rather than pretraining, LoRA is usually why a single workstation GPU is enough instead of a multi-GPU server.
Sources
Source | Publisher |
|---|---|
arXiv | |
Hugging Face | |
Microsoft |
- Publisher
arXiv
- Publisher
Hugging Face
- Publisher
Microsoft
Last verified August 29, 2026.
- LoRA
- low-rank adaptation
- lora fine tuning
- PEFT
- parameter efficient fine-tuning
- adapter weights
- QLoRA
- fine-tuning VRAM