LoRA: Low-Rank Adaptation of Large Language Models

Edward J. Hu et al. · arXiv

TL;DR

Full fine-tuning copies every weight of a large model for each new task. LoRA leaves those weights frozen and learns a low-rank update: two small matrices whose product is added to a weight matrix during the forward pass. The stored task is the small update, not a second full checkpoint.

Why it matters

Hu et al. point at GPT-3 175B as the motivating case. Keeping an independent fine-tuned copy of a model that size, for every task, is too expensive to deploy. Low-rank adapters made specialization cheap enough to run many tasks from one frozen base model, which is why the method spread through open fine-tuning.

Key takeaways

The pretrained weights stay fixed. Only the low-rank factors are trained.
A rank much smaller than the layer dimension is enough for the adaptations they study.
At inference, the update can be folded into the original weight so latency matches a fully fine-tuned model.
Switching tasks means swapping the small adapter, not reloading a full parameter set.

LoRA reparametrization: a frozen weight matrix plus the product of two small trainable matrices — Figure 1 from Hu et al., 2021. Only the low-rank matrices A and B are trained.