Full fine-tuning of a large language model means updating every one of its weights. For a model in the range of 7 to 70 billion parameters, that requires storing not just the weights but the optimizer state associated with each one — momentum and variance terms for Adam, typically — plus activations, gradients, and the weights themselves. The result is a training footprint that can run 12 to 20 times the size of the model’s own weight file, before a single training example is loaded.

Most teams adapting a model to a task, domain, or dataset do not have that budget, and increasingly do not need it. Parameter-efficient fine-tuning (PEFT) methods — Low-Rank Adaptation (LoRA), its quantized variant QLoRA, and the broader family of adapter modules — update a small fraction of a model’s parameters while leaving the rest frozen. This article covers the mechanics of each approach, the trade-offs against full fine-tuning and against prompting, and how to decide which fits a given constraint.

We assume familiarity with the transformer architecture — attention, multi-head projections, and the general shape of a decoder stack — since every method here inserts or modifies weights inside that structure.

Why Full Fine-Tuning Is Costly at Scale

The cost of full fine-tuning comes from three places, and only one of them is the weights themselves.

Optimizer state. Adam and AdamW, the default optimizers for transformer training, keep two additional values per parameter — a running mean and a running variance of the gradient. A 7-billion-parameter model trained with mixed-precision Adam needs roughly 4 bytes (fp32 master weight) + 2 bytes (fp16 weight copy) + 2 bytes (fp16 gradient) + 8 bytes (fp32 Adam states) per parameter — on the order of 16 bytes per parameter, which puts full fine-tuning of a 7B model in the neighborhood of 112 GB for model and optimizer state alone, before a single batch of activations is computed.

Activation memory. Backpropagation requires holding intermediate activations from the forward pass so gradients can be computed layer by layer. Activation memory scales with batch size, sequence length, and layer count, and for long-context fine-tuning it can rival the optimizer footprint. Gradient checkpointing trades compute for memory here but doesn’t eliminate the cost — it shifts it into slower training steps.

Storage and deployment multiplicity. Fine-tune a 13B model for five customers or five tasks, and full fine-tuning produces five complete 13B-parameter checkpoints — tens of gigabytes each, needing separate storage and, in many serving setups, separate GPU memory to keep more than one loaded at a time.

None of this makes full fine-tuning obsolete — for building a foundation model, or tasks where maximal quality on a narrow, high-value target justifies the expense, it remains the ceiling other methods are measured against. But for the common case — adapting a pretrained model to a domain, style, or instruction-following behavior — parameter-efficient methods close most of the quality gap at a fraction of the cost.

LoRA: Low-Rank Adaptation Mechanics

LoRA (Hu et al., 2021) starts from an empirical observation: the weight update learned during fine-tuning tends to have a low “intrinsic rank” — most of the useful signal in the update matrix can be captured by a much smaller matrix decomposition than the full weight matrix itself.

Concretely, for a frozen pretrained weight matrix W₀ of dimension d × k, LoRA represents the fine-tuning update ΔW as the product of two much smaller matrices:

ΔW = B · A

where A is r × k and B is d × r, and r — the rank — is chosen far smaller than d or k, commonly in the range of 4 to 64. During training, W₀ stays frozen and only A and B receive gradient updates. At inference time, the two can be merged back into the original weight matrix (W₀ + BA), which means LoRA introduces zero additional latency compared to a fully fine-tuned model — the merged matrix is the same shape and does the same matrix multiply as the original.

The paper reports applying LoRA to GPT-3 175B by adapting only the query and value projection matrices in each attention layer, using rank 4, training roughly 10,000 times fewer parameters than full fine-tuning while cutting GPU memory requirements by a factor of three — and matching or exceeding full fine-tuning quality on the benchmarks tested (WikiSQL, MultiNLI, and the GPT-3 few-shot suite). Practically, a single base checkpoint can support many task-specific LoRA adapters, each a few megabytes to a few hundred megabytes depending on rank and target layers, swapped in and out without touching the base weights.

Two hyperparameters matter most: the rank r, which controls capacity (too low and the adapter can’t capture the needed update; too high and you lose the efficiency benefit and risk overfitting on small datasets), and which weight matrices receive adapters. Targeting only attention projections is common and cheap; some implementations also adapt the feedforward sublayers, increasing capacity and cost together. A scaling factor, typically denoted alpha, controls how strongly the low-rank update is weighted relative to the frozen weights and is usually tuned alongside the learning rate.

QLoRA: Quantized Base Models Plus LoRA

LoRA reduces trainable parameters, but the frozen base model still has to sit in GPU memory at reasonably high precision to run the forward pass. A 65-billion-parameter model at 16-bit precision needs around 130 GB before any adapter is added — out of reach for most single-GPU setups.

QLoRA (Dettmers et al., 2023) addresses this by quantizing the frozen base model down to 4-bit precision and training LoRA adapters on top of it, in higher precision, so the adaptation itself doesn’t lose the numerical fidelity needed for stable gradients. Three techniques do the work:

4-bit NormalFloat (NF4). A quantization data type designed specifically for the roughly-normal distribution of pretrained neural network weights, rather than a generic 4-bit integer format. NF4 preserves more information per bit for weights clustered near zero, where most pretrained weights sit.

Double quantization. The quantization process introduces constants (scale factors) that also need storing — double quantization quantizes those constants too, saving on average about 0.37 bits per parameter, which adds up across billions of parameters.

Paged optimizers. Borrowing NVIDIA’s unified memory paging, optimizer states move between GPU and CPU memory automatically during the rare batches that would otherwise cause an out-of-memory spike, rather than requiring the full optimizer state resident in GPU memory at all times.

The combined effect, per the QLoRA paper, is that a 65-billion-parameter model can be fine-tuned on a single 48 GB GPU in around 24 hours, with no measurable drop in quality compared to full 16-bit adaptation. The paper’s headline result — a model family called Guanaco, fine-tuned this way — reached 99.3% of ChatGPT’s evaluated performance on the Vicuna benchmark, at a fraction of the compute of a full-precision run. Over a thousand models were fine-tuned across the study to establish these results, itself evidence for how much cheaper the iteration loop becomes once quantization removes the memory floor.

The trade-off QLoRA makes explicit: quantizing the base model reduces throughput somewhat, since dequantizing weights on the fly for each forward pass costs compute a native 16-bit matrix multiply doesn’t. For training, where the goal is fitting on the hardware you have, that trade is usually worth it. For high-throughput serving, teams often merge the LoRA weights and redeploy in a higher-precision format rather than serving directly off the quantized training artifact.

Adapter Modules, More Generally

LoRA is one member of a broader family called adapter methods. The term “adapter” originally referred to something architecturally different: small bottleneck feedforward modules inserted directly into the transformer layer stack, rather than a low-rank update to existing weight matrices.

The original formulation, from Houlsby et al. (2019), inserts a bottleneck module — a down-projection, a nonlinearity, and an up-projection back to the model dimension, wrapped in a residual connection — after the attention sublayer and after the feedforward sublayer in each transformer block, with everything else frozen. Across 26 text classification tasks on BERT, this added only 3.6% additional parameters per task while landing within 0.4% of full fine-tuning’s performance on GLUE.

The architectural distinction from LoRA matters practically: bottleneck adapters add sequential compute, since every forward pass runs through the extra down-project/up-project layers, introducing a small but nonzero latency cost at inference (LoRA, by contrast, can be merged away entirely). The two approaches can be combined or swapped depending on the serving constraint — LoRA where zero added latency matters, bottleneck adapters where the module needs to stay separable from the base weights, as in multi-task setups that route different requests through different adapters.

Other members of this family worth knowing by name: prefix-tuning and prompt-tuning, which prepend trainable “virtual tokens” to the input rather than modifying weights at all; and IA³, which learns per-channel rescaling vectors instead of low-rank matrices, trading some capacity for an even smaller footprint than LoRA. LoRA and QLoRA have become the default for most open-weight LLM fine-tuning workflows, largely because tooling (Hugging Face’s PEFT library, bitsandbytes for quantization) matured around them first — but the idea across the family is the same: freeze most of the network, train a small, structured subset of parameters, and recover most of full fine-tuning’s quality.

Practical Trade-Offs: Memory, Latency, and Quality

Memory. This is where parameter-efficient methods win decisively. LoRA cuts trainable-parameter count and associated optimizer state by one to three orders of magnitude depending on rank and target modules. QLoRA compounds that by quantizing the frozen base model itself, making 65B-scale fine-tuning feasible on a single workstation-class GPU rather than a multi-GPU cluster — often the deciding factor for teams without dedicated training infrastructure.

Latency. LoRA, once merged, adds nothing at inference — the served model is architecturally identical to a fully fine-tuned one. Unmerged LoRA (kept separate so many task adapters can share one base model in memory) adds a small matrix multiply per adapted layer, typically low single-digit percentage overhead. Bottleneck adapters add a comparable or slightly larger overhead since the extra layers can’t be merged away. QLoRA’s quantization overhead is a training-time (and, if served directly off quantized weights, inference-time) cost, not a LoRA-specific one — most production deployments dequantize or re-merge before serving at scale.

Quality. LoRA and QLoRA come within a small margin of full fine-tuning quality on the tasks they were tested against — the QLoRA paper’s central claim is “no regression” versus full 16-bit fine-tuning. Published parity results are strongest for instruction-following and moderate-complexity tasks; for substantially new capability rather than adaptation of existing capability, the smaller effective capacity of a low-rank update can be a genuine ceiling.

Versus prompting. Prompting and in-context few-shot examples require no training, making them the right first move for most tasks — cheaper to iterate, no training data, no overfitting risk. But prompting consumes context window on every request, a cost that compounds at scale, and caps out on tasks needing reliable behavioral change rather than per-request steering. Retrieval extends what prompting can do for knowledge-grounding but doesn’t substitute for fine-tuning when the target is the model’s behavior or task competence, not the facts available to it.

When Each Approach Is the Right Choice

Use prompting or retrieval first when the task can be specified through instructions and examples, when underlying knowledge changes frequently, or when you need to validate whether fine-tuning is worth the investment before committing engineering time.

Use LoRA when you have training data and need consistent behavioral or stylistic adaptation, when you’re serving many task variants off one base model and want to swap lightweight adapters rather than hosting multiple full checkpoints, or when the target hardware can hold the base model at high precision but not a full fine-tuning run’s optimizer state.

Use QLoRA when GPU memory is the binding constraint — a single GPU, a model larger than it can hold at 16-bit precision, or a prototyping setting where iterating on many runs cheaply matters more than the last percentage point of quality. It is, in effect, the enabling technology for fine-tuning large open-weight models outside well-resourced labs.

Use bottleneck adapters in multi-task serving architectures where the adapter needs to remain a distinct, swappable module rather than being merged into the base weights — systems that route different requests through different task-specific adapters at inference time.

Use full fine-tuning when the task requires capability the pretrained model lacks in any latent form, when building a foundation model rather than adapting one, or when maximal quality on a single high-value target justifies the cost.

Most projects starting from “should we fine-tune at all” are well served by LoRA on modest hardware or QLoRA where memory is tight — validate the approach moves the needed metric, and escalate to full fine-tuning only if a measured gap turns up that the low-rank update can’t close.

Frequently Asked Questions

What’s the difference between LoRA and QLoRA?

LoRA freezes the pretrained model and trains small low-rank update matrices on top of it, typically in 16-bit precision. QLoRA does the same thing but first quantizes the frozen base model to 4-bit precision, drastically cutting the memory needed to hold it and enabling fine-tuning of much larger models on smaller hardware with minimal quality loss.

Does LoRA add latency at inference time?

No, when merged. Because LoRA’s update is a matrix product of the same shape as the original weight matrix, it can be added directly into the frozen weights after training, producing a model architecturally identical to one fully fine-tuned. Latency only appears if the adapter is kept unmerged to allow swapping between tasks at runtime.

How much GPU memory does QLoRA actually save?

The QLoRA paper demonstrates fine-tuning a 65-billion-parameter model on a single 48 GB GPU — a scale that would otherwise require on the order of 1 terabyte of memory for full 16-bit fine-tuning once mixed-precision weights, gradients, and Adam’s two optimizer states per parameter are all accounted for. The saving comes from 4-bit quantization of the frozen base weights combined with double quantization of the quantization constants themselves.

Can parameter-efficient fine-tuning fully replace full fine-tuning?

Not universally. LoRA and QLoRA match full fine-tuning on instruction-following and moderate-complexity tasks, but effective capacity is bounded by the chosen rank. Tasks requiring genuinely new capability, rather than adaptation of existing capability, can hit a ceiling only full fine-tuning or continued pretraining resolves.

What rank should I choose for a LoRA adapter?

There’s no universal number — it depends on task complexity and dataset size. Common practice starts in the 8 to 16 range for moderate adaptation tasks, with some setups going as low as 4 for narrow stylistic changes or as high as 64 for tasks closer to full-capacity adaptation. Higher rank increases capacity and parameter count together, so it’s worth validating on a held-out set rather than assuming higher is strictly better.