Knowledge Distillation: Compressing Large Models Into Small Ones
How knowledge distillation trains small student models from large teachers, from soft targets and temperature to sequence-level and reasoning distillation, and where it falls short.
By Institute for Joint Cognition & AI · 6 min · 17 November 2026

Large models are expensive to serve, and most deployments do not need all of their capacity. Knowledge distillation is the family of techniques for training a smaller “student” model to reproduce the behaviour of a larger “teacher,” so that much of the teacher’s quality survives at a fraction of the inference cost. It has moved from a compression trick for image classifiers to a standard step in how small open-weight language models are built.
The original idea: learning from soft targets
The modern formulation comes from Hinton, Vinyals and Dean’s 2015 paper, Distilling the Knowledge in a Neural Network. Their observation was that a trained classifier’s output distribution carries more information than its top prediction. A teacher that assigns a picture of a BMW 0.9 probability of “car,” a small probability of “truck,” and a vanishingly small probability of “carrot” is telling the student how classes relate to each other. A one-hot label throws that away.
To expose this “dark knowledge,” the teacher’s logits are passed through a softmax with a temperature T greater than 1, which flattens the distribution so that small probabilities become visible. The student is trained with the same temperature to match those soft targets, usually by minimising KL divergence, and that loss is combined with the ordinary cross-entropy loss on the true labels. The soft-target term is scaled by T² so its gradient magnitude stays comparable as the temperature changes.
The result is a student that generalises better than the same architecture trained on hard labels alone, because each training example now carries a full distribution’s worth of signal.
Variants: what exactly gets matched
Matching output probabilities is only one choice. The literature since 2015 varies along two axes: what the student imitates, and what data it imitates on.
Intermediate representations
FitNets (Romero et al., 2014) trained thin, deep students to match the teacher’s intermediate feature maps through a learned projection, not just its outputs. Later work matched attention maps, hidden states, or relations between examples. These “feature-based” methods help when the student is much smaller than the teacher and output matching alone gives too weak a signal, but they require the architectures to be aligned closely enough that a layer mapping makes sense.
Task-agnostic compression of a pretrained model
DistilBERT (Sanh et al., 2019) applied distillation during pretraining rather than fine-tuning. The authors report a model 40% smaller than BERT-base that is 60% faster and retains 97% of its language-understanding performance on the GLUE benchmark. That result established distillation as a practical route to deployable encoders.
Sequence-level distillation
For generative models, matching per-token distributions is not the only option. Kim and Rush (2016) proposed sequence-level distillation for machine translation: run the teacher with beam search, then train the student on the teacher’s outputs as if they were ground truth. This is the conceptual ancestor of a very common modern practice, fine-tuning a small model on text generated by a large one.
Distillation in current language models
Several recent model families document distillation as a core part of training rather than an afterthought.
- Token-level distillation in pretraining. The Gemma 2 technical report describes training the 2B and 9B models with knowledge distillation from a larger teacher, replacing next-token prediction on one-hot targets with the teacher’s distribution, while the 27B model was trained from scratch. Meta similarly described using logits from larger Llama 3.1 models as token-level targets when building the 1B and 3B Llama 3.2 models.
- Pruning plus distillation. NVIDIA’s Minitron work prunes a trained model’s width or depth and then recovers quality by distilling from the original, reporting that this uses far fewer training tokens than training the smaller model from scratch.
- Reasoning distillation. The DeepSeek-R1 paper released smaller Qwen- and Llama-based models fine-tuned on reasoning traces generated by the large R1 model. This is sequence-level distillation applied to chain-of-thought: the student learns from the teacher’s worked solutions, not its logits. The authors report that distilling from the large model worked better for the small models than applying reinforcement learning to them directly.
These cases show the spectrum. Logit-level distillation needs access to the teacher’s full output distribution, which in practice means owning the teacher or having its weights. Sequence-level distillation needs only generated text, which is why it is also the form most often discussed in the context of training on another provider’s model outputs, a practice that many commercial terms of service restrict.
Where distillation falls short
Distillation is compression, and compression loses something. The open question is usually what.
Capacity gaps. A student far smaller than its teacher may not be able to represent what the teacher knows, and research on the “capacity gap” has found that a very strong teacher is not always the best teacher for a very small student. Intermediate “teacher assistant” models or progressive distillation are common responses.
Uneven retention. Aggregate benchmark scores can hide where the student degrades. Rare classes, long-tail facts, multilingual performance and robustness to distribution shift are typical casualties. A distilled model that matches its teacher on a headline benchmark should still be evaluated on the slices that matter for its deployment, as discussed in our piece on evaluating LLMs.
Inherited errors. A student trained on a teacher’s outputs inherits its mistakes and biases, with no ground truth to correct them in the pure sequence-level setting. When the teacher’s outputs are filtered or verified, as in reasoning distillation with checked answers, this risk falls, but it does not vanish. This connects to wider concerns about training on model-generated data, covered in our article on synthetic data and model collapse.
Calibration. Students can be more or less confident than their teachers. If downstream systems use output probabilities, for example to decide when to defer to a human, calibration needs to be measured on the student directly.
How it compares with other compression methods
Distillation is one of three common ways to shrink inference cost, and they combine well:
- Quantization lowers numerical precision of an existing model’s weights and activations. It is fast to apply and needs little or no training, but it does not reduce parameter count.
- Pruning removes weights, neurons, heads or layers. Structured pruning reduces real compute; it typically needs retraining to recover accuracy, often done by distillation.
- Distillation trains a new, smaller architecture. It costs a training run but yields a model whose size and shape you choose.
A common production pipeline applies all three: distil to a smaller architecture, prune what remains redundant, then quantize for serving. Our overview of inference cost optimization covers the serving side of that stack.
For practitioners deciding whether to distil, the useful question is narrow: is there a specific task and latency budget for which a smaller model, trained on a larger model’s behaviour, would be good enough? When the answer is yes and evaluation covers the slices that matter, distillation is one of the most reliable ways to cut serving cost without starting from scratch.