What is a Model Distillation?

Rather than training the smaller model from scratch on raw labeled data alone, distillation trains it to match the teacher's outputs directly, often the full probability distribution over possible next tokens, known as soft labels, which carries richer signal than a single correct-answer label would. The student effectively learns not just what answer is right, but how confident the teacher was across all the plausible alternatives, which tends to transfer more of the teacher's learned judgment than training on hard labels alone. The motivation is almost entirely practical: large frontier models are expensive and slow to run at scale, while a distilled student model can run faster, cheaper, and on more constrained hardware, while retaining a meaningful share of the teacher's capability on the specific tasks it was distilled for. This makes distillation especially attractive for narrow, high-volume production tasks, such as classification, routing, or simple extraction, where the full generality of a frontier model is overkill for what's actually being asked of it. The tradeoff is that a distilled model typically underperforms its teacher on tasks or inputs far outside the distribution it was distilled on, since it's specialized to mimic the teacher's behavior on the training distribution rather than reproducing the teacher's full general reasoning ability.