Distillation

Distillation (or knowledge distillation) is a technique for compressing the knowledge of a large, expensive AI model into a smaller, cheaper one. The large model (called the teacher) trains the small model (called the student) to produce similar outputs.

How distillation works

Instead of training the student model on raw data, you train it on the teacher model's outputs. This is surprisingly effective because the teacher's outputs contain richer information than simple labels. When a teacher model says an image is "ninety per cent likely a cat and eight per cent likely a dog," it is teaching the student about the similarity between cats and dogs — information that a hard label ("cat") does not convey.

Why distillation matters

• Cost reduction — a distilled model might be ten times smaller and faster while retaining ninety per cent of the teacher's capability
• Edge deployment — smaller models can run on phones, embedded devices, and environments with limited compute
• Latency — smaller models respond faster, critical for real-time applications
• Budget — API costs are proportional to model size; smaller models cost less per query

Types of distillation

• Output distillation — the student learns to match the teacher's output probabilities
• Feature distillation — the student learns to match the teacher's internal representations at intermediate layers
• Self-distillation — a model distils knowledge from a larger version of itself

Distillation in practice

Many of the efficient AI models you use daily are distilled versions of larger models. Companies often fine-tune and distil large foundation models into specialised models optimised for specific tasks. This is how you get capable AI running on a smartphone.

Limitations

• Distilled models inevitably lose some capability, particularly on edge cases and complex reasoning
• The quality ceiling is the teacher model — a student cannot exceed its teacher
• For highly specialised tasks, training directly on task-specific data may outperform distillation