Self-Supervised Learning

Self-supervised learning is a training technique where the model creates its own training labels from unlabelled data. Instead of requiring humans to label thousands of examples, the model learns by predicting parts of its own input — a missing word in a sentence, the next frame in a video, or the relationship between two image patches.

Why self-supervised learning matters

Labelled data is expensive and slow to produce. Hiring people to label millions of images, transcribe audio, or classify documents costs significant time and money. But unlabelled data is abundant — the internet contains trillions of words, billions of images, and millions of hours of video.

Self-supervised learning unlocks this vast pool of unlabelled data by finding ways to learn from it without human labels.

How it works in language models

Large language models like GPT and Claude are trained using self-supervised learning. The training task is simple: given a sequence of tokens, predict the next one. The model reads "The capital of France is" and learns to predict "Paris." No human needed to label this example — the label (the next word) comes from the data itself.

By performing this prediction task across billions of text examples, the model learns grammar, facts, reasoning patterns, and much more — all without a single human-provided label.

How it works in computer vision

In images, self-supervised methods might:

• Mask random patches of an image and train the model to reconstruct them
• Rotate an image and train the model to predict the rotation angle
• Show two augmented versions of the same image and train the model to recognise they are related

These tasks force the model to learn meaningful visual representations.

The relationship to other learning types

• Supervised learning: Requires human-labelled data (expensive, limited)
• Unsupervised learning: Finds patterns without labels (clustering, dimensionality reduction)
• Self-supervised learning: Creates its own labels from the data structure (scalable, powerful)

Self-supervised learning is sometimes classified as a subset of unsupervised learning, but it has become important enough to be discussed as its own category.

Impact on modern AI

Self-supervised learning is the reason modern AI is as capable as it is. It enabled training on internet-scale datasets that would have been impossible to label manually. Every major language model, and increasingly every major vision model, is built on self-supervised pre-training.