Denoising

Denoising is a machine learning technique where a model is trained to recover original, clean data from a corrupted version. By learning to remove noise, the model develops a deep understanding of the underlying structure and patterns in the data.

The core concept

The idea is simple but powerful. Take clean data, add some form of corruption (noise), then train the model to reconstruct the original. Through this process, the model must learn what the data "should" look like — its underlying structure, patterns, and regularities — in order to distinguish the real signal from the noise.

Denoising in language models

BERT and similar encoder models were trained using a denoising objective called "masked language modelling." The training process randomly masks (hides) words in a sentence, and the model learns to predict the missing words from context. By learning to fill in blanks, the model develops rich understanding of language structure, grammar, and meaning.

T5 extended this idea by corrupting text spans of varying lengths and training the model to reconstruct them, creating a versatile text-to-text model.

Denoising in image generation

Diffusion models — the technology behind DALL-E, Midjourney, and Stable Diffusion — are fundamentally denoising models. During training, the model learns to progressively remove noise from images. During generation, it starts with pure random noise and iteratively denoises it into a coherent image, guided by a text description.

This is why these models are sometimes called "denoising diffusion models." Each step of the generation process removes a bit more noise, gradually revealing the image.

Why denoising is such an effective training strategy

Denoising is a form of self-supervised learning — you do not need human-labelled data because the model creates its own training signal from the corruption process. This means you can train on vast amounts of unlabelled data. The task also forces the model to understand the data deeply — superficial pattern matching is not sufficient to reconstruct corrupted inputs.

Denoising autoencoders

A denoising autoencoder is a neural network trained to compress noisy data into a compact representation and then reconstruct the clean version. These models learn robust features that are useful for downstream tasks like classification, anomaly detection, and data generation.