Pre-training

Pre-training is the first and most resource-intensive phase of building a modern AI model. During pre-training, the model learns general knowledge and capabilities from an enormous, diverse dataset — typically trillions of tokens of text, code, and other data.

How pre-training works for language models

The pre-training objective for most large language models is deceptively simple: predict the next word. Given a sequence of words, predict what comes next. The model is trained on this task across trillions of text examples from books, websites, code repositories, scientific papers, and more.

Through this simple task, the model learns:

• Grammar and syntax
• Facts and knowledge
• Reasoning patterns
• Multiple languages
• Code structure
• Writing styles and formats

The scale of pre-training

Pre-training a frontier language model requires:

• Data — trillions of tokens from diverse, curated sources
• Compute — thousands of GPUs running for weeks or months
• Cost — tens to hundreds of millions of dollars
• Energy — significant electricity consumption

This is why only a handful of organisations can afford to pre-train frontier models.

Pre-training vs. fine-tuning

• Pre-training builds the foundation — broad knowledge and general capabilities
• Fine-tuning specialises the foundation — adapting the model for specific tasks, domains, or behaviours

Think of pre-training as a university education (broad knowledge) and fine-tuning as professional training (specific skills).

The pre-training pipeline

1. Data collection — gathering and curating enormous datasets
2. Data cleaning — removing duplicates, filtering low-quality content, handling sensitive data
3. Tokenisation — converting text into the numerical tokens the model processes
4. Training — the actual compute-intensive process of learning from the data
5. Evaluation — testing the pre-trained model on benchmarks to assess capabilities

Pre-training data matters

The quality and composition of pre-training data profoundly shapes the model's capabilities and biases. Models trained on more code are better at programming. Models trained on more scientific literature are better at technical reasoning. Data curation is one of the most impactful — and least visible — decisions in model development.