Weight Initialization

Weight initialization is the process of assigning starting values to the parameters (weights) of a neural network before training begins. While it may seem like a minor technical detail, the choice of initialization strategy can determine whether a model trains successfully or fails entirely.

Why initialization matters

Neural networks learn by adjusting weights through backpropagation — computing how much each weight should change to reduce the error. If weights start too large, the signals flowing through the network can explode (grow uncontrollably). If weights start too small, signals vanish (shrink to zero). Both situations prevent the network from learning.

Good initialization puts weights in a "sweet spot" where signals propagate through the network at a stable magnitude, enabling effective learning from the first training step.

Common initialization strategies

• Xavier/Glorot initialization: Sets weights based on the number of inputs and outputs for each layer. Designed for networks using sigmoid or tanh activation functions. Keeps signal variance approximately constant across layers.
• He initialization: A variant of Xavier designed for ReLU activation functions, which are used in most modern networks. Accounts for the fact that ReLU zeros out negative values, which would otherwise cause signal shrinkage.
• Orthogonal initialization: Sets weight matrices to be orthogonal, preserving the norm of the signal as it passes through layers.
• Random normal/uniform: Simple random initialization from a normal or uniform distribution. Works for small networks but can cause problems in deep ones.
• Pre-trained initialization: Starting from the weights of a previously trained model (transfer learning). This is how fine-tuning works — starting from a strong initialization dramatically accelerates training.

Initialization in modern LLMs

Large language models use carefully designed initialization schemes that account for their enormous depth and width. The specific initialization strategy interacts with architecture choices like layer normalization, residual connections, and attention mechanisms. Small changes in initialization can affect whether a multi-billion-parameter model trains stably or diverges.

The practical takeaway

Most modern deep learning frameworks handle initialization automatically with sensible defaults. However, when training fails to converge or produces unstable results, incorrect initialization is one of the first things to investigate.