Quantization-Aware Training

Quantization-aware training (QAT) is a technique where the effects of quantization — reducing numerical precision — are simulated during the training process itself. This produces models that maintain higher quality when subsequently quantized for deployment, compared to models that are quantized after training.

The quantization challenge

Quantization converts model weights from high-precision numbers (like 32-bit floating point) to lower-precision numbers (like 8-bit or 4-bit integers). This dramatically reduces model size and speeds up inference. However, the reduced precision means some information is lost, which can degrade model quality.

Post-training quantization (PTQ) applies quantization to a model that was trained in full precision. This is simple but can cause significant quality degradation, especially at aggressive compression levels like 4-bit or lower.

How QAT works

During QAT, the training process simulates the effects of quantization at each step. Forward passes use quantized weights (or a simulation of them), so the model experiences the precision loss during training. Backward passes still use full precision for gradient calculations, maintaining training stability.

By encountering quantization effects during training, the model learns to be robust to reduced precision. Weights adjust to values that are "quantization-friendly" — they remain effective even after being rounded to lower precision.

The practical benefit

QAT consistently produces higher-quality quantized models compared to PTQ, especially at aggressive compression levels. A 4-bit QAT model might match the quality of an 8-bit PTQ model, effectively getting 2x more compression at the same quality level.

When to use QAT vs PTQ

• PTQ: Use when you need quick compression of an existing model and can tolerate some quality loss. No additional training required.
• QAT: Use when quality preservation is critical and you can afford the computational cost of training or fine-tuning with quantization simulation.

Industry adoption

Major model providers increasingly use QAT to produce efficient deployment versions of their models. The open-source community also uses QAT techniques when creating compressed versions of popular models for consumer hardware.