Ensemble Methods

Ensemble methods are techniques that combine the predictions of multiple AI models to produce a result that is more accurate, more robust, and more reliable than any individual model. The principle is similar to asking multiple experts for their opinion and going with the consensus.

Why ensembles work

Individual models make different types of errors. A decision tree might struggle with smooth boundaries while a neural network handles them well. Conversely, the neural network might overfit to noise that the decision tree ignores. By combining their predictions, errors tend to cancel out while correct predictions reinforce each other.

This works because of a statistical principle: the average of many imperfect predictions is often better than any single prediction, as long as the errors are not all in the same direction.

Main ensemble approaches

• Bagging (Bootstrap Aggregating): Train multiple instances of the same model on different random subsets of the data, then average their predictions. Random forests are the most famous example of bagging.
• Boosting: Train models sequentially, where each new model focuses on correcting the errors of the previous ones. XGBoost, LightGBM, and AdaBoost are popular boosting algorithms.
• Stacking: Train multiple different model types, then train a "meta-model" that learns the best way to combine their predictions.
• Voting: Simply let multiple models vote on the prediction and take the majority (for classification) or average (for regression).

Ensembles in practice

Ensemble methods dominate machine learning competitions. Nearly every winning solution in Kaggle competitions uses some form of ensembling. In production, the trade-off is between accuracy and complexity — ensembles are harder to deploy, maintain, and explain than single models.

Common production uses include:

• Credit scoring: Multiple models evaluate creditworthiness, and the ensemble provides a more reliable score.
• Medical diagnosis: Combining multiple diagnostic models reduces the risk of any single model's blind spots affecting the final recommendation.
• Weather forecasting: Ensembles of weather models are more reliable than any single model.

Ensembles with large language models

Even in the era of LLMs, ensemble principles apply. Techniques like "self-consistency" (generating multiple answers and taking the majority) and "mixture of experts" (routing different inputs to specialised sub-models) are ensemble approaches adapted for the generative AI era.

The accuracy-cost trade-off

Ensembles are more expensive than single models — they require training and running multiple models. The key question is whether the improvement in accuracy justifies the additional cost. For high-stakes decisions (medical, financial, safety), the answer is usually yes. For low-stakes tasks, a single well-tuned model often suffices.