Explainability

Explainability (also called interpretability or XAI — explainable AI) is the degree to which humans can understand why an AI system made a particular decision. It is the difference between a model that says "loan denied" and one that says "loan denied because debt-to-income ratio exceeds the threshold, with late payment history as a contributing factor."

Why explainability matters

• Trust — users and stakeholders are more likely to trust AI decisions they can understand
• Debugging — when a model makes errors, explainability helps identify the root cause
• Regulation — laws like the EU AI Act and sector regulations in finance and healthcare require explanations for automated decisions
• Fairness — you cannot detect bias if you cannot understand what drives decisions
• Adoption — employees resist AI tools they perceive as opaque and unaccountable

Levels of explainability

• Inherently interpretable models — decision trees, linear regression, and rule-based systems are transparent by design. You can inspect the rules directly.
• Post-hoc explanations — techniques applied after training to explain black-box models. They approximate what the model is doing.
• Model-agnostic methods — explanation techniques that work with any model type.

Common explanation techniques

• SHAP (SHapley Additive exPlanations) — assigns each feature an importance score for each prediction, based on game theory
• LIME (Local Interpretable Model-agnostic Explanations) — builds a simple, interpretable model around a specific prediction to explain it locally
• Attention visualisation — for transformer models, showing which parts of the input the model focused on
• Feature importance — ranking which input features most influence the model's decisions overall
• Counterfactual explanations — "the loan would have been approved if income were ten per cent higher"

The accuracy-explainability trade-off

Historically, the most accurate models (deep neural networks) were the least explainable, while the most explainable (linear models, decision trees) were less accurate. This trade-off is shrinking as explanation techniques improve, but it remains a consideration in model selection.