What is an Explainable AI (XAI)?

Many high-performing models, especially deep neural networks and large language models, are difficult to interpret directly — the mathematics behind a prediction doesn't map cleanly to human-readable reasons. Explainability techniques try to close that gap: feature-attribution methods (like SHAP or LIME) estimate which inputs drove a given output, attention visualization shows what a model "focused on," and simply asking a language model to produce a chain of reasoning alongside its answer gives a human-readable trace, even though it isn't a perfect window into the underlying computation. Explainability matters for more than curiosity — it's often a regulatory or trust requirement. If an AI system denies a loan, flags a transaction as fraud, or recommends a medical action, the affected person or an auditor frequently has a right to know why, and a business needs to catch and correct systematic errors it can't see in a black box. The trade-off is that the most explainable models (simple decision trees, linear models) are sometimes less accurate than the least explainable ones, so real systems often layer a post-hoc explanation on top of a high-performing but opaque model rather than sacrificing accuracy outright.