Join the DZone community and get the full member experience.
Join For Free
Series reminder: This series explores how explainability in AI helps build trust, ensure accountability, and align with real-world needs, from foundational principles to practical use cases.
In this Part: We lay the groundwork for explainable AI: what it means, why it matters, and what's at stake when AI systems remain opaque.
Preamble
This document is the result of a collaboration between two authors. We aimed to combine theoretical reflection and practical experiments on a topic that matters deeply to us in today's field of artificial intelligence: explainability.
In the first section (PART I to PART IV), Frédéric Jacquet lays the foundations of explainable AI. He explores the ethical, regulatory, and operational issues tied to algorithmic transparency, while presenting the main theoretical approaches and existing tools. This section is intended for decision-makers, AI practitioners, and citizens alike, anyone concerned with understanding the underlying logic behind intelligent systems.
In the second section (PART V to PART IX), Marc-Antoine Iehl presents a concrete implementation of two explainability methods (LIME and SHAP) applied to real-world cases in the medical and financial sectors. Through rigorous experimentation and clear visualizations, he demonstrates how these tools help uncover the reasoning behind complex model decisions, while also questioning their robustness and explanatory power.
This study has two main objectives:
* To provide a clear and accessible framework for understanding the challenges of explainable AI,
* To demonstrate concretely how these challenges play out in the field -- close to the data and the algorithms.
Section I -- Building Trust Through Explainable AI
Before we begin, it's important to highlight a point that is often overlooked: while we expect AI to be performant, it also needs to be understandable.
Because only by making its decisions transparent can we truly trust them, correct them, or challenge them. That's the core of explainable AI (XAI), which this first part aims to shed light on.
1. Introduction
How would you react if a doctor gave you a serious diagnosis based on an algorithm, without knowing why, or what criteria led to that conclusion? What if your employer fired you based on a decision made by AI software, with no explanation and no chance to appeal? Or if the autopilot of a car or a plane made a decision that resulted in a fatal accident, yet no expert could understand what happened?
Through these three situations, inspired by real cases, it becomes immediately clear that to fully benefit from AI technologies with confidence and safety, blindly trusting the technology is simply not enough.
In practice, many AI models operate as opaque systems, making it difficult (if not impossible) to understand how their decisions are made.
Explainable AI (XAI - Explainable Artificial Intelligence) aims to address this issue. The concept is about clearly explaining how algorithms and decision-making processes work. This helps build user trust. It is also through this approach that organizations can align with emerging regulations such as the European AI Act.
Explaining an AI model should not be optional, it should be a fundamental requirement to ensure its ethical use.
2. Defining Explainable AI
What Is Explainable AI?
Explainable AI consists of methods and techniques that make algorithmic decisions easier for humans to understand. In contrast to using AI systems that simply deliver results with no justification, XAI seeks to answer questions such as:
* Why was a specific decision made?
* What factors influenced that decision?
* What could be done to achieve a different outcome?
Explainable AI relies on two main approaches. The first focuses on intrinsically interpretable models, that is, models whose inner workings follow a logic that humans can understand. For instance, expert systems, which have long been used in computer science to structure decision-making based on predefined rules, offer natural transparency in how results are explained.
The second approach relies on post-hoc analysis methods, which aim to explain a model that has already been trained without altering how it functions.
Techniques such as SHAP and LIME are used to identify the factors influencing the predictions of a complex algorithm, making them easier to interpret. Other approaches, such as contrastive and counterfactual explanations, functional methods focused on transparency, intrinsically interpretable models, and techniques based on ontologies and knowledge graphs, also offer complementary solutions for better understanding and explaining AI system decisions.
Why Do We Need Explainable AI?
The Risks of Opaque AI
One of the main reasons explainable AI is needed is the persistence of algorithmic bias, often introduced by the training data on which models are built.
Without explainability mechanisms that allow these biases to be understood and corrected, certain populations may be unintentionally discriminated against, as has been observed in systems for hiring or loan approval, for example.
This opacity also raises a serious issue when it comes to accountability. When an AI system makes a wrong decision (especially in sensitive domains like justice or healthcare) it becomes difficult to determine who should be held responsible: the model's designer, the company deploying it, or the end user relying on it?
The adoption of AI, both in businesses and by the general public, partly depends on its ability to provide clear explanations. An opaque model can generate distrust and limit its use, especially in regulated or sensitive sectors.
The Benefits of Explainable AI
The adoption of AI, both in businesses and by the general public, partly depends on its ability to provide clear explanations. In fact, this tends to strengthen user trust. As a result, a model that can justify its choices is more easily accepted and integrated into decision-making, whereas an opaque model may trigger distrust and limit its use. This phenomenon is especially pronounced in regulated or sensitive sectors.
Explainable AI also helps reduce errors and biases by providing tools to analyze and understand why a model produced one outcome rather than another. It becomes easier to correct these issues, and thus improve the reliability of automated decisions, when the sources of error or unintentional discrimination within a model can be identified.
For example, in medical diagnostic systems, strong interpretability helps prevent biased or inappropriate decisions that could have serious consequences for individuals.
From a legal standpoint, explainable AI plays a key role in meeting regulatory requirements, especially in Europe. Legislative frameworks like the AI Act impose concrete obligations around transparency and auditability of AI systems. They aim to ensure that algorithms do not make arbitrary or discriminatory decisions, and that human oversight remains in place. Organizations that rely on XAI mechanisms are better equipped to comply with these standards, thereby reducing the legal and ethical risks associated with AI use.
Understanding why an AI makes a decision is what gives it legitimacy.
In light of these challenges, to better understand how AI can be made more transparent and comprehensible, let's explore the approaches that help open up the opaque inner workings of algorithms.
Wrap-up and what's next: AI must not only perform but also be understood.
Next, in part II, we'll explore the two main families of explainability techniques and how they help make AI decisions more transparent.
Glossary
Algorithmic Bias: Systematic and unfair discrimination in AI outcomes caused by prejudices embedded in training data, model design, or deployment processes, which can lead to disparate impacts on certain population groups. Detecting and mitigating algorithmic bias is a key objective of explainable AI.
Bias Detection (via XAI): Use of explainability methods to identify biases or disproportionate effects in algorithmic decisions.
Contrastive and Counterfactual Explanations: Explanations that compare the decision made to what could have happened by changing certain variables (e.g., "Why this outcome instead of another?").
Decision Plot: A graphical representation tracing the successive impact of variables on an individual prediction.
Evaluation Metrics for Explainability: Criteria used to assess the quality of an explanation (fidelity, robustness, consistency, etc.).
Feature Importance (Variable Contribution): Measurement or attribution of the relative impact of each variable on the model's final decision.
Force Chart (Force Plot): An interactive visualization illustrating positive or negative forces exerted by each variable on a prediction.
Fidelity: A measure of how faithfully the explanation reflects the true logic of the model.
Global Explanation: An overview of the model's behavior across the entire dataset.
Human Interpretability: The quality of an explanation to be understood and useful to a human, non-expert user.
Intrinsically Interpretable Models: Models whose very structure allows direct understanding (e.g., decision trees, linear regressions).
LIME (Local Interpretable Model-agnostic Explanations): A local explanation method that generates simple approximations around a given prediction to reveal the influential factors.
Local Explanation: A detailed explanation regarding a single prediction or individual case.
Model Transparency: The quality of a model in making its decision-making processes accessible and understandable.
Post-hoc Explainability: Explainability techniques applied after model training, without altering its internal functioning.
Robustness (Stability): The ability of an explainability method to provide consistent explanations despite small variations in input data.
SHAP (SHapley Additive exPlanations): An approach based on game theory that assigns each variable a quantitative contribution to the prediction, providing both global and local explanations.
Summary Chart (Summary Plot): A visualization ranking variables according to their average influence on predictions.
Waterfall Chart: A static visualization showing step by step how each variable contributes to the final prediction.
References
1. LIME: [link]
2. Chest X-Ray Images (Pneumonia): [link]
3. German Credit Dataset: [link]
4. Complete notebook for the LIME case study: [link]
5. Complete notebook for the SHAP case study: [link]
6. Leslie, D., Rincón, C., Briggs, M., Perini, A., Jayadeva, S., Borda, A., et al. (2024). AI Explainability in Practice, The Alan Turing Institute, AI Ethics and Governance in Practice Programme. Participant & Facilitator Workbook. https://www.turing.ac.uk/sites/default/files/2024-06/aieg-ati-7-explainabilityv1.2.pdf
7. AI explainability in practice. (2024). The Alan Turing Institute. https://www.turing.ac.uk/sites/default/files/2024-06/aieg-ati-7-explainabilityv1.2.pdf
8. What is AI (artificial intelligence)? (2024, April 3). McKinsey & Company. https://www.mckinsey.com/featured-insights/mckinsey-explainers/what-is-ai
9. Kite-Powell, J. (2022, July 28). Explainable AI is trending and here's why. Forbes. https://www.forbes.com/sites/jenniferhicks/2022/07/28/explainable-ai-is--trending-and-heres-why
10. Excellence and Trust in AI -- Brochure. (n.d.). Shaping Europe's Digital Future. https://digital-strategy.ec.europa.eu/en/library/excellence-and-trust-ai-brochure
11. Heikkilä, M. (2024, March 6). Nobody knows how AI works. MIT Technology Review. https://www.technologyreview.com/2024/03/05/1089449/nobody-knows-how-ai-works/
12. Heaven, W. D. (2025, July 30). Large language models can do jaw-dropping things. But nobody knows exactly why. MIT Technology Review. https://www.technologyreview.com/2024/03/04/1089403/large-language-models-amazing-but-nobody-knows-why/
13. Francis, J. (2024, November 15). Council Post: Why 85% of your AI models may fail. Forbes. https://www.forbes.com/councils/forbestechcouncil/2024/11/15/why-85-of-your-ai-models-may-fail
14. Bao, A., & Zeng, Y. (2024). Understanding the dilemma of explainable artificial intelligence: a proposal for a ritual dialog framework. Humanities and Social Sciences Communications, 11(1). https://doi.org/10.1057/s41599-024-02759-2
15. Gallese, C. (2023). The AI Act proposal: a new right to technical interpretability? arXiv (Cornell University). https://doi.org/10.48550/arxiv.2303.17558
16. GALDON CLAVELL, G. (n.d.). AI Auditing - Checklist for AI auditing. In SUPPORT POOL OF EXPERTS PROGRAMME. https://www.edpb.europa.eu/system/files/2024-06/ai-auditing_checklist-for-ai-auditing-scores_edpb-spe-programme_en.pdf
17. White, B., & Guha, K. (2025, June 14). Component based Quantum Machine Learning explainability. arXiv.org. https://arxiv.org/abs/2506.12378