Chapter 8

Explainable and Interpretable AI: Human-Centered Aspects

Katie Ulinski, Blade Frisch, Christopher Cischke, Lauren Sprague, Samantha Walker, Jeff Wang, Joanne Wang, Brandon Woolman, and Shane T. Mueller
Michigan Technological University

8.1 Introduction

Explainable AI (XAI) describes algorithms that are designed to help users understand the technical properties of a model. This is sometimes an algorithm that is naturally interpretable (i.e., a set of verbalizable rules), and sometimes new logic built into an existing model (i.e., a narrative description of an image classifier), and it is often a “sidecar” algorithm used to augment an existing model (i.e., a heat-map showing which features were most important). Typically, discussions of XAI start and end with a list of common explainability algorithms and what they provide. Furthermore, research in this domain often takes an existing machine learning problem and adds explainability by implementing one of a dozen or so common algorithms–without testing whether the explanations offered are helpful.

In this chapter, we propose an alternative view: explanation is a fundamentally human process, required to understand the world (Craik, 1943). When an automation or AI requires explaining, this can be supported in many ways. This can include non-algorithmic approaches (training, user help, explorable interfaces), and although it can involve many general algorithms designed under the umbrella of XAI (e.g., LIME, SHAP, and others we discuss later), it can also be supported by task-specific and work-specific interfaces, information displays, and drill-down interfaces often associated with “transparency” and “situation awareness”. Check-box explainability (simply adding an algorithm to a system without testing its effectiveness) is likely to fail, because it can only help if it addresses why a user needs an explanation in the first place.

For a human-centered approach to explanations (Mueller et al., 2020), we first need to understand what explanations are, and the role they play. It is no coincidence that Craik (1943), who developed the first notion of a mental model, did so in his quest to understand explanation. He argued that explanation itself is contextual–we need to understand what explanation someone would find satisfactory in order to create one. He also suggested that the power to explain involves “the power of insight and anticipation”–predicting the future because we understand a process. His primary hypothesis is that this explanation is driven by mental models we form of the world. The property that makes something an explanation is NOT a property of a set of statements, images, numbers, algorithms, or graphics. Explanation is an interaction between the artifacts providing the explanation, the reasoner who is understanding the explanation, and the context/needs they have (Hoffman et al., 2021; Mueller et al., 2020) (see Figure 8.1).

Explanation is ultimately a sensemaking process (Klein et al., 2021), involving abductive reasoning (Hoffman et al., 2020). When using an AI system, it occurs whether or not a developer added an XAI feature. The specific needs depend on the goals of the user, so the best explanation is contingent on their goals and uses. It is informative to review the history of explanation in AI systems–which dates back to the earliest expert diagnostic systems that emerged in the first AI boom in the 1970s and 1980s. This shows how explanations were developed and how they often failed, and the lessons learned from these failures that still apply today.

8.2 The Historical Context of Explainable AI

For a more in-depth discussion of the historical context of XAI, Mueller et al. (2019) covered much of the history and literature on explanation. The first artificial intelligence systems with commercial aspirations were launched in the 1970s. MYCIN (Shortliffe & Buchanan, 1975; Shortliffe, 1976) is often considered the first AI system in the domain of medical diagnosis, and originally consisted of a complex database of medical facts and a diagnostic reasoning system on top of it, which allowed a user (a clinician or physician) to enter facts about a case, interact with the system (e.g., to supply missing information), and get a diagnosis. These AI programs were called expert systems because they were intended to make decisions that experts normally made. For MYCIN, a session would include facts about the case (test results, demographics about patient, signs, symptoms, etc.), rules for carrying out different goals, logic that would allow the system to pursue the solving of goals based on known information, and the ability to ask why and why-not questions which served as basic explanations. The inset below shows a hypothetical reasoning trace, starting from facts and rules to deduce an outcome. Here, the system classifies a bacteria based on tests and morphology, and asks for information to complete the rule.

These systems were very sophisticated even for today's standards, incorporating detailed knowledge and reasoning to form a conclusion about facts based on rules. Building these systems required extensive cooperation with subject matter experts (SMEs) , and so it is probably true that the developers knew almost immediately that explanations were needed, and so these were supported in some of the earliest systems. By 1982, a survey of clinicians by Teach & Shortliffe (1981) found that the most important rated factor about “demands” of automated diagnostic systems was that the systems should be able to explain their diagnostic and treatment decisions to physician users. In comparison, demands like it should never make an error in treatment or planning or should never make an incorrect diagnosis were rated near the bottom of 15 such statements. Physicians wanted explanations more than they wanted accurate diagnoses. This likely reflects the understanding that the physician is ultimately responsible for the diagnosis, and having a system that could help them with their reasoning might be valuable, but only if the reasoning was understandable.

There were many verbal question patterns MYCIN could recognize, which could be done either during or after a consultation. These included not only why a question was being asked, but how a conclusion was reached and why alternatives were ruled out (Scott et al., 1977). But as Teach & Shortliffe (1981) later documented, explanations were demanded by physicians and clinicians, and these could be generated, but in ways that were sometimes not helpful. For example:

In the first example, the consultant asked WHY when MYCIN tries to find out whether the illness was hospital-acquired. The response was that the response is important, and if true would lead to a categorization. But just as Craik (1943) argued, we need to understand what the questioner would deem a satisfactory explanation. They may be wondering “why does it matter”, and a knowledgeable epidemiologist might answer that Enterobacteriaceae is almost never acquired in the hospital; or they may not be sure where it was acquired and want to know if it makes a difference, and really want to see the alternatives. The system succeeded at showing it had a rule that was almost matching and this one piece of information would allow it to draw a conclusion. Other explanatory examples have a similar myopic view–recapping the internal reasoning rather than justifying the conclusion in terms of real-world knowledge.

These systems were quickly rejected by physicians for many reasons. In part, they solved problems the physicians did not have, and importantly, the explanations they offered—even when they could answer why and how questions—often failed to satisfy users. Reciting a rule number, a certainty factor, and “the last question asked was 7” is a faithful trace of the program's control state, but it is not a usable clinical rationale. This failure was emblematic of what has been called the “myth of recap as explanation” (Swartout & Moore, 1993). At this time, researchers began distinguishing between mechanistic explanations that describe how the system works (akin to mechanistic mental models discussed in Chapter 7), and justification, which explain using rationale external to the logic of the system. As an example, a recap-explanation or mechanistic explanation might tell you what sequence of rules was used to deny a loan; a justification tells you the loan was denied because debt was too high. There was also a distinction made between “global” explanations (how a system worked generally; a description that might help form a mental model) and “local” explanations (normally, recaps or justifications of specific cases or situations–why a specific decision was made) (Wick & Thompson, 1992). A physician or loan agent might want a global understanding of the system–is it based on rules, specific cases, or a complex neural network? This can be useful and provide guidance on trust and reliance, but it is very different from a local explanation of why a loan was denied, or why a diagnosis was given in comparison to an alternative. This was one of the early lessons learned about explanation of AI–the explanation you can generate from the system may not be what the user needs.

The interesting part of these developments is that they were about human reasoning and explanation–early failures in technology-focused explanation led to study of what humans want and find satisfying in an explanation, which in turn led to better system development. Parallel lines of work tried to improve on early explanations: GUIDON as a tutor for medical students, and NEOMYCIN as a restructuring of MYCIN's knowledge so that strategic reasoning could be articulated more explicitly (Clancey, 1986; Hasling et al., 1984). By the late 1980s, the “second generation” of explanation systems emerged that took into consideration many of the problems the early systems exhibited (Swartout & Moore, 1993). These included five desiderata that are still relevant (Table 8.1):

Some of these properties are tied to the era of AI–without access to rich on-line information, users may have expected the system to be able to answer questions about definitions. However, many are direct responses to limitations of the MYCIN and similar systems, and their failure to be useful to human workers who they were being developed to help. Many of these also remain informative for modern XAI systems. There is a sense from reading the research at this time that they had taken an early failure and developed a useful system by understanding human explanation needs. However, by the time these second-generation ideas were taking hold, AI funding collapsed, leading to what was called the “AI Winter”. The expert systems of the day that were developed in academic labs found little commercial success, and the explainability approaches built on top of them were mostly abandoned, perhaps aside from the intelligent tutoring systems that were based on the same databases and reasoning logic, helping medical students learn diagnostic reasoning rather than replacing them.

Over the next few decades, explainable AI systems were studied mostly as niche topics within different communities (Moore & Swartout, 1988; Lacave & Díez, 2002; Brézillon, 1994; Sørmo et al., 2005; Woolf, 2007), and within philosophy and psychology (Malle, 2004). Interest in explainability began to increase again as deep learning neural network systems began to show great success, achieving human-like scores on image recognition benchmarks (Krizhevsky et al., 2012), but in ways that the developers did not easily understand. In contrast to the earlier XAI systems which were hand-coded by the developers, who could readily access the reasoning traces, these new systems were opaque to the researchers who created them. At the same time, machine learning in general and deep neural networks specifically was having success in many areas, including general classification (for diagnosis, financial risk) and natural language processing, in ways that even the developers were amazed by (Karpathy, 2016). Following the pattern of the first era of AI success, explanation algorithms emerged almost as soon as the AI systems began to show success. This time, the SMEs included a wider variety of potential users, and so different kinds of explanation requirements emerged. The first users were the AI developers themselves, and explanations were tailored to be helpful to build better mechanistic mental models of the systems, such as the scheme by Zeiler & Fergus (2014) that identified how inner layers of a neural network were tuned to complex images such as faces and dog breeds. Also, Kulesza et al. (2013); Kulesza et al. (2015) examined explanation in service of debugging.

There were also non-technical end-users who needed other kinds of explanations, such as radiologists and physicians who (still) needed justifications for why diagnoses were made, financial industry workers where actuaries, investors, credit rating services needed to understand why a decision was made in order to trust and justify the results, and increasing areas of automation (vehicles, factories, military settings) where users had to rely on systems but had no understanding of how they worked. In comparison to earlier era where the data was carefully collected from experts, these systems could use data from many sources and the validity, accuracy, or legality of using that data becomes important, and may require explanation. As a hypothetical example, a machine learning system might be trained on employee records and be used to give job applicants scores that might predict success. However, there is information in employment records that are not legal to use for hiring decisions, and others that might be legal but unwise to use. These challenges produce social, legal, and societal risks (see Chapter 9) that were not present in earlier AI systems, such that users could be at the whim of algorithms without any recourse. This has led to legal frameworks that began codifying the “right to explanation” (Goodman & Flaxman, 2017)—the right to be told the reasoning for an algorithmic decision. These kinds of explanations may look different from either the mechanistic “recap-as-explanation” present in earlier systems, or even the justification-based explanations, as they also need to be interpretable by individuals who have no specific expertise in the domain or the AI systems (i.e., loan applicants, convicted criminals, university and job applicants).

Along with a host of novel algorithms and approaches to explanation (which we will cover later in this chapter), some frameworks for understanding the bigger picture of explanation began to emerge. For example, Miller (2017), an AI/machine learning researcher, argued that to get explainability right, AI developers needed to examine and incorporate social science, psychology, and others concerned with the human side—and used the analogy of the “inmates running the asylum” to describe the situation where machine learning researchers were developing algorithms of explanation, but not testing whether they were either effective on humans—without considering the human in the loop. For example, Hoffman et al. (2023) examined 160 papers on Explainable AI and found just 32 that used human evaluation. Figure 8.2 situates those measures in the XAI process.

8.2.1 Transparency, Interpretability, and Explainability

In Chapter 7, we discussed how transparency can be used to improve mental models of a system. Transparency and explainability, and interpretability go hand-in-hand, and are not always considered distinct. Other related terms are also used, including understandability and intelligibility. Models that fail to exhibit these properties are called “black box” models, whereas ones that exhibit these properties are “glass box” or “white box” models. Many authors use some of these terms interchangeably— For example, Doshi-Velez & Kim (2017) use interpretability and explainability as essentially the same thing, and Kaur et al. (2020) noted that there is no consensus distinction among the terms interpretability, intelligibility, and explainability. Lipton (2017) argued interpretability as a higher-level category, with transparency answering the question “how does the model work?” and explainability answering “what else can the model tell me?”. Both of these distinctions are dubious in practice, as transparency is used extensively to describe other aspects of the model (who built it, how it was built, where the data came from, social impacts, etc.), and explainability is broadly understood to go beyond just answering secondary questions about the model.

But even if we ignore the social/ethical and societal aspects of transparency, there are some distinctions in how the terms are used in practice. Explainability encompasses a variety of algorithmic approaches ranging from simple algorithms that have more inherent explainability to complex algorithms requiring post-hoc explanation algorithms (Gunning & Aha, 2019), but in practice Explainable AI is typically used to refer to the latter: a small set of approaches that can be applied to a machine classifier to provide post-hoc “explanations”, where the explanation is typically an annotated image (often with heatmaps), a feature importance display (showing where features were important for a classification), a set of examples that illustrate some aspect of the algorithm (other cases where the same or a different decision was made), or a verbal explanation in the traditional sense. In comparison, transparency is often used to describe interfaces and information displays that provide visibility or observability of underlying states and logic to the user; and these might not involve specific algorithms at all. However, the terms are used in different ways by different researchers in different communities, and so these definitions are in no way authoritative.

So one way to frame the distinction is that “glass-box” or “white-box” models generally refer to systems where the decision-making process is both transparent and explainable by design, whereas “black-box” models are inscrutable by design (focusing on performance), but additional algorithms applied post-hoc can provide explainability.

White-box models are often considered to be intrinsically interpretable and one can explain how they make decisions, what the influencing variables are, and how they produce predictions. White-box systems can be examined using interpretable analysis as opposed to post-hoc analysis. Interpretable analysis examines the model transparency, weighs feature importance (SHAP), and plots partial dependence (Ma et al., 2023). It highlights the contribution of each input variable to the output, increasing explainability. Linear regression and decision trees are common examples of white-box models as they provide less predictive capacity but are considered highly interpretable. White-box systems can often be interpreted by laypersons as the output may be in the form of visualizations, natural language explanations, or simplified rule sets (Hendricks et al., 2016; Liu et al., 2022). In addition to having less predictive capacity, highly explainable and interpretable systems are often less accurate. These types of models are the counterpoint to black-box models which prioritize accuracy over explainability and transparency. With both white-box systems and black-box systems, however, the performance of the system depends on the application domain and input data (Loyola-Gonzalez, 2019).

Black-box models such as neural networks, deep-learning models, or complex ensembles produce more capable output, but users may be unable to understand or explain how the model produced the output. Support vector machine (SVM) models are utilized widely, but the mathematical operations that support them are difficult to understand for both machine learning experts and users (Loyola-Gonzalez, 2019). Since these and other black-box models are difficult to interpret and explain, interpretable analysis cannot be utilized. Using post-hoc analysis can explain the behavior of these models. Behavior and sensitivity analysis can be used to examine the responses to various inputs even when the changes to input data are insignificant (Borgonovo & Plischke, 2016). Surrogate models can also be examined to approximate the behavior of the black-box model (Kuttichira et al., 2019). These aspects of post-hoc analysis of black-box models can improve explainability and interpretability but have limitations in fully explaining the decision boundaries or correctly capturing the interactions between features. It is important also to remember that any explanation generated by a post-hoc analysis should be considered an approximation rather than an explicit representation of the model.

8.3 Human-centered Principles of Explanations¹

Lessons for developing human-centered XAI have emerged across a wide range of literature, extending back to the 1970s. In Tables 8.2–8.5, we identify some of the major concepts from this literature, ranging from aspects of measurement (how do you assess explanation), to components of system architectures (what must it represent and what interactions must it support), to forms of explanations that have been shown to be helpful. Each of these represents a principled assertion that has been made by one or more (often many) researchers across several subdomains, often with substantial empirical support.

8.3.1 Measurement Domains¹

Hoffman et al. (2018) proposed a descriptive model of explanation in AI that distinguishes several measurement approaches for evaluating the effectiveness of explanation. One grouping is a variety of measures referred to as explanation goodness (see also Mueller et al., 2019). These measurement approaches are summarized in Table 8.2.

Explanation goodness.

These measures for assessing an explanation fall into two categories: those that require a human user, and those that can be examined independently of users. Obviously, assessing explanations in the context of human users provides the best picture of the effectiveness of the explanation, but other ways can give early guidance as the level of specificity or clarity is examined. We use the concept of “Explanation Goodness” to describe measures that specifically evaluate the explanation, rather than its impact on a person or the work it is supporting.

As an example, Swartout & Moore (1993) proposed that fidelity, understandability, and sufficiency were properties of good explanations. These might be examined via content analysis by subject matter experts or automated language systems. Similarly, Kulesza et al. (2015) identified several goodness criteria, including iterativity, soundness, completeness, non-overwhelmingness, and reversibility; Ribera & Lapedriza (2019) suggested that explanations should follow Grice's (Grice, 1975) maxims; and Wang et al. (2019) referred to this approach as “unvalidated guidelines.” Such evaluation can be helpful in characterizing and comparing explanations. For example, Mamun et al. (2021) examined explanations developed in a collaborative explanation tool to those taken from other explanation corpora using a number of non-user criteria. These included a number of measures using rubric-based measures judged by independent raters, including correctness, scope (global vs. local, as judged by raters), form (what, why, or how-to explanations) and explanation triggers (what prompted the explanation). There were also measures that were scored algorithmically, such as simplicity (reading grade level and reading ease measures). Altogether, these helped demonstrate the nature of explanations being provided.

Hoffman et al. (2018) provided a validated explanation goodness checklist (see Table 8.3) that can be used to guide a goodness evaluation or serve as a lightweight formative evaluation tool. Although this could be used in the form of a user-questionnaire given after a usability study, it can also be useful by a development team to understand what kinds of explanation capabilities are present (especially in conjunction with the Self-Explanation Scorecard discussed below). Developers might sample explanations provided by the system and make judgments about whether it satisfies the criteria, to help understand the nature of the explanations being developed.

Explanation Satisfaction

. In contrast, explanation satisfaction measures are assessed in the context of a user trying to understand a system or achieve a task. In parallel to the goodness scale, Hoffman et al. (2018); Hoffman et al. (2023) provided a validated Explanation Satisfaction Scale² of seven Likert items (1 = strongly disagree to 5 = strongly agree) that ask the user to rate the explanation they received for a named system. The two instruments use closely parallel wording. Table 8.3 places corresponding items side-by-side. Whereas goodness items are answered YES or NO by researchers or domain experts as an independent, a priori evaluation of explanation properties, satisfaction items are rated by users after they have received an explanation.

As discussed in Chapters 5 and 6, explanation should not be expected to simply increase trust—it should only do so in a context in which it is justified, fostering appropriate trust (or calibrating trust).

Mental models.

Figure 8.2 suggests that one's initial mental model can influence how explanations are processed, and that the primary route for explanations to improve trust and reliance is via developing a better mental model. They suggest this leads to more appropriate trust, improved performance, and ultimately more calibrated reliance. One consequence is that measuring a user's mental model is an effective way to judge the effectiveness of an explanation. Mental models have been central to how researchers have understood the impact of explanation. Merry et al. (2021) defined XAI in terms of mental models, and Rutjes et al. (2019) argued that explanations should target mental models. As discussed in Chapter 7, assessing a mental model can take on many different forms, and usually requires a method tailored to the task (Forrester, 1961; Rouse & Morris, 1986; Doyle & Ford, 1998; Klein & Hoffman, 2008). One example, by Kulesza et al. (2015), used knowledge tests to study users' knowledge of a data system they were debugging. In another study, Mueller et al. (2020) asked participants for reasons that a system might fail after showing examples of successful and failed image classification. Measures of understanding and mental models have also been used to show how explanation can help or distort: it can support models of reinforcement-learning agents (Anderson et al., 2020), produce causal confusion (Druce et al., 2021), mis-anchor early impressions (Nourani et al., 2021), go unused as designed so users self-explain anyway (Kaur et al., 2020), or inflate felt understanding (Rozenblit & Keil, 2002). Self-explanation can also repair a model (Chi et al., 1989; Chi et al., 1994; Chi, 2000). For language models, a verbal account is not trustworthy by default (Zhang et al., 2023); some agent designs try to teach a model of system behavior (Kim et al., 2025). Improved knowledge is expected to enhance trust, satisfaction, and joint performance.

Performance, adoption, and reliance.

Another consequence of an effective explanation is improved performance by the human–AI system. In the context of studying human–autonomy trust, Muir (1994) proposed that system performance is a useful outcome of enhanced trust; similarly, Muir & Moray (1996) demonstrated that performance is distinct from trust, insofar as trust could be reduced by errors that do not impact overall performance. In addition, Karsenty & Brézillon (1995) suggested that the goal of explanation in human–machine systems is to improve cooperative problem solving—a type of performance measure. Consequently, better performance is an important potential outcome of an explainable system. At the early stages, this could simply be a measure of adoption as proposed by the TAM discussed in Chapter 3. Better explanations become design features that may provide hesitant users impetus to adopt the system, either through perceived usefulness or perceived ease of use. Moreover, AI-focused TAM adaptations such as those of Borenstein & Howard (2021) identify trustworthiness and transparency as important factors in adoption. Of course, adoption is only the first step along the way, and improved knowledge and appropriate trust are likely to impact overall performance on tasks—either through fewer errors, or by better task allocation (Ahmed et al., 2026).

8.3.2 Explanation Architecture Features¹

Although explanations have been proposed that take on many different forms, researchers have repeatedly discovered and advocated a common set of properties that an explanation “architecture” might support.

Many of these features are based on the finding that good explanations depend on context. For example, Kass & Finin (1988) advocated for user models. A user model might allow an AI system to tailor an explanation to a particular kind of user, or tailor the explanation to a long-term interaction in which the user is learning over time (see Naiseh et al., 2020). Similarly, Brézillon (1994) and Doyle et al. (2003) have advocated for other aspects of context, including understanding goals, the environment, the user's expertise, and so on. Although this appears obvious, it is remarkable that most existing and proposed XAI systems ignore this and offer a one-size-fits-all explanation based on a data-centric algorithm. In this approach, the XAI generates a generic explanation, it is delivered to the user, and it is simply assumed it will make the system explainable.

Another aspect of context is that explanation systems need to provide explanations relevant to the user's goals and knowledge. Explanations are often not necessary, but become useful in response to specific triggers, such as surprise or a violation of expectation (Lim et al., 2009; Hoffman et al., 2018). Systems that can anticipate the appropriate triggers for explanation will be more responsive to user needs. In discussing case-based XAI, S\ormo et al. (2005) pointed to Leake's (Leake, 1995) work on abductive reasoning, which argued that “when to explain” and “what to explain” are central aspects of abductive reasoning (see also Hoffman et al., 2020, for an argument linking explanation to abduction). Similarly, explanations need to be timely. For example, Alam (2020) found that physicians used different explanations at different points in a diagnostic process, and found that explanations only impacted trust and satisfaction of users of a simulated AI system at critical points during the diagnosis—when errors were occurring or diagnoses were changing. An explanation architecture that is aware of this context will provide an overall more useful system. Timeliness and relevance may be difficult to anticipate by an algorithm, but this might be solved by building systems and architectures that have users drive and request explanations when they are needed.

In the context of teaching users about intelligent software, Mueller & Klein (2011) argued for using experiential methods—allowing users to experience a tool, observe success and failures, and especially learn about the boundary conditions of the tool. Similarly, Lakkaraju et al. (2017) argued that explanations should be explorable, iterative, and interactive as well, resulting in an explanation interface that allows users to answer their own questions.

8.3.3 Properties and Formats of Explanations¹

The previous section identified basic properties of an AI explanation system, without addressing the form or content of explanations directly. In this chapter, we do not focus on presenting a taxonomy of algorithms or approaches that have been tried or implemented to support explanation (as done by Chari et al., 2020). Rather, we focus on higher-level psychological considerations that can serve as general templates for effective explanations. Most algorithmic approaches fit one or more of these categories.

One such category, which has been referred to as focus, is whether the explanation is about how the system works (global) versus why a decision was made (local, but sometimes described as a justification). Both appear important, and although the European Union's “right to explanation”—often linked to the GDPR's provisions on automated decision-making (European Parliament and Council of the European Union, 2016; Goodman & Flaxman, 2017)—focuses on local (“understand why a particular decision was reached”), other policies may favor global accounts (those that advocate for understanding how a system works). Recent research has appeared to focus on local justifications, although one famous exception is the visualization of internal deep learning network layers by Zeiler & Fergus (2014). It is also true that global understanding of any complex system can, and often does, come from a mixture of global and local explanatory material. Little research has contrasted the relative benefits of global and local explanation, but Alam (2020) showed that local justification improved immediate satisfaction of decisions, whereas global explanation impacted final judgments of understanding.

In early XAI systems, explanations tended to take on the form of logical rationales (see Swartout, 1977). These still appear in modern systems (see Hendricks et al., 2016), often in conjunction with other methods. Alam (2020) found that rationales were an important mode of explanation used by physicians in diagnosis, but also found that written rationales did not improve user's satisfaction in a simulated diagnosis system, suggesting that other formats such as visualizations and examples may be more powerful.

A more common approach in recent systems relies on examples and cases (see Doyle et al., 2003). One thing examples can do is to help establish a pattern, which can be important for a user who is first experiencing a system. However, it is not obvious how examples should be chosen. Yang & Shafto (2017) identified a Bayesian approach for selecting optimal examples to help teach categories, and Kim et al. (2016) used an example-based approach that highlighted both typical and atypical examples of a class. Examples are a starting point for explanation, but the examples must be chosen so that they help a user understand a category/concept or a decision boundary; randomly-chosen examples may not be helpful at all.

The fact that example-based approaches often use comparison is a specific instance of a general principle of contrast and counterfactual reasoning in XAI (Miller, 2019). Contrast need not be based only on examples, and can take many forms. In general, a contrast is simply an implementation of a scientific experiment: if only one thing is varied and produces a difference, we can conclude that it caused the difference. The dimensions that are contrasted vary considerably in XAI, from examples (positive versus negative), to features (presence or absence), to saliency approaches (identifying whether a feature or region of an image was used in a decision), to counterfactuals (what a system would do if a particular feature were changed).

In fact, many feature-highlighting approaches (e.g., LIME: Ribeiro et al., 2016; Grad-CAM: Selvaraju et al., 2017) are contrast approaches applied to a feature space. These approaches use heat maps (sometimes called “saliency maps”) to highlight regions of a feature space that were critical for a decision. This can be done by an occlusion-based method: a critical region can be identified by determining how the decision changes when different regions of an image are erased or occluded. If a decision changes substantially, this suggests the occluded region is critical for the decision. These systems are sometimes considered localization algorithms, but highlighting of spatial features is essentially localization.

Another kind of contrast relates to the boundary conditions of the algorithm. Any AI system will have regions of competence and incompetence (Mueller et al., 2020), and we have suggested that experiential training can be helpful at understanding the competency envelope of a system, by showing both how the system works and how it does not work (Mueller & Klein, 2011). This approach maps onto global explanations of a system, but its experiential nature relies on cases and local explanations of situations to support this in a tutorial format.

Both AI and human education practice have been influenced by notions of “active learning.” Active learning (see Settles, 2009 in machine learning and Meyers & Jones, 1993 in education) has had a much larger influence in education, but in both fields, it represents a shift in focus from one of the information a learner is expected to master, toward the learner and the activities the learner does to select or engage in the material being learned. Similarly, both AI and psychology have undergone active vision revolutions (see Aloimonos et al., 1988 in machine vision and Findlay & Gilchrist, 2003 in human vision). These approaches consider the active role of the human, and we believe that the field of XAI can also benefit from a similar shift, toward a focus on active and self-explanation.

In support of this, research from the educational literature has shown how explaining things to oneself or to another can be an important means of learning (see Chi et al., 1989), and researchers have considered explanation as a cooperative process between the human and the system (Brézillon, 1994). Indeed, Kaur et al. (2020) discovered that users typically did not use interpretability tools as intended by their designers, suggesting their need for self-explanation is paramount. Related to this is the role of critique in developing an understanding of a system. Active explanation can be guided by critique processes, and research has shown that critiquing can be an effective way of improving planning and forecasting (Veinott et al., 2010). Automated critic-based approaches have been proposed for XAI systems (see Langlotz & Shortliffe, 1989; Guerlain, 1995). In such approaches, a plan or decision is analyzed and critiques are elicited that, for example, compare risks of the suggested plan with alternatives. Here, the XAI relies on the reasoning of the user in a sort of Socratic dialog to encourage self-explaining.

Finally, we note that explanation is linked closely to many areas of research on persuasion, graph literacy, and information visualization (i.e., clarity). This research is relevant in that it shows the impact of information (the explanation) depends on format, social factors, and individual differences in education and experience of the explainee. This needs to be leveraged in order for explanations to be effective, but this also suggests that the same factors that enhance persuasiveness may be mis-used: convincing a user that a system is trustworthy even if that is unjustified. One example of this is the so-called “illusion of explanatory depth” (Rozenblit & Keil, 2002), which describes how people can be overconfident about their understanding of complex systems. Providing more explanatory material may simply lead people to feel that the system is more trustworthy, and might not actually lead to an increase in understanding or an improvement in performance.

8.4 What Explanations Can Provide

8.4.1 Improving Transparency

Explainable AI works to show what is happening when an AI system does something. Creating human-understandable explanations can improve transparency for the users (Das & Rad, 2020). This allows the user to use the AI more productively and prevent bad decisions and conclusions. When the user understands what is going on within the AI system, they are better prepared to use the AI. An explanation of what transparency is and why it matters can be found in Chapter 7.

8.4.2 Improving Trust

Trust in the system is important to the user and their understanding of how and when to use the system. Explainability can help to form this trusting relationship (Shin, 2021). When the user understands what the system is doing or how it came to a specific conclusion, it is easier to trust. The user has something that allows them to form trust with this model beyond trial and error or pure faith alone. The presentation of the explanation can also help to form a trusting relationship. When the presentation is based on the user's knowledge, the user may trust the information more. They are able to form a connection with it since it provides information that they can understand. For more information on trust see Chapters 5 and 6.

8.4.3 Understanding and mental models

Figure 8.2 identifies that good explanations lead to better mental models and improved understanding of the system. This comes initially via past experience, instruction, and training, but a good explanation capability should improve and revise one's mental model. Different types of explanation can help users develop both better mechanistic and functional mental models—understanding the underlying representations and algorithms, but also helping to understand the competence envelope, appropriate use cases, and workarounds for the system.

8.4.4 Improving Understanding of Shortcomings

Understanding what an AI system can and can't do is an important feature of use. Explainable AI can help users understand what the system can't do (Adadi & Berrada, 2018). This allows the user to have enhanced control when something has gone wrong or not worked as expected. The explanation can help to describe why the system may have been unable to do what the user expected. This allows the user to redefine the way they view the system and how they interact with it. Additionally, the model could better improve because you are interacting with it in a way that it can use and understand (Adadi & Berrada, 2018). The system is better able to fit into the expectations of the user since the user is informed about the abilities the system possesses.

8.4.5 Limitations

O'Hara (2020) argued that no AI system can be truly and entirely explainable because the system makes connections that the user is unable to understand. Due to this, O'Hara states that some portion of the system is based solely on trust and that fact can't be changed. Even when you try to create better explanations, some portion of the system will stay obscured to the user. AI doesn't, and shouldn't, make the decisions, but rather be used as a decision aid for the human user. There are some AI systems that do not need human interaction to work, but O'Hara argues that the user is still making the decision to not interact with it.

Another limitation that explanations have is the specificity of the language that they use within the explanations (Yang et al., 2023). Some explanations are too generic, not providing enough information for the user to truly understand the system to the extent that they wish. On the other hand, some explanations use jargon that is too domain specific or too high level that everyday users may not understand. It is extremely difficult and possibly impossible to create an explanation that would work for all users. Explainable AI needs to be based around the specific audience that will be using the AI system. Most metrics for explanations do not include the subjective experience of using the explanation. Rather, explanations are judged by how accurate the explanation is, which won't help to understand how well the user understands the material.

Yang et al. (2023) also explain another limitation within the long-term use of explanations. As the AI system is updated and changed, the explanation must also be updated and changed. If the explanation is too stagnant or can't be updated, the explanation will become completely obsolete. In the worst cases, an outdated explanation may confuse the user about how it works or give them factually incorrect information. For the users that understand that this is wrong, it will break down user trust and make the explanation decrease value rather than add it.

8.5 Approaches to Explanations: Algorithms and Strategies

Molnar (2025) provides a comprehensive text covering many standard interpretable and explainable algorithms. This includes many “interpretable” models (easy-to-interpret by design, such as linear and non-linear regression), as well as “model-agnostic” approaches that are used in conjunction with black-box models. Many of the algorithms are explained in detail at an accessible level, but with sufficient detail for selecting and understanding these systems from the perspective of a machine learning researcher or a data scientist seeking to select or add interpretability to their own system. From the human-centered perspective, the discussion is fairly technical, and so in this section we will cover several families of explanations from a functional and descriptive perspective. These approaches can be supported by a number of specific algorithms, and it is important to recognize that the “best” explanation for a system probably does not depend on choosing the right algorithm, but rather understanding the explanation needs of the user and providing them the information they need to make a decision or take action. Many of the general approaches were covered in Table 8.5, and here we identify and describe some specific algorithms implementing these approaches.

8.5.1 Local Interpretable Model-Agnostic Explanations (LIME)

One popular approach to creating an explainable AI system is to fit a “surrogate model” to a black-box model that is easier to interpret than the full model. For example, a linear regression or a classification tree might be interpretable because the reason for a choice can be narrowed down to a few parameter estimates or rules. The obvious problem with this approach is that if the surrogate model is reasonably good, there may be no need for the original black box model itself. But this is often NOT the case, and the complex classification system or recommender system includes complex classification surfaces that cannot be easily fit by a surrogate. The LIME (locally-interpretable model-agnostic explanation) approach (Ribeiro et al., 2016) attempts to address this by providing a scheme for fitting interpretable models to “local” aspects of a specific decision or classification. That is, it only attempts to describe the model decision in the local context of a specific decision (Molnar, 2025). If the surrogate model is interpretable and can be used to make good local predictions analogous to those of the black-box model, it can help understand why the black-box model is operating as it does for a particular case (and perhaps situations like it). If you take an interpretable model made by LIME and try to apply it to a problem outside of the target domain, it is not likely to make predictions like those made in the target domain, and so its explanations may also be unhelpful.

LIME models can be used on various forms of data, classification models, and with several different surrogate models. Versions exist for tabular data (predicting an outcome based on many numeric predictors), written text, and images (Molnar, 2025). Generally, the output of a LIME analysis is to provide either importance weights or “highlighting” to features. For tabular data, a LIME model can be used to examine the different features used by a model and generate a visualization of the impact they have on a certain predictor, such as what features will have an impact on the amount of bikes rented on a given day. The result may indicate the relative importance that a set of features had on a particular decision. For text data, the features become tokenized words or word pairs, and LIME computes the relative importance of these features in making a particular classification. For example, if a black box model was trained to predict whether a comment on social media is spam or genuine, the output would give a relative importance to each word in the text. For image data, some processing is done to determine important segments of the image which are also treated as features, and then the output might be a highlighting overlay that shows the relative importance of each image segment.

The algorithm requires that the classification algorithm be run many times with variations on the classified case to create a new data set that the surrogate model is fitted to. For tabular data, the model would be fit repeatedly with cases that are variations on the classified case–changing values by small amounts in multiple different combinations. For text data, the data set would involve fitting similar text strings by removing individual words. And for image data, the new data set would involve variations on pixels within each segmented region. In this new data set, any feature that is important in the classification will have a large impact on the classification outcome when the feature is removed, and the LIME algorithm simply attempts to quantify the impact of each feature variation. Ultimately, this will help identify the features that mattered more for this particular case.

This kind of explanation only applies to a particular case (it is a local explanation), which has a slightly different meaning than the “locally interpretable” model referred to in the LIME acronym. One consequence is that the particular feature weighting really only applies to the given case. LIME may determine that a SPAM classifier relied on word features like cost, price, or rebate in one case to classify SPAM, but another message that includes each of those words might no longer be very important.

8.5.2 Shapley Values and the SHAP Algorithm

The SHAP algorithm is a distinct local surrogate model, which uses a different scheme (Shapley values) based in game theory to measure feature importance (Lundberg & Lee, 2017; Lundberg, 2018). In the machine learning domain, the prediction of the model is framed as the “game”, features as the players of the game, with each feature having an impact on the outcome of the game. This impact can be calculated by taking one data point and calculating the prediction for this data point. Another data point is then taken at random and each feature of the original data point is swapped for the value from the randomly selected data point and the prediction is run again. The difference in the predicted value is measured for each feature, and these differences are averaged to calculate the Shapley value (Molnar, 2025). The scheme is similar to LIME, but the logic of how the impact of features is determined differs. The major difference is a technical approach to weighting different cases.

Ultimately, SHAP generates values that attribute to each feature the change in the expected model prediction when conditioning on that feature (Lundberg & Lee, 2017)—its Shapley value. Like LIME, each feature is given an importance value, and relative importance can be visualized with standard bar graphs. Figure 8.3 shows a visualization example generated on a California housing price dataset, showing that the location of the house had the biggest impact on the house price. The pink values indicate negative impacts, and the blue indicate positive impacts of changes to a specific case. This illustrates the local nature of the explanation–it only applies to this particular case. Small changes in the geographical location would put the house in a different neighborhood, and increase or decrease its value, but there is no sense in which this gradient applies generally to the entire state of California. Also of interest is that the Shapley value approach puts all features on the same scale. These scores are not like regression coefficients, whose value depends on the scale of the measured variable, but rather on how important they are to producing the housing price prediction.

8.5.3 Accumulated Local Effects

Although LIME and SHAP provide locally-fitted surrogate models, there are similar approaches that provide “global” interpretations–not dependent on a particular case, but generally applicable to the data set. Typically, for tabular data where many independent predictors are used to make a prediction, this amounts to an examination of each predictor/feature.

One example is the Accumulated Local Effects (ALE) plot. Consider a machine learning model that attempts to predict if a patient may develop diabetes. The model will be trained on certain features (e.g., blood glucose levels) and patient outcomes (a diabetes diagnosis or not). If the modeler wants to understand how each individual feature contributes to the prediction, ALE examines the importance of each feature in an algorithmic fashion and plots them. Since this does not relate only to a single individual prediction or outcome, it is a global explanation of the model. Another type of analysis in this space is a Partial Dependence Plot (PDP). The main problem with PDP occurs when features are highly correlated. For example, people with high blood glucose levels may have a specific genetic marker, and also get less exercise and consume more sugar. It may be hard to discern the influence between these features. ALE plots, due to the method of computation, are more able to discriminate between these features. Largely speaking, ALE plots are thought to be interpretable by any layperson, provided that they understand the real-world details of the feature shown in the plot. Figure 8.4 shows a single ALE plot where a simple neural network was built from an open-source diabetes dataset.

The plot shows that as insulin levels increase (along the horizontal axis), the predicted contribution to a diabetes diagnosis increases (along the vertical axis). Since we know that a feature of diabetes is elevated insulin production due to insulin-resistant cells, this makes sense. An individual can find where they fit on the plot with relative ease, but their local information is not included, and the ALE plot accumulates the impact of the variable across all observed levels of the other variables. The hash marks at the bottom of the plot indicate the specific observations that led to the prediction. This “rug” plot shows that there are many data points for insulin levels less than about 300, but very few past 500. The model should be considered more reliable for the ranges on which it was trained.

ALE plots are often shown in a panel illustrating how all features impact the classification (see Figure 8.5, which includes the insulin plot). By examining combinations of features, we can determine the impact of changing any individual dimension for a specific case, or give blanket predictions about relationships. Although new variables could be generated to look at specific interactions between measures, this is not done by default. Thus, it cannot show the impact of feature interactions by default. For example, if the impact of glucose depended on insulin level, we could not determine this from the base ALE plots.

A number of other algorithms have been used for a similar outcome. For example, Lou et al. (2012) and Caruana et al. (2015) used generalized additive models to provide capable models that were interpretable using plots similar to the ALE approach. Other alternatives are also covered by Molnar (2025).

8.5.4 Explainable Neural Networks

Artificial neural networks are perhaps the epitome of a black-box model. Even if we ignore Generative AI systems and large language models, the processes, data, representations are often inscrutable to users and developers themselves. Explainable network nodes can provide some insight for the data scientists to make decisions about which nodes are important and how many layers would be appropriate given the problem space. These nodes can be informative to those that understand how the NNs function, but they are of little help to the end user trying to make sense of the output.

Image classification NNs might be beneficial for some users who specialize in identifying abnormalities (e.g., radiologists, doctors, quality control technicians). These explainable systems might provide extra evidence and draw attention to regions where an issue occurs and provide some form of reasoning. For example, an explainable neural network could provide some reasoning to a radiologist where it detected cancerous cells based on classifications throughout the layers of the network. One such NN classified COVID-19 from lung CT scans with an explainable architecture that lets users inspect the network's decisions (Soares et al., 2024). If NNs can provide reasoning and steps to identify problems, then they might provide some valuable information to specialized end-users who can read the data output.

Alternatively, a post-analysis explanation of NNs might benefit some users. If the nodes of a network can be explained as a binary decision, then the network might be explainable for end-users. NNs can become more explainable to describe what happens at the node- and layer-levels with language end-users can understand. Blazek & Lin (2021) used this approach to make an explainable neural network that classifies complex shapes.

A number of heatmap-based approaches have been developed that produce overlays indicating the relative importance of different regions of an image in making a decision. These include model-agnostic surrogate approaches such as LIME, but there are other schemes tailored to image recognition networks. These include inferred saliency maps, and gradient approaches similar to LIME that attempt to infer the importance of features, but using the actual classifier model instead of a surrogate. These include GradCAM, Guided backprop, and a handful of similar strategies that highlight which regions were important for the classification (see Adebayo et al., 2018).

8.5.5 Example-based explanations

Example-based explanation methods select particular examples of the dataset to explain the behavior of machine learning models rather than creating summaries of features (such as feature importance or partial dependence). Representing an instance of data in a human-understandable way is crucial for example-based explanations (Molnar, 2025). Because an instance may consist of hundreds or thousands of (less structured) features, listing all feature values to describe an instance is usually impractical. When there are only a few features or we can effectively summarize the instance, example-based explanations work well.

Example-based explanations help in building mental models of machine learning models and the data the machine learning model has been trained on. For example, a doctor was seeing a patient who was coughing and had a fever. The symptoms reminded her of previous similar examples. Thus, she inferred this patient may have the same disease. She ordered a blood sample to test him. The blueprint for example-based explanations is: if thing B is similar to thing A and A leads to Y, then we predict that B will also lead to Y. Implicitly, some machine learning methods are based on examples.

Sometimes AI makes classification errors, like misclassifying bird images as airplanes (Sigler, 2022), as in Figure 8.6. Example-based explanation can retrieve other images from the training data that are most similar to the misclassified bird image in the latent space. Analyzing these similar images reveals that both misclassified bird images and similar images were dark silhouettes. To understand the reason, the study expanded the similar example search and showed the 20 nearest neighbors. Among them, 15 were airplane images, and 5 were bird images. The results indicated a lack of bird images with dark silhouettes in the AI training data because only one of the 5 bird images had a dark silhouette. Therefore, improving the model can be achieved by collecting more data with bird silhouette images.

However, examples alone are not likely to provide sufficient explanation to a user. One reason for this is they have to infer the pattern from explanations that may not exist. One may need to provide either an explorable interface to examine cases, such as the “Discovery platform” described by Klein et al. (2023) (Figure 8.7), or carefully craft and describe examples to illustrate boundary conditions, typical cases, or counterfactual cases, as we will discuss next.

8.5.6 Counterfactual Explanations

Counterfactual explanations are ones that create potential scenarios based on a small change that produces a different outcome. They produce a “what if?” explanation to show what the outcome of a scenario could be if one aspect, mainly the input, of the scenario were different (Verma et al., 2020).

A very common example of a counterfactual explanation is when applying for and being denied a loan (Verma et al., 2020). If someone is denied a loan, they want to know why, especially what factors are contributing most to the decision. If their credit score was higher, they had a different zip code, or had higher income, would their outcome be different? A counterfactual explanation would provide them with information about what aspects of their information are affecting the decision, such as “Your credit score is too low for approval. The average credit score for loan approval is X. If you improve your credit score to X, you can get the loan.” This explains why a person may not have gotten approval, shows which factor is causing the issue, and what the range is to be approved.

Counterfactual explanations are typically used to explain the outcome to the user, such as in the loan example. However, there is the possibility of using counterfactual explanations to predict human behavior by examining all possible changes. According to the MIT Technology Review, Spotify has begun to use counterfactual analysis and explanations to predict user listening behavior to provide music recommendations (Heaven, 2023). Instead of providing an explanation to the user about what change in the input could lead to a different outcome, Spotify is providing developers and algorithms with explanations of how user listening behavior could change to narrow down personal recommendations on the platform (Heaven, 2023).

8.5.7 Explainability and Interpretability for LLMs

Large language models (LLMs) refer to transformer-based neural language models that contain tens to hundreds of billions of parameters, and which are pre-trained on massive text data (Singh et al., 2024), e.g., PaLM, LLaMA, and GPT-4. While they have shown impressive capabilities in understanding and generating human-like texts, LLMs are notoriously complex “black-box” systems. This is because their inner working mechanisms are opaque, and the high complexity makes model interpretation much more challenging.

Importantly, the LLMs bring a new set of problems for explainability and interpretability. They still are inscrutable so that we do not easily understand how they come to a response, and they also have important implications for privacy, fairness, and non-discrimination principles, thus implicating legal and ethical risks (Singh et al., 2024).

However, they also present new unique challenges. Probably the most important challenge is that an LLM is self-programmable, so that it engages in algorithms and heuristics to accomplish tasks layered on top of the transformer architecture. Just like people, it can make and follow verbal plans to solve problems. This means that the algorithms used to solve problems are now divorced from the underlying architecture (Marr, 1982), which can make the results difficult to validate or verify. And just like humans who have limited ability to introspect about their own processes (Nisbett & Wilson, 1977; Ericsson & Simon, 1984), LLMs may not be able to accurately describe the underlying processes used to come to a decision—even though they can verbalize an explanation. Finally, many generative AI models are combinations of several specialist models involving machine vision, written language, and skills to write code to solve more difficult problems on a local computer. It is likely that the explanation algorithms tailored to text, images, or tabular data do not work “inside” the LLM that is nevertheless using various black-box models. This means that the special-purpose explanation algorithms may be unusable. If you ask an LLM to identify a picture of a bird, it may be correct, but if you want an explanation, it may give the same kind of fabricated verbal descriptions that were mixes of verbal descriptions and class definitions, rather than the critical features used to make the classification (Hendricks et al., 2016). Nevertheless, they are explanation machines—they provide verbal explanations of activities, rationales, plans, and tasks, so unlike many other systems where explanations are added on as a post-hoc analysis, the explanation is built in to the text-based nature of the prompting system.

Adding to this concern is the phenomenon of hallucination, wherein LLMs may generate incorrect or baseless explanations, leading to potential confusion or misinformation. Some have argued that hallucinations are implicit to the architecture itself (Zhang et al., 2023). This means that the output of these systems is not completely trustworthy by default, which requires a secondary level of supervision and checking. But together with its self-programmable nature, architecture independence, and facility with verbal explanation, this means that one cannot always completely trust the explanation, justification, interpretation, or other account of the system.

They also have important data-related issues that require consideration from legal and copyright perspectives. To some extent, these systems are like a complex database that stores and retrieves knowledge, including information covered by copyright. This information is not legal to redistribute, and so some view it as problematic that LLMs can produce this information.

Furthermore, the sheer immensity and opaqueness of LLMs, characterized by models containing tens or hundreds of billions of parameters, present significant hurdles to interpretation efforts. Understanding the inner workings of these complex systems requires innovative approaches to distill essential insights and present them in a comprehensible format, thereby facilitating understanding and trust in LLM outputs.

Zhao et al. (2024) categorize techniques based on the training paradigms of LLMs: a traditional fine-tuning-based paradigm and a prompting-based paradigm. The traditional fine-tuning paradigm involves initially pre-training a language model on unlabeled text data, followed by fine-tuning it on labeled data from a specific domain. During fine-tuning, additional fully connected layers are appended above the final encoder layer, facilitating adaptation to various tasks. This approach has demonstrated success primarily with medium-sized models like BERT or RoBERTa, typically containing up to one billion parameters. Explanations within this paradigm predominantly revolve around comprehending how self-supervised pre-training enables models to grasp language fundamentals and how subsequent fine-tuning equips them to effectively address specific tasks.

Conversely, the prompting paradigm employs prompts, such as natural language sentences with blanks, for zero-shot or few-shot learning without requiring additional training data. Models within this paradigm are categorized into base and assistant models. As LLMs increase in size and training data, they exhibit notable advancements, including the ability for few-shot learning through prompting. Explanations concerning base models center on understanding how these models utilize their pre-trained knowledge in response to prompts, although they often encounter difficulties in following user instructions and may generate biased or toxic content. To mitigate these shortcomings, base models undergo supervised fine-tuning to attain human-level abilities, such as engaging in open-domain dialogue. Explanations in this context concentrate on elucidating how models acquire interactive behaviors from conversations, with the objective of aligning responses with human feedback and preferences.

Finally, the evolving architectures for LLM systems may provide some opportunities for better explainability. For example, the Retrieval-augmented generation (RAG) approach supplements trained-in knowledge with retrieval from sets of documents, and because those documents can be vetted and identified as accurate, output from the model can even link to those documents as provenance for the information (Hou et al., 2026). In addition, “agentic” and “multi-agent” systems orchestrate a set of special-purpose agents working together to generate output. Some hallucinations can be mitigated by having individual agents whose job is to verify information from external sources (Li et al., 2025). Similarly, multiple LLM agents can be used to support trustworthy explainability by having multiple agents independently identify responses and come to a consensus (Bandara et al., 2025), or by engineering agents that attempt to teach users and encourage an accurate mental model of the system behavior (Kim et al., 2025).

8.6 Human-centered methods for evaluating explainability

8.6.1 Trust in the XAI context

For explainable-AI evaluation, Hoffman et al. (2021); Hoffman et al. (2023) published a short Trust in XAI scale as part of a battery that also measures explanation goodness, satisfaction, mental models, curiosity, and human–AI performance. The trust scale is eight Likert items (1 = strongly disagree to 5 = strongly agree) plus an open prompt for reasons. Item 6 (“I am wary…”) is reverse-coded.⁴.

This instrument was developed to characterize trust in a system in the context of explanation provided by the AI. It is a state rating of a named tool applied during or after use, rather than a propensity-to-trust scale. It is also explicitly componential in the Chapter 5 sense: it incorporates confidence, predictability, reliability, safety of reliance, efficiency, wariness, relative competence, and liking. Scharowski et al. (2025) later supported its use in AI contexts more strongly than a single Jian total. Table ch06-xai-trust in Chapter 6 reprints the items; Table 8.6 repeats them here.

8.6.2 The Self-Explanation Scorecard¹

One important perspective from understanding the psychology of explanation is that, for explanation-enabled AI systems, the system may provide images, text, or figures serving as an explanation, but these are typically pieces of information used for self-explanation. Different kinds of information provide better explanation affordances, with some providing little or basic support for self-explanation, and others providing rich information. Klein et al. (2019) described a “Self-Explanation Scorecard,” which is a set of capabilities provided by an AI system that specifically support self-explanation.

The Scorecard is an ordinal ranking of explanation types that map generally onto the complexity of the required reasoning that it supports. This order is not an assertion about which levels are likely to be most effective for any specific case, or that a particular explanation at a higher level will necessarily be more effective than another explanation at a lower level. However, we do suppose that in general, higher-level systems provide more relevant information to support the user in their attempt to self-explain. The levels of self-explanation and examples are described in Table 8.7.

Although many AI systems have no explicit support for explanation (null), many expose different features to help understand the analyses done by the AI. A user will typically self-explain by blending these features with other information at higher levels of the scorecard. An AI system that shows examples of successes supports self-explanation because it grounds abstract processing in specific cases. AI systems that expose decision logic can help the user understand rules for why an outcome occurred. Failures show places where the AI does not work, and can help understand limits of the system. Any of these simpler levels can form the basis for comparisons, contrasts that help a user make sense of critical factors in the AI. Finally, explanations that help users diagnose errors and failures might be the most powerful, as they enable understanding of the contrasts that make the difference between success and failure.

Together, these human-centered approaches offer a distinct approach toward understanding explanation in AI and automated systems. Frequently, an algorithm such as SHAP or LIME is implemented in an application to provide checkbox explainability, providing additional output but not understanding the human decision, the work, or the explanation need, and never determining if the explanation helps understanding, trust, adoption, or performance. These approaches (trust in XAI, explanation satisfaction, and explanation scorecard) provide three simple ways human factors approaches can be deployed to provide better design and evaluation of explainable systems.

8.7 Design Principles for Human-centered Explainable Systems¹

Tables 8.2–8.5 distilled a range of scholarship that has established architectures, properties, and measurement approaches that have been used to develop explainable systems in the past. However, simply applying these patterns to new systems is no guarantee for success, and not all of the principles, as stated, are immediately actionable. These are measures, architecture features, and explanation types that have been shown to be effective in some contexts, but they are likely to fail in others. Any strong conclusion about explainability must be based on empirical evidence, and requires testing in the context of actual work with the intended representative users of the system.

Nevertheless, we may be able to identify some of the most central principles that should be observed when developing human-centered XAI. These systems may indeed start with an algorithmic explanation, but in order to succeed, will need to work in the context of a work domain to help the user achieve their goals. Some of the principles we have identified include:

Explanation is a process, not an artifact.

The property of being an explanation is not a property of statements, visualizations, or examples. Explaining is a process by which an explainee and explainer achieve common ground, or a user self-explains a system with information displays provided by the system. XAI designers should recognize that an explanation is not merely an artifact delivered from an algorithm, but must be understood by a user to be effective.

Work matters.

It is impractical to develop a useful and usable explanation system outside of a work context. Explanation relates the tool to the user knowledge in the context of the goals and tasks, and whether a particular algorithm or visualization will support that work cannot be known in the abstract. Human-centered design principles suggest involving the user early and throughout the system development process (Greenbaum & Kyng, 1991; Deal & Hoffman, 2010; Hoffman et al., 2010; Clancey, 2020).

The importance of active self-explanation.

The “spoon feeding” paradigm is oblivious to the fact that irrespective of whatever material people receive by way of explanation, they still engage in a motivated attempt to make sense of the AI and the explanatory material. Developers should recognize this and focus explanatory systems on information that empowers users to self-explain, rather than simply delivering an output of an algorithm that is intended to serve as explanatory.

Build explanatory systems, not explanations.

Rarely does the initial explanation coming directly from the AI algorithms provide a useful explanation, let alone an ideal one. It must be accompanied by other things (instructions, tutorial activities, comparisons, exploratory interfaces, user models, etc.) to succeed.

Combined methods are necessary.

Much work on XAI involves testing a particular concept or algorithm in isolation. But when designing an explanatory system, multiple kinds of information can complement one another. For example, both global and local explanations may be justifiable and reinforce one another. Showing examples that establish a pattern may play a different role than contrasting examples which establish a critical causal relationship. Using examples of related heatmaps may be more powerful than either examples or heatmaps in isolation. And it is crucial to keep in mind that actual work contexts involve multiple systems, so the user is continuously challenged with their own “system integration” task.

An explanation can have many different consequences.

Often, developers create and test explanations to determine whether they work. However, different explanations can have very different effects. This includes differing effects on qualitative assessments (satisfaction, trust), versus knowledge measures and performance measures. The explanations should be tuned to the goal, keeping in mind the fact that people may be led to trust and rely upon an AI system simply being given more and more information about it, whether or not that information leads to better or deeper understanding.

Measurement matters.

Because explanations can have their impact in a number of ways, and so can be assessed along many dimensions (goodness, satisfaction and trust, knowledge/understanding, and performance). Designers should identify what consequences the explanation should have in order to develop an appropriate measurement and assessment approach.

Knowledge and understanding are central.

Much of the research on XAI focuses on algorithmic visualizations, which distracts from the fact that the focus of explanation is on developing a better understanding of the system. Understanding leads to appropriate trust and appropriate reliance, and therefore overall better performance with the system.

Context matters: users, timing, goals.

An explanation is not a beacon revealing the truth. The best explanation depends on context: who the user is, what their goal is, when they need an explanation, and how its effectiveness is measured. Developers should consider use cases, user models, timeliness, and attention and distraction limitations for their explanations.

The power of differences and contrast.

A central lesson of XAI is the utility of contrast, comparison, and counterfactuals in understanding the boundary conditions of a system. A useful exercise is to first develop learning objectives for an explanatory system, and identify the contrasts necessary to support those objectives.

Explanation is not just about transparency.

If something is transparent, you cannot see it. This word is widely misused. What is needed are systems whose workings are apparent, that is, readily understood and not hidden (the “black box metaphor”). Especially in the context of fairness in algorithmic decision making, many have advocated “transparency” as an approach to explanation. This can never be enough, because a user may still not understand how a system works even if its algorithms are observable and visible. Other methods (contrast, global explanation and local justification, examples, explorable interfaces that permit hypotheticals, etc.) will be necessary in most situations to harness observability and develop understanding. In “real world” work contexts, people always feel some mixture of justified trust, unjustified trust and justified and unjustified mistrust. These attitudes are in constant flux and rarely develop in a smooth progression toward some ideal and stable point. Trust can come and go in a flash. When the AI fails in a way that a human would never fail, reliance can collapse.

The need for explanation is “triggered.”

Too often, XAI systems deliver an explanation regardless of whether one is needed. However, explanations are not always necessary. In normal human reasoning, explanation is triggered by states such as surprise and violations of expectation. Advances in XAI will come when systems begin to understand situations that are likely to engender surprise and violate user expectations.

Explanation is knowledge transformation and sensemaking.

The achievement of an understanding is not just the learning or incorporation of information; it involves changing previous beliefs and preconceptions. The acquisition of knowledge involves both assimilation and accommodation, to use Piagetian terminology. The power of explanation is that it can activate fast “System 2” learning modes that quickly reconfigure knowledge with minimal feedback, bypassing the slower “System 1” trial-and-error feedback-based learning often used to understand a system. XAI systems should harness this by attempting to identify the user's current understanding (so that it can better predict how to transform this knowledge), and support the information that will help make these transformations.

Explanation is never a “one-off.”

Especially for AI systems that learn or are applied in dynamic contexts, users often need repeated explanations and re-explanations. How has the algorithm changed? Are these new data valid? XAI systems might benefit from considering the long-term interaction with users, even in simple ways like recognizing that once learned, an explanation may not need to be given again unless something has changed.

8.8 Conclusions

Explainability is a property of a human–AI work system, not simply the output of an algorithm attached after the fact. A human-centered approach to XAI needs to first understand the explanation needs of the user and the reasoning modes and information that will support these needs. There can be many ways to satisfy these needs, including non-algorithmic approaches (training, user groups, web forums, explorable interfaces). There are a variety of algorithms that have been developed to provide either global explanations (about how the system works in general) or local explanations or justifications (about why a specific decision was made). These can often address explanation needs of users, but at times they may require modification, customization, combination with other approaches, or augmentation with non-algorithmic approaches.

To avoid check-box explainability, developers should evaluate the impact of explanation on users. This chapter describes a number of such approaches, with measurement distinctions among goodness, satisfaction, mental models, and performance. In addition, we describe a formative evaluation (the self-explanation scorecard) and a set of user-centered principles that can help create better explainability by design. The approaches reviewed here all point to the same requirement: to establish explainability, the systems need to be designed for the user's work, tested with people, and revised as the system and its context change.

Acknowledgments

This chapter was originally developed as part of a course project for HF 5430 Human-AI Interaction, Michigan Technological University, Spring 2024.

Author Contributions

KU, BF, CC, LS, SW, JW, YW, BW: writing and research of individual sections; STM: Conceptualization; background research, writing in all sections; general editing. Several sections and tables were adapted from Mueller et al. (2020).

Conflicts of Interest

The authors declare no conflict of interest.

AI Usage Statement

Generative AI models were used for additional research, to identify missing concepts, to support better organization, and for editorial tasks such as formatting, evaluating grammar/clarity and citation collation, glossary development, and creation of figures. Content, text, and ideas are otherwise original to the human authors.

How to Cite This Chapter

Ulinski, K., Frisch, B., Cischke, C., Sprague, L., Walker, S., Wang, J., Wang, Y., Woolman, B., & Mueller, S. T. (2026). Explainable and interpretable AI: Human-centered aspects. In Shane T. Mueller (Ed.), A Handbook of Human-AI and Human-Automation Interaction. https://pages.mtu.edu/~shanem/human_ai/