Chapter 2

Human-AI Interaction Research: An Overview

Katie Ulinski, Brandon Woolman, and Shane T. Mueller
Michigan Technological University

2.1 Introduction

All technology ultimately involves humans: it is made by humans, for humans, with the goal of helping humans, and sometimes with the side effect of harming humans. Despite this, technology is often constrained by engineering compromises, practical trade-offs, and a lack of understanding of the user. This can make it difficult to use, hostile to subgroups of users, hard to learn, unreliable, and unable to be trusted. Various fields of research and design, including human factors, ergonomics, human-computer interaction, cognitive systems engineering, human-systems integration, and UI/UX design have emerged to improve design and evaluation methods in order to ensure technological systems that are directly used by humans are usable and ultimately useful to them. To paraphrase Walt Whitman, the field is large, and contains multitudes.

AI and automation are often thought to have the goal of replacing the work of humans, perhaps placing it outside the realm of human-machine interaction. Although this is certainly true in some cases (traffic cops wear white gloves to help drivers see their hand signals; traffic lights avoid the need for improving the human-glove interface), other problems are amplified (e.g., when autopilot is engaged, a pilot may become more likely to lose situation awareness) and new challenges are created (e.g., human-automation hand-off protocols may be required that were not previously needed). As discussed in Chapter 1, (Bainbridge, 1983) identified “ironies” of automation, suggesting that automation may make some tasks easier, but these often lead to changes in how the human works–monitoring, supervising, and taking over in difficult situations. So the approach in many of these disciplines is to understand the interactive system and how work changes to support it.

A number of related fields of study focused specifically on human-AI and automation interaction have consequently emerged. Some are used essentially interchangeably, but others refer to specific approaches or techniques. In this chapter, we will discuss some of the prominent schools of thought and research and design communities. To close the chapter, we will describe a basic taxonomy of the kinds of functions intelligent software (AI and automation) typically has that separates itself from general technology, and may be the avenue for interventions that ultimately improve human-AI interactions.

2.2 Schools of thought for designing AI and automated systems for human users

Research and practice in human-AI and human-automation interaction takes place in a number of communities of practice (Xu et al., 2021). As discussed in the first chapter, cybernetics (Wiener, 1948) was and remains an appropriate blanket title, but many other relevant terms have been coined. Among many, these include cognitive systems engineering (Woods & Roth, 1988), joint cognitive systems (Hollnagel & Woods, 2005) human-centered AI (Shneiderman, 2021), human-AI interaction (Amershi et al., 2019), human-AI collaboration (Wang et al., 2020), human-automation interaction (Sheridan & Parasuraman, 2005), human-machine teaming (Lyons et al., 2019), human-AI teaming (Berretta et al., 2023), mixed-initiative interaction (Allen et al., 1999), AI-assisted decision making (Zhang et al., 2020), human-in-the-loop AI (HITAI; Zanzotto, 2019), and there are certainly others. Although there is strong overlap among many of these areas, this section will discuss the basic approaches for three of them: human-AI interaction, human-centered AI, and cognitive systems engineering. As shown in Figure 2.1, the distinction between these involves what the target is emphasized. In HAII is the broadest term, and is the study of two agents (human and AI) and the interaction between them. HCAI attempts to “center” the human, and that AI systems should be designed with human goals and purposes in mind. CSE views humans and their tools (including AI and other machines) as interdependent entities in a joint cognitive system, and examines how the system is perterbed and changed by adding automation or intelligent machines.

2.2.1 Human-AI Interaction (HAII)

The field of human-AI interaction (HAI or HAII) is a generic term arising from an analogous domain of human-computer interaction, which is concerned with user interfaces and how users interact with computers, and more generally technology as a whole (Carroll, 1997). HAII might properly be considered a subfield of HCI, although Xu et al. (2021) proposed that HAII is an emerging interdisciplinary field. In HAII, AI is the focus of what humans are interacting with, assuming that AI systems have some interface that users are aware of and utilize (Wienrich & Latoschik, 2021). This includes both the hardware and software levels of an AI system and assumes that both have some effect on the user's interaction. Most often, when discussing HAII, the AI portion refers to a system that has intelligent components or capabilities, rather than an individual standalone AI system.

HAI/HAII is comprised of a multitude of numerous disciplines. For example, Xu (2019) identified that it involved three main components: First, AI should be ethically designed; it should avoid discrimination and should not replace humans. Second, the technology should reflect human intelligence and should be used to advance the capabilities of humans. Third, human factors principles should be used to design systems that are explainable, useful, and usable. However, Xu et al. (2021) subsequently identified that HAII is at the nexus of HCI, AI, human factors, design and engineering, psychology, social science, data science, cognitive neuroscience, and computer science. These overlapping fields involve a number of intersecting research domains, including explainable AI, intelligent interaction design, human-AI collaboration, augmented cognition/intelligence, human-machine hybrid intelligence, AI machine behavior, human-controlled autonomy, and ethical design (see Figure 2.2).

Amershi et al. (2019) offered a design-oriented approach to characterizing human-AI interaction. Based on extensive review of the literature, they identified and codified 150 AI-related design principles and recommendations, distilling these to 18 guidelines in 4 broad categories. Interested readers can refer to Amershi's complete set of guidelines, which we summarize in the four contexts identified by the authors in Table 2.1. Interestingly, many of the specific guidelines are not specific to AI-enabled systems, but are relevant to many kinds of systems. For example, guideline G1, “Make clear what the system can do”, is good design advice for almost every consumer and electronic product.

Overall, the broad field of HAII may be criticized in a number of ways. First, it places the user and the system as separate “agents” and their interaction as the thing that needs to be studied. Other approaches center the user in the process (Human-centered AI), or consider the “system” as involving the humans, the technology, and the interactions (e.g., Cognitive Systems Engineering), or consider the human and intelligent system as cooperative players on a team (mixed-initiative interaction; human-AI teaming).

2.2.2 Human-Centered Artificial Intelligence (HCAI)

Human-centered AI (HCAI) is an evolution of the human-centered design movement to tackle similar problems in AI interaction (Xu et al., 2021). Shneiderman (2020) proposed that HCAI is a “Second Copernican Revolution”, which in contrast to the first revolution (that displaced humans from the center of the universe), returns them to the center of the design process, and supports AI applications that should be developed with the human user experience as the central approach. This approach is more prescriptive than the term human-AI interaction or many of its component disciplines, because it suggests a particular approach to research and practice.

In comparison to general human-centered design problems, most users do not and may be unable to understand how AI makes decisions and processes information. This creates special challenges for human-centered design of AI that may not be faced in other technologies.

Some HCAI theorists reject the idea that AI will one day replace humans or have all the same abilities as humans. Instead, they argue that humans should use AI as a tool that helps them to enhance their natural abilities, to make better decisions, and to provide abundant information (Shneiderman, 2021; Shneiderman, 2020). According to this ideology, AI should be designed and used with this goal in mind and with the understanding that AI cannot surpass humans. Importantly, Shneiderman argues that designs that attempt to mimic or recreate human-like abilities (including cognitive capabilities) will give way to more effective designs that take advantage of the unique advantages of the machine. Human traffic cops were replaced not by a robot with a whistle and arms, but by a network of lights and sensors that can manage traffic across an entire roadway more efficiently and effectively; a human washing dishes was not replaced by the Jetsons' humanoid robot maid Rosie, but by a box that washes dishes. Similarly, successful AI will not necessarily act or look like the human performing the same job.

Interactive human-centered artificial intelligence (IHCAI) describes practices within the discipline of HCAI that work toward enabling interactive exploration and manipulation of AI, purposefully designed with human benefit in mind (Schmidt, 2020). IHCAI should allow for users to understand why a specific conclusion is drawn and to understand how changing different factors affects the output. This should be in terms that a nonexpert can understand (Riedl, 2019). This will aid users, given that they are unable to form a theory of mind about AI systems. Users should be able to see who has control over the AI system and to know what data, knowledge base, and information the system is using. Understanding the function, purpose, and who is controlling an AI system allows for the user to remain in control of the system.

2.2.3 Cognitive Systems Engineering

A third important perspective is that of Cognitive Systems Engineering (CSE Woods & Roth, 1988), which arose to study how automation and technology were impacting work systems. These include industrial and control-room settings including power plant management and space flight and mission control, air traffic control and commercial aircraft, healthcare settings, military systems, and the like. Unlike the focus of HCAI and HAII, these often contain teams of individuals (instead of a single user) with overlapping but distinct specializations and responsibilities and networks of systems (instead of a single tool) with varying levels of reliability and capability. Each of these can be considered cognitive agents, and the goal is to characterize the cognitive work the system produces and how its parts interact. The initial goal was to study the humans as part of the system, and see how changes in the automation change the roles and interactions.

Although there does not appear to be a subdiscipline of CSE devoted to AI per se, the community has learned and adapted through several generations of AI and automation systems to understand AI as a component of a larger system, to understand how it changes the cognitive work that the system performs, and to address how to incorporate this into system design.

2.2.4 Allied and Adjacent Fields and Sub-Disciplines

There are other labels used to describe aspects of the intersection of humans, AI and automation, that are distinct focuses of some researchers. Some of these are akin to the research themes in Figure 2.2, but others are brands committed to specific approaches, researcher communities who do not only study human-AI systems, or other technical approaches. We mention and define them here to help further characterize the landscape of human-AI and human-automation research and design.

2.2.4.1 Human-Machine Teaming

This goes by several related terms, including, Human-Autonomy Teaming, Human-Agent Teaming, and Human-AI teaming, and is often abbreviated HAT or HMT. A common feature of all of these research approaches is that they take the team as the focus, and view AI or automation as a member of the team (which normally also involves human members). Whether AI can or should be treated as a teammate versus a tool remains somewhat controversial (Klein et al., 2004; Shneiderman, 2022; Naikar et al., 2026), but the approach is useful because it builds on a massive research base focused on human teams, and recognizes that teams involve both shared and unique information, goals, and skills, so HAT often involves studying humans and intelligent systems as collaborators with shared goals, complementary capabilities, and mutual dependence (Berretta et al., 2023; Chen & Barnes, 2018; Bradshaw et al., 2017). Chapter hmt covers this approach more completely.

2.2.4.2 Socio-technical Systems

Socio-technical systems research studies work settings as jointly shaped technical and social subsystems, arguing that automation and AI succeed only when both are designed together (Baxter & Sommerville, 2011); participatory and contextual design approaches in Chapter 3 apply this view.

2.2.4.3 Resilience Engineering

Resilience engineering examines how people and organizations monitor, adapt, and recover when they encounter surprise, stress, or failures (Hollnagel et al., 2006). The mission of this discipline is to design systems that are resilient to disruption, and study factors that impact this resilience. The focus can often be on primarily human or socio-technical systems where the stressors are not automation or AI (such as emergency rooms). However, technology, automation, and AI are often important weak points in the resilience of systems, and potential solutions to improve resiliency. It is closely related to cognitive systems engineering and work-domain analysis (Chapter 3).

2.2.4.4 Naturalistic Decision Making and Macrocognition

Naturalistic decision making studies how experts make judgments under time pressure, uncertainty, and real operational constraints, often using methods such as the Critical Decision Method (Klein, 1998). The community that has developed around these methods has expanded the domains of research to study many kinds of cognition in the context of work and socio-technical systems, which has been referred to as macrocognition (Klein & Wright, 2016). Overall, this frequently involves studying human-automation and human-AI systems, often with the focus on how expertise impacts and is impacted by intelligent software tools.

2.2.4.5 Human-Robot Interaction (HRI)

Human–robot interaction is the study of how people perceive, communicate with, and work with physical robots. Here, robots involve the entire range, from “dumb” industrial robots to humanoid and autonomous robots to social robotics and robot toys. The emphasis is often on embodiment, safety, and shared workspace (Goodrich & Schultz, 2008). This is discussed further in Chapter hmt, and overlaps with social robotics discussed in Chapter 4.

2.2.4.6 Social Robotics

Social robotics focuses on designing robots that communicate, coordinate, and relate to people in socially meaningful ways, often leveraging anthropomorphic cues to support acceptance and interaction (Breazeal et al., 2016). Here, although the robotics often have multiple complementary integrated systems for control, machine vision, language, planning, etc., the focus is usually not on algorithms per se, but on the social, emotional, or trust responses. This is discussed more in Chapter 4 and trust implications in Chapter 5.

2.2.4.7 Coactive Design

Coactive design is an approach to human–robot teaming that focuses on interdependence as the central design problem for designing interactive systems with humans and intelligent agents. Its key principles are to design for observability, predictability, and directability (Johnson et al., 2014); see Chapter hmt.

2.2.4.8 Distributed Cognition

Distributed cognition treats thinking as spread across people, tools, representations, and environmental structures rather than confined to individual human thinkers (Hutchins et al., 2000). The approach does not focus on intelligent machines, but rather how humans extend their cognition to embed memory, perception, and other cognitive processing in the environments they build and design. For example, in “How a cockpit remembers its speeds”, Hutchins (1995) examined the cognitive operations of the commercial aircraft cockpit system that includes teams of pilots, sensors, displays, printed documentation, and automation, describing how artifacts within the cockpit (marks on the speed indicators, lookup tables throttle positions) distribute the cognition outside the mind of the pilot.

2.2.4.9 Human-Centered Machine Learning and Human-in-the-Loop Machine Learning

Human-centered machine learning is a design philosophy akin to Human-centered AI, with perhaps a slightly narrower focus. It involves how machine learning systems can be designed and trained to support humans in general, and users specifically (Holstein et al., 2019). This involves prioritizing trustworthiness, user workflows, fairness concerns, and practitioner needs, and extends to societal/ethical issues.

HCML should not be confused with human-in-the-loop machine learning (HITL-ML). In general, HITL-ML are approaches to embed humans in the machine learning process to enable AI/ML systems to learn and use information that may not be easily accessible otherwise. This can involve human data labeling, model correction, validation, and selecting preferred outcomes, and otherwise interactive refinement of learned systems (Mosqueira-Rey et al., 2023). A specific and popular method for implementing this approach is Reinforcement Learning with Human Feedback (RLHF), although HITL can refer to many other human activities that are used for machine learning, such as human annotation or labeling, error detection, etc. HITL-ML can support HCML if it is used specifically for that purpose, such as supporting broader goals of alignment (having the AI operate in ways consistent with human values).

2.2.4.10 Mixed-Initiative Interaction and Planning

Mixed-initiative interaction describes systems in which humans and computers can each take the lead, contributing their respective strengths toward a shared goal (Allen et al., 1999). This research subdomain has its roots in conversational agents, often with the goal of a human and AI system interacting via spoken language to accomplish a task. Although research on this predates conversational chatbots like Siri, Alexa, and LLM-based systems, many of the examples in this research domain anticipate the kinds of interactions envisioned by modern conversational AI assistants and LLM chatbots. For example, Novick & Sutton (1997) used the following scenario to illustrate a mixed-initiative interaction, where S is the dialog system and U is the user (see Figure 2.3). Here, the final user response U4 implies that the system should take initiative and provide a suggestion such as S5b; a system that declines to take control as in S5a is argued to lead to repetitive interaction and unconstructive exchange. Here, in comparison to typical interactions with AI chatbots/assistants that tend to be command-response driven, this approach has the potential for interactivity, extended use of in context, and correcting/changing/adapting information based on feedback.

Although mixed-initiative interaction traditionally involves linguistic interaction, the human and machine often have a common or shared task or goal that is non-linguistic. For example, Mohan & Laird (2012) demonstrated mixed-initiative interactions in a tank/missile game in which components of a task could be handed off between the human and agent; coactive design (above) extends similar ideas to embodied human–robot teams.

2.3 Evaluating AI and automation for human use

In this section, we provide ideas arising from the different fields that can help a human factors engineer analyze a system involving intelligent software or AI, anticipate problems, and understand potential solutions.

2.3.1 Analyzing Capabilities and Limitations

AI and automated processes make life easier for workers all over the world. Automated machinery on assembly lines significantly increases production while reducing human interaction with products. Despite the rewards of automation, human-computer interaction can prove challenging for workers when problems occur. Troubleshooting heavy machinery can not only be confusing, but dangerous for the operator. While not every instance of troubleshooting AI and autonomous processes is hazardous, they can be confusing and difficult to correct. A number of researchers have identified three broad aspects of systems that users should understand: the system's capabilities and purpose, its limits and limitations, and how to troubleshoot problems and work around its limitations. Borders et al. (2024) suggested that capabilities and limitations of technology are critical aspects of a mental model, and that workarounds are related capabilities of the users. Likewise, Hoffman (2017) discussed justified trusting and justified mistrusting: understanding the competencies and limits of a system.

2.3.2 Components of an AI or Automated System

AI and automation systems generally have a number of information-processing functions they support (see Table 2.2). These include at least one of the following functions: representation and modeling, data handling and data generation, understanding computation and algorithms, and output, display, and visualization (Mueller et al., 2009). To improve human-AI interaction, a designer should examine and understand each of these areas as potential domains for improving or supporting users of the tool.

This support may take on a number of forms that enable human-centered AI design. For example, one might consider two aspects of data and input: training data and user input. Identifying where training data comes from may help inform a user about the capabilities and limitations of the system, and it may suggest ways the tool might be improved by allowing the user to change or update a training corpus. Considering how information is input may lead to a redesign in interaction. By considering the underlying representation of a GPS-based routing system, one might learn that optimization does not incorporate traffic or construction; a user who expects this will be better prepared to work around this limitation. In image classification, algorithms often compute a “soft-max” value that looks like a confidence or probability score, and allows ranking of multiple likely options. But this can be easily misinterpreted. By considering this, a developer may decide on an alternative algorithm or output that is less likely to be misinterpreted. In any of these cases, the consequences might be design changes, changes to the task, or improved training and user instruction.

Considering these aspects of an intelligent system does not mean the user or operator needs a deep understanding. It may be enough to help the user develop a functional understanding or working knowledge of that aspect of the system. For example, a train engineer may understand which controls are responsible for stopping or controlling speed, but they may not need to understand the physics of diesel-electric locomotives. Similarly, a driver with antilock brakes (ABS) is unlikely to know what the sensors and algorithms are used to detect when a wheel is slipping, but they may be able to learn situations and conditions in which the ABS is likely to fail and how to work around its limits. As discussed in Chapter 1, Conant's \citeyearconant1970 good regulator theorem applies: for a user to be a good regulator of a system, they must have an internal model of that system. But that model does not need to be complex or complete to be useful.

2.3.3 Shared Responsibility

Fully autonomous systems are becoming more prevalent, but often have fallback human operations, supervision, monitoring, and control. Even fully autonomous robo-taxis such as Waymo are under the control of the rider (who sets its destination), and have fallback human operators (or tow-truck operators) to assist when they get truly stuck. But complete automation is not always desirable. Humans bear the responsibility for systems, and so they need to provide input and have some level of control over them. They currently share some responsibilities with AI to complete tasks in autonomous systems. Manual and semi-manual controls tend to give the user more freedom but can decrease speed and efficiency (Andersson, 2008). The extent of user control depends entirely on system and AI capabilities, which derives from the complexity of the task/action. Shneiderman (2020) suggested that this shared responsibility is not simply a one-dimensional mixture. Rather, systems can be designed so that human control can range from low to high, and automation control can also range from low to high (see Table 2.3).

In theory, human-required control should decrease as AIs become more complex and capable of operating autonomously. In a fully autonomous system, the user assumes the role of a supervisor; they have no control over operation but oversee processing, perform troubleshooting, and error correction. However, in reality automation may increase the need for humans, but at a different point in the system. Today, most systems do not operate fully autonomously; the user still fulfills this supervisory role while working with the system. Operators of autonomous vehicles put trust in the system to follow a road and avoid accidents, but still monitor the environment and system behavior. Failure to supervise might end in a failed handover and result in an accident.

2.3.4 Boundary Conditions and Competence Envelopes

All systems have their limitations, and it is up to the user to determine the boundary conditions and proper use conditions (competence envelope) for the system. The competence envelope refers to what the technology can do (or can do well) in different circumstances (Hoffman et al., 2013), and the boundary conditions might be considered the dividing point between the situations where the technology performs well and where it fails. Whether this is relayed in a manual or guide, or through first-hand experience or second-hand discussions with other users, understanding the breaking points of a system is necessary to avoid disaster. In an industrial setting, boundary conditions might be set by the manual, or the machine may have built-in limiters. In most situations the user can monitor and fine-tune around these set conditions and optimize the system to find a balance between speed and accuracy of the system.

Having a strong understanding of boundary conditions can help users increase trust in the autonomous system, because it may move them from a position of general distrust (maybe because they have seen failures) to one of encapsulated trust (knowing conditions inside the boundaries of the competence envelope in which they can trust the automation). As a user becomes familiar with boundary conditions of their system, they can understand under which conditions they can trust it, and either anticipate this (to take over or stop the system) or avoid it through other control means. This increased trust was demonstrated in a study of smart home automation, in which participants displayed more trust in semi-autonomous systems over full autonomy (Schomakers et al., 2021). Users seem to prefer to have enough control to avoid violation of boundary conditions. As a supervisor of an autonomous system the operator relinquishes this control to the system and trusts it to keep within boundary conditions.

2.3.5 System Errors and Malfunctions

System malfunctions and mistakes are also characteristics of many automated systems, and this can be distinct from its competence envelope, but can have some of the same consequences. When a problem occurs, the user must identify a course of action to correct it. In this supervisory role, the operator may have liberties with how to troubleshoot and most effectively solve the problem (Ogorelysheva et al., 2023).

Aspects of AI and automation that are important for users are primarily determined by the complexity of the system and how difficult problems are to solve. A user that fails to understand the purpose of the system may misuse it or not recognize errors. Many issues can be spotted if the user or operator can understand the key functions of the system, including how data is collected and processed. Understanding and identifying boundary conditions allows the operator to avoid mistakes. As AI and autonomous systems become more complex and self-sufficient, less human interaction will be necessary and the operator will fall into a complete supervisor role.

2.3.6 Shifting Bottlenecks and Transformed Cognitive Work

One recurrent theme in cognitive systems engineering is that automation and AI cannot replace the human, who has ultimate responsibility. Although they may replace and displace the previous work and can reduce workload, this creates new bottlenecks at different places–sometimes in more critical situations (Sarter et al., 1997). This can occur through clumsy automation (Wiener, 1989), in which the benefits occur for low-workload routine periods, but the cost is that a human must take over during time-critical, peak workload, high-tempo periods–which they may be less well-equipped to deal with because they offloaded routine work.

Much of the impact of AI chatbots can be examined through the lens of how it transforms work. For example, in education, a student can use it to displace difficult and challenging exercises that teach them how to think or operate. This is having huge impacts on many traditional educational exercises and disciplines, including the teaching of computer science, the practice of writing and revising, and on-line coursework in general. AI makes it easier for a computer science student to complete the work, but shifts the learning to a point in the future where they are unfamiliar with the concepts they are supervising the construction of. Similarly, AI systems can help develop detailed plans of action for carrying out both small tasks and large projects. But if the team turns over planning to an agent, the bottleneck may shift from a planning process (which is made easier) to more failures during the plan (since they no longer bring their expertise to the planning process, and may no longer recognize how if the plan is working out). Here, a small savings during initial planning could be transformed into a catastrophic failure during execution.

2.4 Summary

This chapter describes several research communities concerned with designing AI and automation for people. In addition, it outlines some useful concepts that can help a human factors engineer analyze a system that involves human-AI interactions, and identify possible failure points, improvements, and the like.

Acknowledgments

This chapter was originally developed as part of a course project for HF 5430 Human-AI Interaction, Michigan Technological University, Spring 2024.

Author Contributions

KU: Writing and background research for Section 2. BW: Background research and writing for Section 3. STM: Conceptualization; background research, writing sections 1–4; general editing.

Conflicts of Interest

The authors declare no conflict of interest.

AI Usage Statement

Generative AI models were used for additional research, to identify missing concepts, to support better organization, and for editorial tasks such as formatting, evaluating grammar/clarity and citation collation, glossary development, and creation of figures. Content, text, and ideas are otherwise original to the human authors.

How to Cite This Chapter

Ulinski, K., Woolman, B., & Mueller, S. T. (2026). Human-AI interaction research: An overview. In Shane T. Mueller (Ed.), A Handbook of Human-AI and Human-Automation Interaction. https://pages.mtu.edu/~shanem/human_ai/