Metacognitive Intelligence in Human-AI Teams

Groups of people make effective decisions in part because they have sophisticated means of exchanging metacognitive information when they work together. Metacognitive information can take the form of confidence appraisals, explanations, or other means of conveying mastery, challenges, and workload. It can also be subtly embedded in one’s prosody or nonverbal cues. This information is used by partners to delegate assignments, produce collaborative assessments, and harness the benefits of the group’s collective expertise.

Teams consisting of humans and bespoke AI agents are playing an increasingly central role in decision-making, including in critical governmental and military settings. Recent research in my laboratory has worked to (1) identify the specific ways in which metacognitive information is embedded in human communication, (2) develop agents that possess analogous metacognitive capacities and sensibilities, and (3) assess the benefits of metacognitively sophisticated agents on human-agent teamwork.

In this talk, I will review several projects that span this agenda and that illustrate the value of thinking explicitly about metacognitive processes in human-AI interaction. In the first project, we examine human team decision-making in general-knowledge and estimation problems and identify the critical components of metacognitive exchange that make that exchange successful or not. In the second project, we develop and assess algorithms for scaling confidence assessments from neural networks, with an eye towards identifying algorithms that are scalable, broadly applicable across a range of architectures, and exhibit the superior calibration that is the hallmark of human confidence ratings in most circumstances. In the third project, we demonstrate that humans that are paired with agents that supply metacognitive confidence assessments in an estimation task outperform humans that are paired with metacognitive naïve agents. In a "bonus" portion of the talk, I will discuss how AI agents can be designed to accommodate human metacognitive illusions and enhance teamwork by, counterintuitively, compromising the accuracy of their advice.

Human-AI team decision-making is only likely to reach its full capacity when team members have aligned means of expressing and coordinating metacognitive states. To pursue this agenda, researchers will need to develop of novel methods and models for assessing metacognitive capacities for team decision-making.