Abstracts: February 29, 2024

face with these tools. So people really struggle with, you know, writing prompts for systems like   Copilot or ChatGPT. For example, they don’t even know really where to start,   or they don’t know how to convert an idea they have in their head into, like, clear instructions   for these systems. If they’re, sort of, working in a field that maybe they’re  less familiar with, like a new programming  language, and they get an output from these  systems, they’re not really sure if it’s right 

or not. And then, sort of, more broadly, they  don’t really know how to fit these systems  into their workflows. And so we’ve noticed  all these challenges, sort of, arise, and some  of them relate to, sort of, the unique features  of generative AI, and some relate to the 

design of these systems. But basically, we  started to, sort of, look at these challenges,  and try to understand what’s going on—how  can we make sense of them in a more  coherent way and actually build systems  that really augment people and their  capabilities rather than, sort  of, posing these challenges?  Right. So let’s talk a little bit  about the related research that you’re building  on here and what unique insights or  directions your paper adds to the literature. 

explanations that tied all these things  together. And in general, I’d say that   human-computer interaction research, which  is where a lot of these papers are coming out  from, it’s really about building prototypes,  testing them quickly, exploring things in an  open-ended way. And so we thought that  there was an opportunity to step back and to  try to see how we can understand these  patterns from a more theory-driven perspective. 

And so, with that in mind, one perspective  that became clearly relevant to this problem  is that of metacognition, which is this idea  of “thinking about thinking” or how we, sort  of, monitor our cognition or our thinking  and then control our cognition and thinking. 

And so we thought there was really an  opportunity here to take this set of theories and  research findings from psychology and  cognitive science on metacognition and see how  they can apply to understanding these  usability challenges of generative AI systems. 

really quite relevant. And so what we did was we then dove into the metacognition   research from psychology and cognitive science to really understand what are the   latest theories, what are the latest research findings, how could we understand what’s known   about that from that perspective, from that, sort of, fundamental research, and then  go back to the user studies that we saw in 

human-computer interaction and see how  those ideas can apply there. And so we did  this, sort of, in an iterative way until we  realized that we really have something to work  with here. We can really apply a somewhat  coherent framework onto these, sort of,  disparate set of findings not only to understand  these usability challenges but then also  to actually propose directions for new design  and research explorations to build better 

systems that support people’s metacognition. HUIZINGA: So, Lev, given the purpose of your   paper, what are the major takeaways for your readers, and how did   you present them in the paper? TANKELEVITCH: So I think the key,   sort of, fundamental point  is that the perspective of  metacognition is really valuable for understanding  the usability challenges of generative 

AI and potentially designing new systems  that support metacognition. And so one  analogy that we thought was really useful  here is of a manager delegating tasks to a  team. And so a manager has to determine,  you know, what is their goal in their work?  What are the different subgoals that  that goal breaks down into? How can you  communicate those goals clearly to a team,  right? Then how do you assess your team’s 

outputs? And then how do you actually  adjust your strategy accordingly as the team  works in an iterative fashion? And then at  a higher level, you have to really know how  to—actually what to delegate to your team  and how you might want to delegate that.  And so we realized that working with  generative AI really parallels these different  aspects of what a manager does, right. So  when people have to write a prompt initially, 

they really have to have self-awareness of  their task goals. What are you actually trying  to achieve? How does that translate into  different subtasks? And how do you verbalize  that to a system in a way that system  understands? You might then get an output and  you need to iterate on that output. So then  you need to really think about, what is your 

level of confidence in your prompting ability?  So is your prompting the main reason why  the output isn’t maybe as satisfactory as  you want, or is it something to do with the  system? Then you actually might get the  output [you’re] happy with, but you’re not  really sure if you should fully rely on it  because maybe it’s an area that is outside of your  domain of expertise. And so then you  need to maintain an appropriate level of 

confidence, right? Either to verify that  output further or decide not to rely on it, for  example. And then at a, sort of, broader  level, this is about the question of task  delegation. So this requires having self-awareness  of the applicability of generative AI to  your workflows and maintaining an appropriate  level of confidence in completing tasks  manually or relying on generative AI. For  example, whether it’s worth it for you to 

actually learn how to work with generative  AI more effectively. And then finally, it  requires, sort of, metacognitive flexibility  to adapt your workflows as you work with  these tools. So are there some tasks where  the way that you’re working with them is,  4 sort of, slowing you down in specific   ways? So being able to recognize that and then change your strategies as necessary really   requires metacognitive flexibility. So that was, sort of, one key half of our findings. 

And then beyond that we really thought  about how we can use this perspective of  metacognition to design better systems. And  so one, sort of, general direction is really  about supporting people’s metacognition.  So we know from research from cognitive  science and psychology that we can actually  design interventions to improve people’s  metacognition in a lasting and effective  way. And so similarly, we can design systems 

that support people’s metacognition. For  example, systems that support people in  planning their tasks as they actually craft  prompts. We can support people in actually  reflecting on their confidence in their  prompting ability or in assessing the output that  they see. And so this relates a little bit to  AI acting as a coach for you, which is an idea  that the Microsoft Research New York City  team came up with. So this is Jake Hofman, 

David Rothschild, and Dan Goldstein. And  so, in this way, generative AI systems can  really help you reflect as a coach and  understand whether you have the right level of  confidence in assessing output or crafting  prompts and so on. And then similarly, at a  higher level, they can help you manage  your workflows, so helping you reflect on  whether generative AI is really working for  you in certain tasks or whether you can adapt 

your strategy in certain ways. And likewise,  this relates also to explanations about AI, so  how you can actually design systems that  are explainable to users in a way that helps  them achieve their goals? And explainability  can be thought about as a way to actually  reduce the metacognitive demand because  you’re, sort of, explaining things in a way to  people that they don’t have to keep in their  mind and have to think about, and that, sort 

of, improves their confidence. It can help  them improve their confidence or calibrate  their confidence in their  ability to assess outputs.  Talk for a minute about real-world  impact of this research. And by that, I  mean, who does it help most and how? Who’s  your main audience for this right now? 

and actually be more intentional and be more  aligned with their goals. And so this is,  this is an approach where not only can we go  beyond, sort of, automating specific tasks  but actually use these systems to help people  clarify their goals and track with them in a  more effective way. And so knowledge workers  are an obvious, sort of, use case or an  obvious area where this is really relevant  because they work in a complex system where  5 a lot of the work is, sort of, diffused  

and spread across collaborations and artifacts and softwares and different ways of working. And   so a lot of things are, sort of, lost or made difficult by that complexity. And so systems,   um, that are flexible and help people actually reflect on what they want to   achieve can really have a big impact here. HUIZINGA: Mm-hmm. Are you a little bit   upstream of that even now in the sense that this is a “research direction” kind of paper.  

I noticed that as I read it, I felt like this was how researchers can begin to think about what   they’re doing and how that will help downstream from that.  Yes. That’s exactly right. So  this is really about, we hope, unlocking a  new direction of research and design where  we take this perspective of metacognition—  of how we can help people think more clearly  and, sort of, monitor and control their 

own cognition—and design systems to help  them do that. And in the paper, there’s a  whole list of different questions, both  fundamental research questions to understand in  more depth how metacognition plays a role  in human-AI interaction when people work  with generative AI systems but also how we  can then actually design new interventions 

or new systems that actually support people’s  metacognition. And so there’s a lot of  work to do in this, and we hope that, sort of,  inspires a lot of further research, and we’re  certainly planning to do a  lot more follow-up research. 

and act in line with them. And so, in a way,  it’s kind of about building tools for thought,  which was the real vision of the early  pioneers of computing. And so I hope that this,  kind of, goes back to that original idea. HUIZINGA: You mentioned this short list   of open research questions in the field, along with a list of suggested interventions. You’ve,  

sort of, curated that for your readers at the end of the paper. But give our audience a little   overview of that and how those questions inform your own   research agenda coming up next. TANKELEVITCH: Sure. So on the, sort of,   fundamental research side of things, there are a lot of questions around how, for example,   self-confidence that people have plays a role in their interactions with generative AI  systems. So this could be self-confidence in 

their ability to prompt these systems. And  so that is one interesting research question.  What is the role of confidence and calibrating  one’s confidence in prompting? And then  similarly, on the, sort of, output  evaluation side, when you get an output from  generative AI, how do you calibrate your  confidence in assessing that output, right, 

especially if it’s in an area where maybe  you’re less familiar with? And so there’s these  interesting, nuanced questions around  self-confidence that are really interesting, and  we’re actually exploring this in a new study.  This is part of the AI, Cognition, and [the] 

Economy pilot project. So this is a  collaboration that we’re running with Dr. Clara  Colombatto, who’s a researcher in University  of Waterloo and University College  London, and we’re essentially designing  a study where we’re trying to understand  people’s confidence in themselves, in their  planning ability, and in working with AI  systems to do planning together, and how that  influences their reliance on the output of  generative AI systems. [MUSIC PLAYS]