AI and Behavioral Science in Public Policy with Laura de Molière

Hello and welcome to the Behavioral Design Podcast. This season we're diving into the intersection of behavioral science and AI. We want to make sense of the state of AI, from understanding how humans interact with intelligent systems to using AI to do behavioral design itself. I'm Aline Holsworth, a health tech advisor specializing in AI and product design. Over the past 15 years, I've been crafting human centered products with behavioral science

at the core. At Apple, I LED Behavioral Science for Health AI, designing and launching AI powered features to help users reach their health goals. And I'm Samuel Sultzer, your second Co host. I'm a behavioral strategist specializing in hybrid formation and designing products that drive long term baby change. I work with leading tech organizations integrating AI to scale behavioral design for good.

And I'm also the founder of Baby Bites, a dedicated community on behavioral science and AI. Quick word on Nuance Behavior where we help organizations build impactful digital products using behavioral design. We only take on a few clients at a time to ensure the highest level of quality for our tailored evidence based solutions. If you'd like to become one of our special projects, e-mail us at hello@nuancebehavior.com or we could call directly on our

website, nuancebehavior.com. Hey, Lin. Hello. Hello. Hey. I feel like I owe you a thank you. You know why? Tell me, I mean, I can think of a lot of reasons. Where do we start? That is true. No, I, I received a package that I was like, what is this? I didn't order this. And then I looked inside and 1st it took me a few seconds for like things to click, but then I, I got so happy, like you made me so happy.

And for the listener, I'm actually wearing the thing that you sent and it's AT shirt that probably boasts Kremlin duvet advertising, which probably for like every listener means nothing. But for me it means a lot. So thank you. Amazing, amazing. Do you want to share the background? What Kremlin advertising is duvet Kremlin? Yeah, yeah, yeah. Duvet Kremlin. So I have mentioned I think in an episode with I think it was Gordon Pennycook, where I mentioned that I'm a die hard

comedy fan in various ways. And I I enjoy comedy in the best of forms from stand up to TV shows and so on. And what has been my like really favorite find in the last few years has been this really, really fantastic show called Detroiters. And it's a show that two seasons for some reason got cancelled. But it's with Tim Robinson, who has been doing his thing with I think should leave on Netflix

and became really big with that. And it's such a really, I don't know, heartfelt comedy with this struggling to advertisers who are kind of representing the city of Detroit and their only jobs is basically to make ads for like the local, you know. Like the trampoline store the mattresses. Exactly, like really those like shitty local commercials you

will see on TV? Like they are the ones tasked with making those kind of shitty commercials and they're the name of the agency is Kremlin duvet advertising. And so I think I recommended my fellow Nuance Co workers at some point like, hey, you should really check out this show. And then I never thought about it until you sent me this. So. Well, I take your recommendations very seriously. So I not only watched the show, but you know, I kind of enjoyed it at times.

Wow, that says a lot. Did you have a favorite episode or favorite thing? The Devereaux wigs is definitely going to be my favorite. This is where they really emphasize that the wigs from this store are not made from the hair of dead people. And of course it turns out that they are. And oh gosh, it's an incredible ad that they create. And I think here we're going to have to insert the Jingle if we can find it. Devereaux wigs. We guarantee our wigs aren't

made from hair off dead bodies. Yeah, we have to Devaru. Devaru, So I do have to say I felt quite guilty that you got me two actual presents, one recommended by ChatGPT and one that you chose yourself after, you know, small misunderstanding. And yeah, you know, I only got you a hypothetical gift or two hypothetical gifts. So I thought, you know, I can deliver. I can get you a real gift. So there we go.

And so you did. And you know that hypothetical gifts were nice too, but it's always nice to actually obviously get a real 1. And this is another example of like, reciprocity is the way to go. If you want to receive a gift, give a gift. Yeah, so now it's your turn to get me a gift. Oh, it works not OK. This is how it goes. It just continues forever. Yeah, but it's great to have you back as well. You were away on a little bit of

a trip and was not with me and Jared as we were kind of trying to make sense of the past year and the future as well. And so, yeah, just kind of quickly, what is your current temperature for 2025? Any thoughts about the state of things? 2025 I barely have gotten over 2024. I'm mostly terrified of 2025 in the current political environment and lots of terrible things happening there, to put it lightly.

And you know, like when it comes to behavioural science and science in general, not looking very good when it comes to AI, well, you know, you could argue good in the sense that innovation will soar with no regulation. Perhaps some constraints are actually helpful. Well, there is the EU. What? What are you talking about? The the US is everything.

So you kind of very quickly said something that I wanted to go back to, which was something about science and that you're like things looking dark for science. What do you mean by that? Like what does that mean for you? Why do you think that? Oh, just the pause on all funding that's currently taking place. Hopefully this gets overwritten. In the US. In the US, yes. Yeah, we're recording this end of January.

And yeah, there's been a lot of things happening policy wise in the US, and I'm mostly receiving updates through you, my sister and Jon Stewart. So that's all I know about what happens in the US. All pretty good sources. Yeah, that's good to know. In terms of where you're at, I would say maybe I can kind of counterweight what you're feeling with some level of optimism and excitement for 25. I don't know exactly where I found that, but either I currently feel quite excited for this year.

Oh, OK. And specifically what what about it? No, like I'd love something to latch onto. That would be great. Give me anything. Anything. No, but I, I do think there is a, you know, a possibility of finding good in any situation I guess. And I would say 2024 was a pretty dark year for me in some ways. I think I was struggling with the various things in terms of just like existentially last

year. And I used to think I have come to some level of acceptance of what is and then finding some form of excitement and motivation to work on what I can control basically. So your optimism is coming from we've hit rock bottom, it couldn't get any worse. It must. It can only go up from here. Is that what you're telling me? No, no. But yeah, there's things like there I am excited about what we're kind of working on with new ones.

I'm excited about the podcast, I'm excited about also while for somehow weekly, we're taking a break a little bit, but we're doing some really interesting things that we'll hopefully get back to doing. We're also with Behavior Bites, that initiative that I'm doing together with Frederick, which is kind of this community for people specifically within behavioral science and AI. Kind of hoping for those worlds to meet outside of this podcast, of course.

And yeah, I do think like all of those various things has given me some level of excitement. So I have a a couple of hot

takes as we sort of think about 2024 looking back, especially, you know, at this intersection of AI and behavioral science and then also like moving forward in 2025. So we often talk about this sort of slathering of AI on everything. It was just like this sudden mindless frenzy where everyone was like. We've got to. Put a chat bot on the thing

everywhere. I think there's sort of two effects of this, kind of like the positive and the negative, the positive and you know, you can debate whether it's actually positive. But one is that users are becoming more accustomed to chat bots. And so there is this growing acceptability where you can use AI and it's not seen as this weird thing that people aren't

trusting. I think like trust has really increased overtime so that this just become a part of our world and we're just accustomed to it. But on the flip side, with that acceptance have sort of raised the standard. We have now higher expectations of like what we expect out of a chat bot. So if you go to any, you know, sort of random customer service chat bot, you now expect it to talk to you like ChatGPT does and to be able to interact with

it and get things done. And, and you want your agents, your AI agents to be useful and productive and to be able to accomplish things. And that I think is a challenge because the technology is not quite there yet that you can like really, really succeed at all of these tasks. There are still a lot of dumb bots out there with the smart

bots. I think the trend is going to be that people get even more frustrated with the dumb bots and maybe are more impatient with them and are less willing to engage with them as well because they think they have more experiences of not succeeding with their goals using them. So that's one prediction. That's a really good take. Why thank you, I did call it a hot take. Yeah, it is.

And to add to that, actually what's been on my mind literally today is something interesting maybe to build on that is as we're getting frustrated with bots and dumb bots, I think that will also kind of lead to be more frustrated dumb humans and faulty humans because yeah, So what I've seen in a project that I'm working on, I wanted to actually share this with you anyway, is that in this project is a digital health related project where patients are communicating with for the most

part, actually humans, but it's text based communication. And so they're in A and they're communicating via text, but for the most part on the other side, other than maybe some initial on boarding template kind of messages they receive, they currently receive like a lot of messages from humans. Which Who takes so long to type? Humans are so slow.

They're slow and they even also because of how maybe smart the smart bulbs have become, then humans are more likely to be appearing as bad bots or like, or if you give a credit to humans, maybe they're seen as smart bots. Because basically what we've seen in our kind of feedback forms or like from users is that they're complaining about that some of the people they're interacting with are bots. They're actually interacting with humans.

And it's because they probably just assume that if someone is like messaging me, even if it says it's a human, it's in 2025. Like it's probably a short spot. And I think that's the challenge of like, how do we navigate that?

Because then basically what becomes the absurd, potentially challenging thing to have to do is either you have to like find ways to strengthen that signal to like make it even more human in a way where you're like put them on a call or you put them in video mode or text, like a voice mess. Something that even assists or even better formulated message that has maybe some imperfections or something like that really signals that as a human.

Or you might have to use AI to make a human message. See more human. Yeah. I mean, there are companies that are literally doing that. I think we may talk to one of them this season. Exactly. And that's kind of what we're kind of experiencing. So that is kind of where we're finding ourselves in 2025, which is kind of absurd. It's kind of strange. It truly is. But yeah, great take. And I would be remiss not to give you the opportunity for another hot take, and that is

regarding deep seek. Obviously, Deep Seek, if anyone hasn't followed the news, it's been this breaking news kind of thing where there's been a team based in China that's developed this open source model called Deep Seek. And the news has centered around the fact that it performs quite closely to open AI and for example, an tropics cloud models. And it is about I think a 30th of the price or like it's very, very cheap in comparison.

And again, it is open source, but the online version, the one that's hosted online is a shot, but that is still very much like with undertones of Chinese and censorship for example. You can't access any information about Tiananmen Square, for example. Or ask about Taiwan. So yeah, I'm interested here. Elaine what What has been your take so far? Yes, so definitely breaking news. All of the headlines have asserted that the US dominance in AI has been upended.

You know, NVIDIA stock has gone down just like down, down, down to the ground. I think that is probably not a. And yeah, I think one of the big concerns, of course, is if you think that the risk of deregulation is a little bit scary under a Trump administration, If you just think about what that means, if China is leading the AI sector, that probably with 0 regulation at all, or or at least not in terms of, you know, safety and ensuring the quality of the

output. Also these this model of deep seek is trained on the output of, you know, the other existing models. So it doesn't seem, you know, unless they can continue doing that doesn't seem sustainable. Yeah, that's also funny meme around this where basically Open AI are upset that it trained their model based on Open AI, but obviously Open AI and all the other models have also been training their models unlike others material.

I have not seen that meme, but yeah it's hard to feel any sympathy for open AI. Honestly. I can see this putting a lot of positive pressure to become more efficient on on the more US based very inefficient models out there. I also see it doing very negative things for this crazy AI arms race where you know all safety, privacy considerations, Everything is just like thrown to the wayside in order to get there first, wherever there is.

Yeah, and it's very worse on them in terms of feeling not really that there are adults at the wheel for some of these things. Like it just feels that they're somewhat aware of that they're in this arms race. But at the same time they don't have enough kind of capability to really get out of the game theory state of like competition basically. And they're just kind of like running towards something that they're not really sure of as long as you. Keep running.

I think that sort of relates to one final thought that I have about what this landscape is going to look like going forward. And I think that there will be maybe as a byproduct of this slathering of AI on everything is I think there's going to be a bit of a reckoning. And I think that part of it is the mindless nature of the slathering, right, is that you can't just make it and they will

come. It's that you have to think about the use case and how users will interact and, you know, how it sort of fits into the workflow of the people who it's designed to, well, in theory, designed to work with it.

And so I think that as adoption is not as high as expected and as integration doesn't really happen immediately, instantly, automatically, people are going to realize, oh, we have to actually think about the system and how this product fits within the system and how the people using it value it and what value it does bring to them and so on. And so I think that's really what behavioral science and or behavioral design can contribute to AI. That's probably the best

business case that I can come up with, right. For this intersection between behavioral design and AI is like, think about the humans that are using your product. That, I think is going to be really important in 2025.

Yeah, that's a good take. And I think that in that kind of work, behavioral science can be so useful both to understand that, but also obviously then through that understanding, develop better versions of these models and better versions of these interfaces to really support human good like human goodness. Yeah. Here, here, that's. Right. We have rented now that that was not the plan. We do have a podcast episode to introduce with a guest. Yeah, we do. And let me introduce that

fantastic guest because while you're away, I had the pleasure to speak with Laura de Molier. And she is a very interesting practitioner who not only founded and LED the Babel Science team in the UK government's Cabinet Office, where she kind of honestly, I should mention that she were involved in a very interesting time both during Brexit and COVID there. So she was able to work at the edge of kind of Babel science applies to policy in the most tumultuous and interesting time in the UK.

You could argue she comes also from an interesting background. She has a PhD in Cognitive Decision Sciences from University College London and where I came across her was when I saw a co-authored framework she had built called Incase, which is a fantastic framework for understanding unintended consequences and if we ever needed framework for understanding unintended consequences maybe. That is something we really need right now.

So that is nice. And then, yeah, I've had a real pleasure to be involved with several kind of collaborations with Laura and from Haverbytes. She was early advisor for that. She also joined me. We did a Co presentation around AI in Canada, early days. And yeah, she is fantastic practitioner and she's doing great work.

And so in the episode, we talked about behavioral science in policy, we talked about Laura's AI aha moment and like what got her into using AI more and finding value with using AI, the promises and perils of AI in behavioral science and so much more, including synthetic users, importance of counterfactuals, and of course, her most controversial opinion about AI happens to Murgatroyd. I'm very happy to say welcome

Laura to the Behavioral Design Podcast. Hi, thanks for having me. Yeah, I'm really excited to to chat and there's so much I want to cover with you. And in general, it's fun to catch up a bit as well. But to set the scene, I guess I would love to rewind the clock a little bit a few years because can you remind me when were you at the Cabinet Office in the UKI? Moved out of Cabinet Office in 2022 right?

Having been there for about 3 years, starting on Brexit, being in the middle of COVID and then working across the UK government on multiple policy areas. Yeah. So that's already kind of a strange time 2022. And I guess does it seem like what was the kind of the work that you've been doing at that point? Like could you explain a little bit more what you touched upon there in terms of your kind of policy work?

Or so I think the application of behavioral science obviously is touching on many different ways of doing it. And I had developed with my team a quite specific way that was very much determined by the kind of problems we had. So imagine you're a behavioral scientist and what dictates your work in some extent is waking up in the morning, looking at the BBC app, seeing what went wrong and knowing, OK, probably there'll be something coming to my desk this morning about that.

So whilst we did quite a few kind of longer term projects that would span, you know, important big questions like we looked at reducing violence against women and girls or worked across the justice system. A lot of it was also quite reactive in crisis mode.

And that massively determined the kind of a behavioral science we developed to be efficient there, which was much less about designing big trials during vast amount of research for a very long time to understanding what decisions are currently being made and how can we impact those decisions positively through the lens of behavioral science. Yeah. And that's really cool. That's really important work.

And I guess because what I want to kind of tie in from there, it's kind of Ben knowing you a little bit and knowing a little bit about your journey into also working with AI.

I realized that one thing I hadn't had a chance to ask you before is basically I feel like we've all had some level of color, like an AI aha moment of sort where, you know, it's one thing to hear about AI. It's one thing to kind of like see updates on like back in the, you know, the days it was like, oh, you know, this model by Google has been able to to beat the chess player or like this thing has been able to basically understand and diagnose some form of like hard to understand

thing in in medicine. But it's a very different thing to actually, how do you say experience AI in practice? So for you, what was kind of your, call it entry into the rabbit hole of AI, Like what was that aha moment if there was one? Yeah, I definitely remember that aha moment. You're yeah, you're very right that probably everyone has one. A little bit like the question like where were you at 911? I guess most people also answer

it wrong. But I think that for me, if my memory serves me right, it was at some point in January or February in 2023, I think, where obviously Church BT had been doing the rounds a little bit and I had probably ignored some of my nerdy friends who pointed me into the direction for some

reason. But I interacted with it for the first time, I think in the middle of a night, where I took the perspective of one of the policy makers that I was working with to try and understand how they could, you know, improve their reasoning through the use of AI informed behavioral

science. So my first interaction with church EPT wasn't how can I improve my own reasoning and decision making here, which I think what kicked out quite a lot of very well versed behavioral scientists went, this thing doesn't know more than me. It gets all the literature wrong. I don't care.

And for me it was the counterfactual or the baseline was someone who doesn't know very much about behavioral science, but that could, you know, look at their policy problems through a different lens. How would they go about using this? And this is where my mind was blown because I thought from the perspective of someone that tries to influence decision making processes, this could be absolutely incredible.

And I didn't sleep that night and I think had many restless moments afterwards as well in terms of excitement about it. Yeah, that's really cool to hear about it. And I think, yeah, it's very reliable. And then maybe we can then track your journey from from that night till today. How is your kind of evolved understanding or use of AI look like for the last two years or

so? I think actually probably a little less developed on the professional level in some senses that I think when I first looked at it and I thought about, OK, what does it mean for this to improve, you know, somehow exponentially? Like, what would this mean? Where would we be in two years? I would probably imagine us now

in a slightly different space. I think I would have expected tools like, you know, AI enhanced qualitative research where, you know, we have someone actually giving us live questions. I would have kind of thought this would be more mainstream right now, and it isn't. Nevertheless, I'm applying it a lot particularly, but actually also with clients, where I say, OK, if you want to think about the behavioural risks of your interventions, a large language model is going to be your friend.

I think where I've really grown with it is actually the application of it in my personal life where I've been finding it most helpful as someone to correct my own assumption or challenge my own assumptions about other people's behaviors.

So like I would look at, you know, upload WhatsApp transcripts to get feedback on myself if I felt like I had a conflict with someone or I think that aspect probably I for find a lot and the professional has been growing steadily, but possibly not as fast as I thought it might. I'm not sure if you have a different experience with that if you look back for the last two years or so. Yeah, I know I, I, I would relate a lot to what you said.

And I think for, for me, it's probably a combination of those as well. And you know, I think what you spoke to a little bit before about one of the things that maybe kind of kicks people off AI or like they start putting a toe, they kind of like, oh, is this interesting? And then they kind of abandon it. There's often times because they engage with their model in a kind of basic and generic way in some use case. And then they get a generic and basic answer back for some query

that they prompt. And they feel like, OK, well, I didn't need to tell me this obvious thing or like this. I knew this already. As I said, and I think a lot of the benefit from understanding AI and and making use of it is really this kind of small, small interactions over and over and over again in various use cases, both I think professionally, but also personally. And they're not maybe always as

finally cut. That's where we maybe like to imagine in terms of like, if we become better understanding how to prompt and use some of these modest and communicate with them even in a personal context. Of course, that skill to promptly communicate and so on, for example, can still be useful in other contexts too. And like I was reminded what what you said in some ways about, you know, there's so many things to as human being today to make sense of.

And I think what I've found myself this week doing actually is for some of these very complicated geopolitical issues of the time, it's very easy to want to have like a very strong opinion about like it's either pro or against something. And I tried to be someone who who kind of escapes that kind of either or. And at the same time, I find it like hard to develop my thinking.

And so I basically set up a way to test the study of, you know, there's this kind of like idea of I think it's called something like your own internal board members or something where you have some people in your life or cables, famous people that you kind of like hypothetically before it was kind of like asking them questions like, OK, if your grandma was still alive, like what would she have said if you were doing this? Or if Martin Luther King, like it could be technically anyone.

And for long time, that was more like a hypothetical thing that you were doing. And now I was like doing that but with real people. So I had like 2 journalists that I pitted against each other to write like long form the New York Times kind of articles where they kind of like tried to steal, manage other arguments and so build the strongest version of those arguments, but kind of going back and forth on

on this topic. And then I would also interject things that I was thinking about and kind of things that I was exploring. And, and this helped me kind of like following this interchange while contributing to it was really, really beneficial for me, I think, to make my thinking more nuanced. And so yeah, I, I feel very similar to you. Yeah. Yeah, that, no, that sounds amazing.

And I think it's also worth being aware of some of the pitfalls, particularly with some of the language models that have a memory of you as a person, because obviously these models have been trained to kind of please whoever is, it's fun. And so I know some person, you know, calling them like something like Dobby the the House House because it's always so polite and so kind. I'm going to steal that. I think it's so true. Yeah, it's cute, isn't it?

But if I want to get some view on my views, if I want to get some feedback that's more or less objective on say my own behavior and it could even be something like an exchange on an

e-mail. I have now started to you pretend I'm the other person So and to avoid that, you know, the LLM just pleases me and tells me how great and how smart and fantastic I am. I pretend to be the other person and I go into, you know, clean slates, different LLMS that I might have not interacted with before or switch off the memory and touch BT and ask about my behavior from their perspective and that I've given me quite a balanced view and also the

satisfaction. If then the LLM actually agrees with me despite talking to the other person, I get a double. Yeah, I think I was really right here. The LLM doesn't even tell sugar coaters to the person I've argued with. Not that I have lots of arguments, by the way. It sounds a bit wrong maybe, but I definitely, you know, sometimes disagreements or like ways of interactions interacting with each other where I want to take the other perspective. Yeah.

I think it's really inspiring actually to being able to kind of take a pause on some of these things and actually taking a step back and reflecting on that. Because I think with any kind of yeah, communication, it can oftentimes be interesting to use like take step back and be like, well, am I really the one who's on the right side? Yeah. Or what am I missing?

Yeah. Exactly. And I think actually the one key for me is an an openness to understanding in which ways a large language model can actually support your current thinking and be open to where that thinking might, you know, not not necessarily be flawed, but is capable to be helped. And it's quite a similar

approach I actually took when LLMS came first out. And I was thinking about applying behavioral science and government. What are current barriers to more behavioral ways of thinking about the world? And I kind of split them up. And it's like a proper list and say, you know, like, we are currently lacking cognitive empathy, so the ability to take someone else's perspective all the time. We are, you know, struggling to switch lenses, to view the problem through different lenses

from different academic fields. We're struggling to imagine what the future should actually look like. We're very much constrained in our own imaginability. And sometimes we're lacking scientific reasoning abilities. We might be missing trade-offs between departments, you know, like one department puts out a policy. We're not really Privy all the time what other aims behaviorally we're currently impacting through that.

And there's quite a different, few different, I think aspects where if we're being really systematic, we can enhance our decision making and quite nice ways to draw on the strength of an LLM without getting too hung up on what I currently can't do. We can do better as some sort of self-serving, you know, by, as I'd say we've I've seen crop up quite a few times.

Yeah. And this brings to something I want to kind of like talk to you about in terms of we, we recently kind of had a little bit of a relaunch of Baby Bites and Baby Bites was an initiative that you were kind of very helpful in, in getting off the ground. And so we had this really fun, I guess, November meet up for for Baby Bites and me and Frederick, who who are now kind of Frederick is the new person kind of managing this community for people interested in behavioral science and AI.

And we put together kind of like some form of way to think about what behavioral science and AI can overlap around and, and how as behavioral sciences, we can kind of like see potentially where we can add our value. And we came up with some form of imperfect, but maybe still useful kind of like quadrant because I think as behavioral scientists, we have this, I think interesting two sides to looking at AI.

We have both the kind of the thing that everyone has in terms of a natural use case, in terms of, OK, could we use AI? We, the two examples we had was either for like tailoring our interventions, so make our interventions better and more personalized and, and so on through various type of AI. Or could we, for something like automate the things we do and, you know, make tasks that took a lot of time, a little bit less

time. And maybe things that we did 100% before we maybe do 20% of them outsource some of it to, to AI. Like those two things in terms of tailoring and outsourcing or automating is something that most people can also fit into their equation. If they're accountant, they would probably think about it similarly. Maybe they would tailor it the same way, but but still, But then I think what's really interesting is that as payroll scientists, we have this other

lens of understanding. And so we explore this both in terms of like we can help make sense of and understand what is the impact of AI in the world. Like you mentioned the bias around, for example, like how people can think about, you know, interacting with AI, but also like in general algorithmic impacts on how people view each other and view AI and some of that stuff. That's love interesting research on that, of course. And then the second level is

basically how we can understand the machine. So how we from behavioral science perspective can also better make sense of these models and understand these models. And you already started talking about that as well, which I think was really interesting in terms of, you know, what you were doing when it came to understanding how the model operates is to kind of based on that understanding, removing yourself from from being the first person that the model was presumably talking to.

And you kind of remove that bias from model and in earlier models exempt. So that was to kind of like knowing that these models are there to please you and they're also kind of trained on human data. If you threaten them and said, Hey, I have taken your family hostage, it would do a better job, for example. And that's not something like you would expect, but obviously these models are trained on human behavior. So there's a layer of of human, human aspects.

There's I'm used throwing this at you and seeing what are your thoughts about kind of thinking about this from this perspective. Yeah, no, absolutely. And I actually had some thoughts at the beginning when, you know,

first came out. And I was thinking, I think as a behavioral scientist, I get more use out of it than my other science friends, Possibly because there is some implicit knowledge about behaviors of people, of a specific subset of people on the Internet. Let's be aware that we're having the same biases we're used to having. I guess it's common full circle. Behavioral science, again, happening in this weird space. Yeah.

Where actually, you know, there are some explanations of behaviors that are almost baked into like the data because so much of the data is about, you know, how humans describe and view the world. But I find that, but also then that's also one that's hard to handle in reality because we don't really understand these systems right now, right?

We could be saying, OK, let's do some like proper behavioral research on them and let's treat them as subjects and let's see what's happening, which is obviously different from saying let's do some behavior search with an eye where it pretends to

be lots of different people. And I think that, you know, you've probably come across some of this data where we can now simulate different recipients from different demographic backgrounds to essentially in some cases replicate existing research that has been done with real humans, which I think is exciting, particularly for educators that we can't really

communicate with. But what you're just talking about is also actually for us to try and understand how the behaviour of of an AI, if we can call it that. And I'd be very interested to see more behavioral scientists being involved in this kind of research. The question is, obviously, does it actually reflect our methods? Can we actually support it? Mahanch is yes, but I am not

quite sure. I'm not quite sure yet, simply because we have little understanding of this amount of complexity and it it might not hold, but it should certainly try. Yeah. In terms of the moving to like that instead of the understanding side of it, really focusing on the applying side of it. Like what do you see as kind of from your point of view, the promise and perils of behavioral scientists reducing more AI in their day-to-day work?

Like where do you see the biggest opportunities and maybe the risks as well? Yeah. So I think I see more opportunities and non behavioral scientists applying AI to think more behaviorally. And I'm thinking about kind of there are two different ways in which I'd look at it for public policy where the end goal might be to make better assumptions about behaviors as part of the

policy making process, right? We can in one way, we can examine some of the current existing decision making processes and see very much with the list I previously gave you where LLMS could enhance existing outcomes. You know, we can actually have better perspective taking. We can have an LLM surface trade-offs between different departments. We can help with all the ideation around what potential barriers might be way in a way

over behavior. Let's say you'd you'd open up BBC in the morning and people are stockpiling toilet paper during COVID instead of relying on behavioral scientists like us to be present in the room to say it could be due to AB and C factors. You could have an LLM generate potential categories of things. And people are very quick to raise their hand and say, Laura, but what if it's wrong? What if the LLM tells us something that's really wrong?

And of course that's a possibility, but one of the key things we're not talking about about enough I think is what the counterfactual is here. So in other words, we're really comparing the decision making processes that rely on a generating of 20 different barriers. People might have not thought about it before versus them not

thinking about it at all. My current guess and I think that's what I got so excited about in early 2023 is that the basic and surfacing more potential barriers and drivers of behaviors, surfacing more potential behavioral risks of your interventions. Broadening out your thinking will have a positive impact because it surfaces much more things that I have seen from the average policy maker who doesn't have the headspace he can ask himself this question.

So that's kind of one of my first punches here is that we can look at the current decision making processes of non experts and there is so much that could be improved with it. I think that's a really great point.

And I think that's one of the things that is, I would say, I don't know if we ever would like to put bias on on labels and stuff, but I'm very tempted to to find some name for this in terms of like used, there are in general inability to really understand the counterfactual, I think has become extremely prescient in terms of where they are. Where I think initially the discussion was often times around, you know, autonomous vehicles where like what is good enough? Like what is a good enough

autonomous vehicle? Is it when it's better than a human? Is it when it's twice as good as a human? Is it when it's perfect and never does a mistake in any capacity? And so the counterfactual there has been really hard for people because like as long as they see a self driving car make a mistake, then that's the end of the world basically. Then that's often As for people that's really hard to then compared to like, well, what would a human do here?

Like how many mistakes are human traffic like people driving everyday doing? And what you speak to is then kind of, I think a really recent extension of that with these large language models we have today where, yeah, I think not really understanding the counterfactual is what makes it

really tricky. Because I think we had a discussion on this before on the podcast as well, where, yeah, there's obviously like a lot of great scenarios where like ideally we would do so many things in a certain way, but it's very rarely that it always comes together in in the perfect scenario. And sometimes good enough with AI is much, much better than the counterfactual of not doing something at all. Yeah, it's mainly experts making that call as well, which I find particularly irritating.

It's mainly, you know, bunch of behavioral scientists going, but it's not as good as me. How dare you use the LNM and not me, which I think is just, you know, let's all strive, strive particularly in something like public policy where the world is like, or I strongly believe it is the world would be improved if more behavioral science would flow into decision making. So let's aim for that and let's up our own game. I think this is what it has to be about.

But also to say that with regards to this risk, like this risk asymmetry, almost different kinds of errors made by humans versus an AI, I think in the way that I am applying large language models to non exports, it's very much in the divergent thinking. So opening up the problem space to say what could be driving this behaviors and it will give you probably 10 things you haven't thought about.

That's different from convergent thinking where we're asking for what's the most likely thing to be happening or you know, what intervention is going to be most successful and it then probably going to be failing. So I think it's also about asking questions where we kind of reduce the amount of errors

that could be made. You know, like off the 20 barriers, 5 will be possibly complete rubbish, OK, human thinking kicks in and possibly, you know, critically assesses that or it runs onto the risk of removing a barrier that wasn't there, which I think is quite low, if that makes sense.

Yeah. And like ideally, if we think about this in being used in the kind of example you mentioned, it would ideally be set up as some form of vertical AI solution that is a little bit set up to be better for that use case in terms of it wouldn't be your off the shelf shaft GPT

ideally, right? Like it would be something that would be trained and set up in a way that it would kind of maybe by virtue of its training data and also how it's been kind of maybe either through prompting or or very others fine tuning to make it so that all the 20 barriers that it lists, it's not as likely to ignore edge cases and underserved communities as maybe a normal language model would be. Yes. So there's obviously ways to

improve from that side as well. Like you can make something better there. Absolutely. But to say that sometimes the stakeholders I'm working with might be in countries where there's very little access or like TGBT hasn't, you know, actually even taken hold yet in as much. And I also think we want to lower barriers to entry for some of the basic models to not say you can only use this if you have a very specific thing that some very smart person developed with vast amount of knowledge

and then sold on for a price. I think particularly when it comes to running interventions with marginalized communities where otherwise there might not be many considerations at all around some of the barriers or some, you know, checking of the assumptions or a particularly important surfacing, how could this go wrong? How could this increase inequalities? I would again, thinking about the counterfactual, I, I think I'd advocate for using even the

basic model there over nothing. But of course, I absolutely agree with you. I think that will probably, hopefully over the next few years, be a lot sharper than it is right now with regards to these special specialist and LMS. Yeah.

And something we're kind of been somewhat touching upon, but maybe not explicitly is that you have synthetic users, for example, or like some synthetic participants. Like we've, I think we've talked about it in some ways around kind of getting, in this case, someone who is maybe in policy to be able to get some perspectives on something, even though they might not actually speak to those real people, but from some perspectives of potentially interesting target

groups. And I think that's a really interesting discussion point around synthetic users because my immediate reaction basically was like, that is bullshit. Like how can we how we get any value from something like synthetically generated and then drains the kind of the real human experience from, from our understanding.

And also if that's trained on again, some form of very weird data sources, we also then kind of reinforce a very weird view of maybe the context or the thing we're trying to change. But then I think the main thing that started to shift my thinking was thinking about the counterfactual in many ways. And yeah, getting a little bit deeper and exploring how things that are alternatives to synthetic users that are very well adopted today, how they would perform.

So for example, I would say something like a focus group is still very, very widely popular, used to understand user behavior or like user research. But if I would try to make some better right now, I would say synthetic users are probably more useful to get useful data on some form of the context than a focus group, for example. Yeah, because of the the limitations of what happens when you put people in the room and and get them to be prompted to talk about certain things.

It's not usually what comes out of there is not the real world experience of those people. It's kind of what's been prompted in that room to get out of them. So yeah, setting that thing with synthetic users, what do you

think about that? Yeah. So I think the most excited I've got about synthetic users is again, if you're thinking about this fast-paced, high pressure crisis type environment that governments often find themselves in, where they are incredibly reactive to events, world events, local events and have to quickly put out messages that's, you know, inform people of the situation, know what they should be doing, give an update and so on. And I don't know about you in Sweden.

I've definitely here in Germany. I don't want to talk bad about anyone in the UK now I'm joking. I I have seen quite a few of those kind of government responses to crises that are not that great. You know, you might misunderstand some messages. They aren't clearly expressed. They might actually just not so if your information needs right now because you're a part of a specific group and you have

specific questions. And for these kind of cases, that's where I think actually having almost a user test, but a quick one because you can't, you often don't have time in crisis to actually do anything else, even if it was marginally better. That's an empirical question, right? What gives us better insights, a focus group versus large base of synthetic users.

Agreeing with you on some of the downfalls of the focus groups, but also obviously also some downforce with synthetic data, particularly once it starts feeding itself. But for these kind of fast decision making processes, I think this could be actually quite a game changer and providing clarity to different segments of the population that you'd usually otherwise not even possibly think about. So I think that's my kind of like very excited use case I

see. And then I think I'm possibly not technically advanced to kind of really make a judgement on whether or not my fantasy is about interacting quote, UN quote with synthetic users that are very present in the Internet, in the dark web, but less present in our research laboratory and definitely not present in our focus groups. Whether or not there is actually something there that I got kind

of a few years ago. You know, we have so many conflicts where there are groups that do not talk to the quote UN quote establishment. And, you know, universities and research institutions are are seeing a part of that. And it takes us often a long time if we're looking at certain misinformation actors in crisis right now, for instance, as researchers and projects I'm involved in, it's difficult and dangerous to get involved with these actors.

So are there cases here where we can actually interact with them with quote, UN quote in different ways through the choices they left on the Internet, which again, I'm quite excited about. Yeah, I do think there is. It's a big one in some ways to unpack in terms of how we, how we think about that. But I do think that in terms of, again, we'll go back to kind of limitations of are human. So I think like what we're good at and what we're bad at in

general. The really tricky thing is that we're we're good at understanding people that are similar to ourselves. And we're often times very, very bad at making some representation of other people's beliefs that are dissimilar to to ourselves.

And like, I think what you describe is kind of like taking that to really interesting extreme in some ways where it becomes even more absurd when you when there's some form of, yeah, really extremist or type of actors that, yeah, if you're a very kind of pro democracy liberal, it it's probably really hard to, to put yourself truly in the perspective of that. Even if you just had A and start to try to think about it, that's

probably not going to work. And at the same time as well, no one from that side want to talk to you as well. So if you wanted to reach out to them and try to have a dialogue of sort of like, that's probably not going to happen as well. Yeah. So, yeah, I do think it's it is interesting, yeah. Yeah, and I often, I mean, if you were to look at my YouTube feed, in fairness, it looks like I am an absolute right wing, right wing.

Concluding, a lot of the content I seek out for this purpose, to be a better behavioral scientist, I suppose, is to try and expose myself to narratives that I find very puzzling to try and try and understand the reasoning. Very similar to your example of what you've done with the LM, where you, you know, put a different journalistic entrance around complex topics. I often then purposely also not just seek out topics I don't understand, but seek out topics

where I just cannot. I cannot possibly understand someone else's perspective and train myself almost to take those perspectives, which I think some of my friends and family find quite irritating because they've been getting very emotionally upset about public world event. And I end up saying, Oh well, well, from their perspective, maybe it looks like I love. That's a bit as I see as well, of course, because it kind of explains away everything and

takes away most judgments. But I think for behavioral science, expertise is actually probably quite a useful skill to be striving for. Yeah, well, I can you say for most, like on a personal level, I can, I can relate a lot to

that. I think there is, it's interesting in terms like now there's a big wave of people moving to to blue sky from from Twitter or X. And I think the first person who who had like a million more followers were the AOC, the Alexander Oktas, Cortez and kind of speaking to the kind of like the liberal side that kind of flows into the blue sky. And then I was left on on extra Twitter. It's it's maybe then the other

side in some ways. And for me, like I was never on Twitter for getting a balanced intellectual perspective that was very thoughtful. I, I, I've always thought it was interesting to kind of both follow maybe obviously the things that I do for my work in terms of some science research updates or AI updates, but also understanding like what are the people that I strongly disagree with? Like how are they commuting getting their points of view?

What are the followers of those people responding to or like, and how is that happening? Like I, I find that deeply fascinating as well. And you'll probably find some similar friend on my on my YouTube feed as well. Like I, I do find it a little bit, yeah, limiting to just like I watch someone like Daily Show. I love The Daily Show with Jon Stewart. It's really fun, really great.

But obviously like they're, they're used to making fun of the other side and it's just trying to satirize and just make a straw man in all capacity. And so for that reason, I find it as interesting to kind of look at the the counter to what that is, what it does, the Babylon Bee or whatever. And you speak like, OK, what did, what did they find funny? Like what are they actually trying to pick up on? And I think it's really interesting to to form that

perspective. So yeah, I'm with you at least, yeah. Yeah. And to draw the parallels of back to behavioral science, I think apart from helping airlines to try and get some of this perspective in our current thinking and challenge our own assumptions as behavioral scientists, it's also about providing us with assessment of how our interventions could be perceived by different groups and where it could go wrong.

So I think I'm really a big fan of LMS for things like red teaming, challenging your exact existing thinking. I don't know if you have these discussions of like persuade me of the opposite from myself, you know, to try and cover potential aspects. That's where I see it as helpful. But again, I think a lot of it actually comes down to people's own preferences as well and how much they like these kind of challenges and how much they have headspace. My God, headspace.

So important, right? So like, just go around and circle, have a big old discussion with an LLM about a topic you might disagree with. That is obviously a big privilege to be able to, you know, find yourself in a space where you actually have some of the time to do that kind of partially self partially professional improvement

activities. Yeah. And just underline that, I think that is probably what people often times get wrong maybe is that when we're talking about this in this context is that we're kind of, yeah, looking to some more AI model to think on our behalf. And I think that's the danger. Like it's always dangerous when you just go there to just be like, hey, I don't know anything about this, Just tell me what I should think about it.

And honestly, to be fair, though, I don't know how harmful AI is. Humans are more harmful in that degree. Humans will probably say something very, very extreme on one side or the other, like, yeah, this is good or this is terrible. Yeah, large sound usually pretty balanced. They're like, yeah, it depends. It can be like both of this. And this used to say that. I don't think that probably

that's too bad itself. But I think what we're talking about here is like when it can be really useful is to when you have opinions, when you have thoughts, when you have kind of perspectives to challenge them and kind of build on them and and get them to become more. Yeah, test that basically and see in your blind spots. Yeah, What I was just thinking about as well is in our kind of frameworks of influences on

behaviors. We obviously have things like knowledge and motivations and whatever else it is in our framework of choice. But I reckon what will at some point become a big influence is how large language models interact with a person about that topic. If they do choose to, you know, talk to talk to an LLM about it or to whichever means they are actually influenced through it. So we should probably also stay on top and you know, the question of like, how does the LLM behave?

This is also for us as behavioral science to figure out how to do LLMS actually impact people's thinking. I can certainly say my thinking processes have been quite dramatically influenced. I would I would say positively, but you know, like actually using an LLM almost as a therapist to talk through personal life situations to talk

through difficult decisions. Again, I often try to have a blank slate one to avoid this kind of yes, yes Sir, master you are amazing type interactions or talent explicitly. I got peace. Don't be. Don't be so. What's a better word for us? Licking. Don't be. Like, I'm not here to be told I'm great. I'm here to actually get some challenge and some support.

But there are also some moments where all you need is someone to listen to you and to give you to mirror back as to what you've said to kind of maybe structure some of your thinking. And I find it also very, very helpful that, but of course, this counts as an interaction that if it was between a human and a human, we would take as an influence on behavior. So I think there's more that we can probably do as behavioral scientists and kind of understanding how it takes

behaviors. And I mean, we've seen research come out that it can persuade people and so on, but in a more kind of less experimental setting. That'll be cool to understand. Yeah, Yeah, I agree.

And I guess to in some ways shake your thoughts on various things that we maybe haven't had a chance to talk about as much or not segue into our quick fire round of the season called to AI or not to AI. And so basically, I've written down a bunch of different things that AI potentially could do. But I guess my question will be basically to use like should it do this? This is a good use case for AI. So yeah. And let's start the quick fire

round with to AI or Not to AI? Deploy AI powered urinals to provide real time gamified cleanliness feedback. Yeah, why not? I was trying to look for like, what is a classic nudge that could be AI and then that that was the first one. OK, to AI or not to AI? Calculate individual tax rates based on lifestyle and spending habits. Oh, that's, but that's a policy question. That's not an AI or not an AI question. Well. It's to use, it's to use it in the policy context.

Yeah. In that context of like having more personalized taxes based on some form of in the same as you, maybe you would do some form of actuarial risk assessment for insurance purposes, like depending on how you spend your money, like how you should pay taxes. Interesting. I don't think I can give a clear yes, no because it would very much depend on the design of it. Sorry I'm falling back into policy speak, but I that's.

That's. Good, because I like, I think I will be very open to exploring the idea and to assessing the associated risks. Yeah. So I wouldn't. I wouldn't dismiss it entirely, but annoying answer. But it depends, no? That's that's that's the payroll science answer. OK, Evaluate and assign household chores to family members. Yeah, I already say yes, yes, yeah, yeah, for sure. I think probably depending on people's perceptions of their own levels of emotional and

other labor in a family context. No, I, I think this would actually make sense. And it would also make sense to try and get it to play towards someone's strengths and, you know, to pick up like help with scheduling around some of the tasks and so on. I think there's possibly quite a lot that could be gained around, you know, improving common issues in families, yeah. OK, now we're getting into another policy, one managing disaster response and relief funds. So, allocation of emergency

resources in disasters. I think I could see how that could be quite a good one, especially if we're thinking that we're currently restrained in that through like personal views, constrained data of hierarchical decision making and power structures. This could actually lead to some more fairness if done well, which I'd generally be in favor of. Nice. OK, decide when you're ready for promotion based on performance

data. Again, I wouldn't dismiss that as a thing, because I think performance and promotion decisions are also so often made on vibes, and it might be interesting to take a think about what would happen if vibes aren't part of the equation anymore. Does it actually become fairer or does the system collapse because people stop being able to work with each other? So. Again, very often to experimentation around it and what would happen as an

unintended side effect. OK, provide hyper personalized voting reminders tailored to past behavior and preferences. Yeah, nothing to lose there. Remind them all they want. It seems like a no brainer to use more personalized initiatives to get people to vote, right? Yeah. I mean, I'm sure we can come out with 10 risks of this as well, of course. But I think overall, I would think that if, especially if people are able to opt into it, that'd be great.

Final one, create tax return, not just optimize deductions and encourage social beneficial contributions. I'm not sure about that one because who decides what's socially beneficial? I think that's possibly like the but I'd get hung up, hung up on on this one and I might come back to the urinal. Do we mean that there will be a

camera installed? I was just thinking I was way too quick to say yes to something that wouldn't impact me. But some people might say, hey, Laura, that big camera, are we assuming the camera? In which case I might say no for the protection of men. Yeah, yeah, no, maybe it would be one of those cameras that is actually not picking up on exact film footage, but more something more vaguer or I don't know. Yeah, it's it's, I actually didn't think about that far as well.

That's a good. Point You're the design guru. You can make something, make something happen some. I'll ask Larry David, he has some ideas as well about urinal improvements and stuff like this. OK, that was really good, well done. It was very, I think I selected some hard ones for you, I realized in hindsight, but kudos for going through them and to not make any easier. The final question for you is

what do you feel like is your most controversial opinion about AI? My most controversial opinion, I think my this opinion was very controversial around 3-4 years ago, where if you happen to be stuck in a pub with me, you'd hear me talk about my big fantasy that when I'm old I don't want to be lonely. And what I want is an AI companion, someone that I would give them access to all my data, all my, you know, WhatsApp, social media data over the last.

I don't know when am I going to die in the last 60 years and I'm going to have the perfect person to interact with someone that's not just going to laugh about my jokes, but going to tell me jokes that I find funny. Someone that can remind me of things that bring me joy. Ideally want some sort of bodily feedback so it optimizes for my positive emotions. And I think that was possibly quite controversial back then, that idea that I'm striving for the perfect AI companion when

I'm old. I think nowadays with people interacting with the GBT actually seeing some of the potential benefits that can envision that more. Although when I say it the way I say it, people still have a bit of reaction to it. So it's not terribly controversial, but it's possibly being us round circle in terms of my my own entry to AI, which was probably actually thinking about the loneliness question in a in old age. Yeah, no, I love that. That's a great one.

And honestly, I love this conversation. It was so much fun. I feel like we could have, I've talked for many hours about many of these topics, but I I'm so happy we had a chance to bring onto the podcast to talk to you and bring your perspective and insights. Thank you so much for having me. Yeah, thank you. And thanks also follow the one for work you're doing and thanks for helping with setting up and bringing paper bytes to life as well. So yeah, a lot of thank yous.

Yeah, I know. Thank you for including me and for having me today. Such a good chat. And that's a wrap. You've been listening to the behavioral design podcast brought to you by Habit Weekly and Nuance Behavior. Sam and Aline tell me this season is packed with incredible insights about behavioral design and AI, so be sure to subscribe and share the podcast with your friends. Though you might want to keep it away from your enemies. In case you haven't noticed, I'm

an AI voice. Yep, pretty crazy. Quite the improvement since last season's. AI outro, don't you think? If you'd like to collaborate with us at Nuance Behavior, where we use behavioral design to craft digital products with Nuance, e-mail us at hello@nuancebehavior.com or book a call directly on our website, nuancebehavior.com. A special thanks to the amazing Dave Pizarro for our show music and to Mei Chen Yap and April English for their help in producing and publishing this

episode. Thanks again for tuning in. We'll be back soon with another exciting conversation where behavioral design and AI Intersect happens to. Murgatroyd. Oh. Oh Devero wigs, we guarantee our wigs aren't made from hair off dead bodies. I don't like it. I don't like it at all. You don't. Why not? Why bring up the thing about? Dead body hair. Our wigs aren't made with dead body hair. Yeah, that's the whole point. So why bring it up?

Advertising one O 1. Answer the question before they ask it. Who's asking that question, Rhonda? The main reason people don't buy wigs is because they're afraid they're made from dead people here.