Personalized AI with Amy Bucher

Hello and welcome to the Behavioral Design Podcast. This season we're diving into the intersection of behavioral science and AI. We want to make sense of the state of AI, from understanding how humans interact with intelligent systems to using AI to do behavioral design itself. I'm Aline Holsworth, a health tech advisor specializing in AI and product design. Over the past 15 years, I've been crafting human centered products with behavioral science

at the core. At Apple, I LED Behavioral Science for Health AI, designing and launching AI powered

features to help users reach their health goals. And I'm Samuel Sultzer, your second Co host. I'm a behavioral strategist specializing in hybrid formation and designing products that drive long term baby change. I work with leading tech organizations integrating AI to scale behavioral design for good. And I'm also the founder of Baby Bites, a dedicated community on behavioral science and AI. Quick word on Nuance Behavior where we help organizations build impactful digital products

using behavioral design. We only take on a few clients at a time to ensure the highest level of quality for our tailored evidence based solutions. If you'd like to become one of our special projects, e-mail us at hello@nuancebehavior.com or we could call directly on our website, nuancebehavior.com. So Eileen, I'm interested to know, have you ever considered buying AAI hardware or like wearable device? Have I ever considered buying one? I mean, AI is already in all of

our devices. So I feel like that's a trick question. Maybe that's true. I know what you're getting at, yes. The clip that goes into your shirt pocket that listens to all of your conversations, you're not talking about like, FaceTime or like, you know, your sort of standard encryption, No. That's right. Yeah. Like the humane pin that you describe or the rabbit, R1 I don't know if you saw that at all. Orange red thing. Google Glasses.

No juice? Yeah, Yeah. No, no, I have not ever considered purchasing one of these. OK. Well, I think they're interesting because they have created quite a bit of hype and interest especially in the last year or like 1 1/2 year or so. And the one you mentioned, Humane Pin and also the one I referenced to Rabbit R1.

Both of them have followed this kind of like curve of being like really, really hyped and people getting excited about especially the R1. But then once I actually get to use it, they've like used crashed and burned like it just hasn't been received very well. Lots of. Excitement from the early adopters and then a big, big dip. Yeah, not even early adopters, like people from pre-orders before they actually get this

thing. And then once they get the thing, they realize like this is crap basically. And so in terms of ingredients, I feel like one thing that they have in common is that they have 100% hype and syrups and behavioral science in terms of being used. And I think that's a little bit of maybe the theme, of course, in this season that we're going to kind of look at payable science and AI and how they overlap or how we can see and understand each from different perspective better.

Because I think that's one thing that's like a big loss is this kind of idea that use because you can assign a very intelligent system, like some form of trained language model, doesn't really mean that you per extension know how people are going to use it to interact. With right, right.

I mean, obviously I am biased as a behavioral scientist, but it completely boggles my mind that you could even consider making a piece of technology primarily for humans to interact with without considering how humans will interact with it. It just like, wait a minute, it's, it's, it's I don't know how much of my bias is playing into that, but yeah, like, I would love for someone to explain to me why behavioral science is not absolutely essential to success.

This situation. When I worked at a large consumer electronics company a few years ago, we were very interested in using behavioral science and AI together and really tried to intentionally look at how they could work together to make each other better.

So AI helping behavioral science, but also behavioral science helping AI and and really kind of thinking of the two as a two way St. So if you think about how behavioral science could contribute to AI, how an understanding of human behavior can help both generate better predictive models, maybe eliminate bias and so on. But then on the other side of things, how AI can make behavioral science better. So as a research method or intervention technology, for example.

And and we talk a lot about that in other episodes. But today, when we talk to Amy, we're really gonna focus on the 1st St. how behavioral science can contribute to this AI tool. And of course, the ultimate goal it always, at least in our opinion, is how do we change behavior for good using this combination of AI and behavioral science. And I feel like that really came through in our conversation with

Amy as well. Yeah. And I'm interested because so you, for example, you led behavioral science for healthy AI team at Apple and having have this kind of exposure for you, is there in commonality? Like do you see anything that this AI side of things and also the behavioral science side, do you have anything in common? Lots of things. Many, many things.

One sort of overarching theme that I noticed as I was working in the the weeds of AI and behavioral science was just how both of these fields are often packaged as these miracle cures. And our behavioral science listeners will relate to this, I'm sure. And it's so frustrating, right? This idea that you can just throw in a bunch of nudges and all your behavioral problems will be solved or even your non behavioral problems, like everything will be fine.

Well, the the the exact same situation machine learning engineers and scientists are faced with. Well, there's this perception that you can just automate everything with AI and you know, we don't need to work anymore. And you know, of course, like the reality is that neither of these things are magic. They both require a lot of work

and thoughts and intentionality. But yeah, we often did laugh about this similarity and just how wrong the perception is and how this hype is countered by the reality. Yeah, I mean, it is almost hilarious where you kind of see in the same way as maybe people naively at some point with like thinking about throwing use notice at problems to solve

them. It's also a little bit like the last couple of years you've seen this kind of like panicking of like, OK, we have to make our products AI somehow. Let's throw in some form of AI function and that's going to make the user experience better somehow. We're not really sure why or how, but we're just going to throw AI everybody else. Is doing it so. We must Exactly. No, but I think that's super great tech and I think it's so true.

And yeah, we have a great guest in this episode that we explore

some of this and much more with. And yeah, so let me introduce Amy Buker. She's the chief here, officer at Lerio. And if her name sounds familiar, it could be for many good reasons. It could be that you have read your book Engaged, which is a really fantastic book. Find one either getting starting behavioral science or applied behavioral science or just wants to get some insights for how it can be done in practice.

You might have seen her in some form of conference speaking or you've listened to the podcast before, because we had her in a fantastic conversation in a previous season and she's one of the cases where actually there's so many people to speak to. And so we tried to invite new people to the podcast.

But with Amy, we want to have her back not only because she's a fantastic representative of the field and practitioner, but also because since we last spoke to her, she has really been thrown into this intersection of AI and behavioral science with her work, Illyrio. And she's been focusing on specifically working with and understanding how AI systems can be used in tandem with behavioral science to personalize health interventions.

So this includes looking at how to support patients going through some form of chronic disease management, but many different types of context and scenarios. And her work specifically using precision nudging, I think it's even references hyper precision looks like kind of how to make these interventions both effective, but also scalable, personalized and equitable. So super excited to bring her back for another conversation. And yeah, what do we cover?

So as you might expect, we talk all about how Amy does just this, applying AI to personalized health interventions and specifically how she uses reinforcement learning to nudge users toward healthier behaviors. Amy shares her findings from a recent scoping review where she looked at the landscape of AI being used in digital behavior

change interventions. And for me, I don't know about you, Sam, but one take away was just that behavioral scientists might need a little bit of help understanding what's AI and what isn't. We cover why behavioral science is so, so essential for getting it right in AI and especially when it comes to health behavior, and so, so much more, including Amy's most controversial opinion in AI and what it would take for cat memes to be personalized to her

satisfaction. Happens to Murgatroyd. I'm happy to say welcome back, Amy. Good to have you on the podcast. It's great to be here. I'm so excited to talk to you both again. Yeah. And I guess the first question I

want to dive into is AI. Think still rather recently shared scoping review that you made with a few of your colleagues looking at something really relevant for the system. Given that we're kind of like trying to understand this kind of intersection between behavioral science and AI making sense of, OK, what is AIBS in terms of like behavioral science and what is AIBS in terms of bullshit and kind of seeing between the lines of like which is which.

And I think this scoping review was quite interesting in that it looked at how AI and ML is used in digital, maybe change interventions. And so, so Daveen, maybe you can give us kind of like the summary of what you found, like what was the main takeaways and then we can maybe dive into do some further findings as well. Sure and I love that you characterized it as what is BS and AI cuz we actually had the same 2 conceptions. Like from a a sort of legitimate

scientific work perspective. I really wanted to know how is this happening in the real world? Like what is the landscape out there of the different AI tools that are being used to change real world behaviors? And I was going to add through digital means that AI pretty

much is through digital means. But the other, the other thing, the sort of more personal reason is we have noticed that there's a lot of companies or groups who talk about doing AI and then when you look under the hood and, and carefully read the description of what they're doing, it's not really AI at

all. It might be something like a hard coded algorithm or a thing that I found a lot in the scoping review were interventions, products that called themselves AI. But we're using a Wizard of Oz setup for the testing. So it was basically mocked up. There was a live person behind the screen making it appear that there was an artificial intelligence interacting with somebody, and so we ruled those out as well. That might be an early prototype for an AI, but in and of itself

it's not. Normally my team, my behavioral science team will do these sort of encapsulated projects that we can run. It's a side passion project without involving colleagues elsewhere in the company because it can be difficult to get time. I mean, you know, it's it's hard to get time from other teams. But with this paper, we knew from the beginning we had to co-author with at least one of our AI scientists. So one of our co-authors is our chief AI scientist, Chris Simons.

And the reason why is we needed help as well figuring out like what are the legitimate search terms for AI? And then once we have done our lit review and identified some papers, we really needed Chris to read some of them in detail and tell us, does this meet your definition as a career computer scientist who specializes in ML of AI? And so that was unusual for us to absolutely require an AI colleague to collaborate on the paper.

When once we built our search terms, we ended up finding a total of 3000 exactly, which I think looks fake, but it was 3000 exactly papers that met those high level inclusion criteria. And by the time we went through all of our filtering and review, we got it down to 32 papers that covered only 23 different interventions or products.

And you know, of course, these are the ones that have gone through the trouble of publishing in the peer reviewed literature, which means typically you have to have some kind of results. Like there's the barrier to being included on this list was somewhat high. And I know there are companies out there that are doing real AI to change behavior in the real world.

But in the scientific literature, we only found these 23. And it was interesting to me as someone who spent her career in industry, only six of them were commercial products. The others were all developed for research purposes and may or may not ever be commercialized and made available to the general public, which to me seems like a little bit, well, it's a pain point with the academia industry crossover that I knew.

But it's, it's really poignant to the example of it because some of these products were great. But yeah, we we wanted to break down what types of AI are being used and who's doing it. And one of the things that really surprised me was that all but two had proprietary AI where they developed that technology themselves. And Lyrio is among them. Our, one of our papers actually met our inclusion criteria. We have developed our own. And I, I know that there's good reasons to do that.

You know, you want to own the IP, you really want to create something that's competitively differentiated and and owning it makes a lot of sense. But it's also really limiting because first of all, the barrier to doing that is much higher than using products like ChatGPT and integrating that

into your product. There was one that came up in our review that had cobbled together some products from Amazon and Microsoft, but you know, these commercially available and they probably got that stood up much more quickly than somebody who grew up their own technology.

And the other thing is, when everybody develops their own technology, it limits the ability to compare a bit because you don't truly know the intimate differences between what one company develops and what another company develops to understand how does this work and how do we draw conclusions about the effect of having our behavior.

Yeah. And I think that speaks to kind of this interesting thing that happened where I feel like in the last three years especially, there's been like this booming quote, UN quote, like AI tools. And it feels like all these

tools are in some ways unique. But if again, looking under the hood, it goes back to maybe you see like 3 or 4 foundational large language models that are used like either opening eyes, either entropics, maybe some like llama model, or, you know, a lot of them are basically using basically the same things

on the hood. And then they use basically create a product out of interface, like how to kind of make it easier for people to maybe directly make use of those models and so on. And yeah, is that something similar that you came across as well or maybe? Yeah. Well, actually on on large

language models specifically, we found that they're basically not being used in patient or consumer facing behavior change technology. And I wasn't surprised by that. I was actually pleased by that. I don't think this is a controversial take, but I don't think that large language models are ready for prime time in some scenarios like providing medical

information to patients. And actually, Lyrio was just a sub awardee on a grant from ARPA H, where we're going to be looking at exactly that kind of thing. We're going to be helping to develop some quality tools for mental health chat bots that would help detect if the information it produces is credible and complete. That's the sort of thing that's missing right now.

And there's research on the use of large language models in healthcare that from my read, the place where I've wound up is it can be appropriate for some clinician facing scenarios because someone who's a clinician has gone through that education and professional experience can put the bullshit meter on it. They can read something that's in a patient portal developed by LLM and they can edit that and

make it accurate. There's a paper that I've cited a few Times Now where they had a Gen. AI create first draft responses to patients in the patient portal and then the provider had to review review them before they sent it out. One that really struck me was the ChatGPT or the the Gen. AI said, yeah, patient, you're taking 20 milligrams of this drug, but you take it twice a day in the 10 milligram dose. It's fine if you get the 20 milligram pill in your subscription to save money and

just split it in half. And when the clinician read it right away, they said, no, there's no score line on that 28 milligram tablet. You're not gonna be able to accurately divide it in half. That's like a human quality information that would be very difficult to get out of the LLM. And the clinician has the expertise to instantly recognize that it's missing. We're on the patient side.

If they'd received that original message, you know, at best they would have learned through hard experience picking that pill up at the drugstore and seeing they couldn't break it in half. So I, I, I think LLMS are not really ready for prime time for that scenario. And that was kind of reflected in the evidence we found in the paper where there were several different interventions that were using natural language

understanding. So they were deriving intent and meaning from free text entered by the user, but whatever they were responding back with had been pre written by an expert.

That's an interesting and I guess one of my reflections from also reading this and I think you have it in the conclusion in terms of one of the significant opportunities for people in Marvel Science is to become more like able at knowing and understanding terminology in AI. And we can touch upon that already a bit, you know, and I guess I want to hear like a little bit from your journey working with Leary, for example, like you've had a chance for the last couple of years to really

work closely with like you mentioned colleagues that are like coming from the maybe more like hardcore machine learning backgrounds or similar. Like how do you think about kind of like developing that terminology and how how's that been for you? It's been hard. The Dunning Kruger effect is really all I'm living it. I know more about AI than I've ever known, and I'm also so aware of the limits of my own knowledge.

But I think what I'm getting good at is actually what I like to see on the B side side sometimes too. And I've been really passionate about making the tools of behavioral science available widely to people of different backgrounds and different professions. But I do also believe there's a point at which you might be working on a problem and you say, oh, I need someone who really has, you know, trained in this and reach out for that

expertise. And I, I think I'm in a similar place with AI where now I can talk about it intelligently and correctly to a point. And I've gotten pretty good at recognizing what that point is, which I think is probably about as good as it's going to get. I'm never going to replicate, you know, Chris, who I mentioned our chief AI scientist has his PhD in computer science and, you know, worked at Oak Ridge National Labs leading teams over there. Like, I'm not going to learn what he knows.

But the the other good thing about it is one of the things he's taught me is that AI is. Some of these models that we're using, the supervised training models, like they benefit from subject matter expertise in the domain where they're being used. When he frames it that way, we become essential partners. And that's kind of a really cool thing about working with the AI is that we can't independently operate.

We have to work together. And so the questioning is going back and forth, which makes it more comfortable for me to say, Chris, is this written correctly? Is this really what I'm supposed to be saying? Does this mean what I think it's meaning? And I learned a lot through that kind of thing, too. And what would you say are some of the most common misconceptions about AI, at least in terms of from the perspective of the behavioral science community?

You mentioned hard coded rules and really algorithms that don't adapt as being AI but but what else is there? Yeah. Well, IA big one I'm seeing right now that drives me up the wall is I think people are increasingly in conflating generative AI with AI as an overall category. They say AI and they just mean generative AI. Adjust me. Yep. Yeah. And that's just such a small subset of what AI can do. And it's not the type of AII work with our product uses

reinforcement learning. And I am actually a huge fan of reinforcement learning. I think it can be very powerful in behavioral science. And so I guess it's not a misconception so much as a lack of awareness of additional AI tools and approaches that could go well beyond or augment what Jen AI could do. And again, I also think Jen AI is not ready for prime time with that patient or consumer facing

content. So to hyper focus on that I think is really, really limiting for a behavioral scientist who wants to change behavior. Yeah. And, and it definitely seems like this is a phenomenon that's really only come about in the past couple of years. I remember when Gen. AI was really pretty niche, right? You only heard about natural language processing in these like much more narrow use cases. And now it's everyone is using the LLMS. I mean ChatGPT was a game changer.

Putting a public facing, easy to use interface on an LLM just totally changed the world. I mean, that's an important lesson, right? Do you think that lesson can be transferred to other use cases of AI? Probably, actually. So reinforcement learning that I

work with, we call each algorithm an agent. And I'm clarifying that because sometimes we'll talk about the agent decisions and people are like, wait, who's this agent? It's an algorithm. When we set up our interventions, they are multi agent systems and each agent is responsible for a specific choice. But those choices ladder up. So with Lyrio and Precision Nudging, we're sending personalized messaging and we might have an agent that decides what channel we use for a

person. Is this person getting an e-mail, a text message, a push notification? Another agent might actually decide on the headline that the e-mail gets. And a third agent decides on what's the body copy of this e-mail? What are we actually saying to this person? And then yet another agent might decide when we send it.

And each of those agents there, we we consider what we do, a type of objective driven AI. That's a term that Yan Likun, who's the head of AI at Meta, has coined in the last year or so. And reinforcement learning is a perfect symbol of that because what you do when you train those agents, when you set up those algorithms is you give them objectives. So the objectives in our case are almost always we want the person who receives this message

to complete the target behavior. And I always use the example of mammogram. That's one of our most mature interventions where we have pretty strong evidence. And it's a very straightforward behavior because what we're asking people to do is open our message, schedule your mammogram, attend that mammogram. And what we can basically do is tell the various agents that are making decisions about how we communicate with somebody that it will be rewarded. And again, that's conceptual.

It's not a money thing. At the highest level, if someone has that behavioral achievement, they've actually finished the mammogram, but as a lower reward if they schedule it and a lower reward reward yet if they interact with the message. And in that way, we're sort of able to prioritize the meaningful behaviors. That's something where you could have an interface that people interact with and you could start to play around with, you know, what do we make the most

important here? And especially when you get into more complex behavioral patterns, you know, mammograms, easy to talk about because it is so straightforward. But in reality, if someone has a chronic condition or even is healthy but is orchestrating all their healthcare, like Wellness visits and vaccines and cancer screenings, that can get pretty

complex. And that's that again, is where, you know, I think there's some room to pull levers and twist knobs, so to speak, and figure out what is the right way to orchestrate. But again, if you set those objectives, reinforcement learning is going to start to figure out what is the best way to get this person to get to that objective under the scenario that it's implemented in. Yeah.

And. I think it's interesting in terms of like there's been increasingly use of the term agent, but I do think that most uses where I see agents being referenced, it's actually not with that reinforcement part to it. It's often times a little bit simpler in some ways. It's often times we have some form of setup that automates that.

You know, if this thing is triggered, then that activates this agent that maybe uses some form of let's say open AIS large language model to execute on some form of like quite simple thing. And it's pretty good. You can do some cool things just by doing some of that stuff, but it's still not used enough time to reinforcement learning.

So I think, I think that's actually still like a pretty good indicator of maybe a little more advanced use cases where often times there's people involved that are also coming from lately the machine learning background or have a more technical backgrounds as well. Whereas they they can be like a lot of Bros on YouTube. This is like, hey I I designed an AI agent or something. And that that actually is where I mentioned in the paper that we did the scoping review.

Only two of those interventions used publicly available and I can't remember the name of it, but the one that didn't use the Microsoft and the Amazon used it was some kind of web interface that allows you to basically say if this, then this, you know, deploy this. And so that was how that one worked. Maybe quick thing to tie up before we move to the next question. What are them the Pareto principle terminology terms that people should know about within Tai?

Is there some like really important ones that people in Babel science should you know or be able to better understand? I think we've talked about some of them already. So things like generative AI and understanding that's a subset of AII think the term objective driven AI is very useful.

I'm not one to jump on a buzzword that an executive at a buzzy company like Meta has has coined, but in this case I think it really does a nice job both describing what certain types of AI are built for and also helping to distinguish it a bit from generative AI. Although there, you know, those could overlap a bit in the Venn diagram.

I can imagine scenarios. We included a glossary in that paper that Chris actually wrote, but then I edited to make sure that it was understandable to someone like me. And we're just about to actually make a blog post version of that on our website. That because we think that's really important, that these terms are at least of passing

familiarity to people. And I'll tell you, I can't use all of those terms comfortably in a conversation myself, but there's, you know, the ones that I use, the ones that sort of affect our product. So I understand things like supervised learning and unsupervised learning.

And you know, where that supervised learning is really where our behavioral models come to bear because what we're essentially doing is looking at what the agents want, want to do and course correcting them based on our expert knowledge and the logic models that we set up with our content. So for me, understanding that kind of thing is important.

But I think for the average person, just understanding that the models need to be trained, that there's different ways to train them, and maybe even that the quality and origin of the training data is really impactful on what the final product looks like. I don't think you need to understand all of the guts of how that works, but to know that's a piece of it is really critical.

And you're working with contextual bandits, which I like to think of as kind of like superhuman experimenters, right? They're like sending little bits out into the fray and seeing what comes back and then choosing the winner and then moving on to the next experiment and just optimizing in that way. Well, and you'll like to The early papers, the ones that my colleagues cite all the time about contextual bandits are based on the video game StarCraft.

So when you say like shooting things out like. Yes exactly, almost literally. That's so funny yeah some of the AIML terms just crack me up every time I see them. The multi armed bandits and just like I hard to take seriously but. QQ Learning is one that I hear sometimes and I think it might actually be in our glossary, but I don't know exactly how it works, but I always think of Star Trek. Hard to not go to sci-fi really when you're when you're talking about AI in general, right?

Totally. So I would love to really get

into the nitty gritty of what you're doing at Lyrio with the large behavior model with precision nudging. Like I want to understand how does it all work? What are you doing with your bandits? What for, you know, the the goodness of science and other people trying to solve for health problems out there. Like what can you share about how how this works, where behavioral science fits in, and how AI and behavioral science are really like, really working together to make your

interventions amazing? Sure. And I think I can share quite a bit actually, when we begin designing a new intervention, it's my team, the behavioral science team that is first on call. So we almost always design interventions because we have a client who has a use case that we're helping them to solve for it. So there's an external partner in the loop, and what that provides for us is the opportunity to research the behavior in context.

So we work with the client to understand their objectives. And a lot of times these are things, no surprises here. They want to close the gap in care. They want to improve their reimbursements from insurance companies because they've raised their quality or they're meeting a certain metric better. Sometimes it's a population health goal. We want to get X percent of our population below this health metric.

We have an example right now with a client, Cone Health, who is awesome, awesome, awesome, and has been really wonderful about allowing us to talk about the work that we do with them. They're helping us to, well, we actually have launched it already. We have an intervention for

hypertension management. And one of Cone Health's goals, because they're really committed to HealthEquity, is not only to get, you know, meet meet the sort of target metrics that are nationally available for hypertension control, but also to make sure that they're black patients also meet that metric. Because typically if you look at a patient population, if it meets that metric, on average, it tends to be white patients who have met it. And then black patients

specifically will lag behind. So Cone is committed to making sure that that gap doesn't exist in their population, which is awesome. But one of the first things we did with them was understand that was specifically a goal of theirs because then that allows us to design the rest of the intervention appropriately for that. We identify the target behaviors. We do our, you know, beautiful literature review to understand the determinants as seen in other research.

And then this is where having that partner is also really helpful. We will do research in context with them as well. So we talked to, you know, particularly like their clinicians actually to understand how hypertension management happens and you know, what are the sort of contextual barriers that we need to understand and design for. We put all of that into a logic model. This is all still my team.

So we have a document that we create that we call our strategy matrix and we actually outline what are the determinants that we've prioritized to addressing this intervention? What are the mechanisms of action through which we could influence those? What are the specific behavior change techniques that we are going to operationalize in our messaging to use that MOA and address the barrier?

And then the content and visual design teams are part of behavioral science at Lyrio and they work on operationalizing all of that into assets. We create lots of little content components, so we sort of atomize what goes into a message so that we have more things that we can select from to personalize. And we also create visuals that operationalize the behavior change techniques, which I think is pretty unique for us. And we're starting to accumulate

some data. That approach is helpful in both engaging and converting people to the behavior. So that the idea of the hypothesis was if we can kind of infuse the BCT into somebody's brain through multiple channels, let's do that. We build out all those assets and around here is where we really start interacting with the AI team because some of the things that we need to ensure that they do, they have to train the agent or the agents on the

content that we've developed. And we're using our logic model to do that. So that's where part of our expert knowledge is infused. One of the steps of training the agent is doing a semantic conversion of the text. So if you've seen like encoder decoder models, Bert is Google's tool for some of this. And I believe we actually do use Bert at one point in our process. That's a semantic conversion. So it looks at, are these words similar or different in a multi dimensional space?

How would they cluster together? And this is actually a place where our expert knowledge makes a big difference because there are pieces of content that might seem very similar from a semantic level if you don't really know the meaning of the words, but you know that the words are kind of similar. But if you're a human being and especially behavioral scientist and you read them, you're like, those are very different.

And so we can basically use our logic models to force a separation between items that are different in terms of their meaning, where the model might have otherwise put them close together because of their words being similar. So there's that piece of it. And then the other piece of it is helping to set the reward function. So I again, I use that really simple example of a mammogram where the highest reward function is achieving the mammogram, actually getting it done.

But with some of these more complex behaviors we have to think about, you know, how do we prioritize them? What is really the objective from a behavioral perspective and how do we build that into the algorithm? So that's what it's maximizing for. It would be really easy to abuse this sort of approach, and I would suspect that Facebook probably uses some of this stuff, for example, because of the way that their news feed is

created. And sort of putting the next thing in front of you that keeps you on the screen like that is an objective that you could train an agent for. In our case, we specifically don't want to do that. I would rather you not even read the messages if I can somehow otherwise get you to do the behavior. So we just want to be really careful that whatever we're rewarding the agent for is the thing that's really most helpful

to our objective. And sometimes that's also thinking from a behavioral scientist perspective. If you have somebody who has a lot of healthcare needs, it's not always the most sort of medically impactful thing you want to go after first. Because people who have a lot on their plates in terms of changing their behaviors and addressing their health might be better served by trying something easy first, you know, building self efficacy and

notching that small initial win. They may be better served by, you know, doing something that is easier for them for some reason. Maybe they live a lot closer to the facility where certain healthcare appointment happens than the one where the like, the big appointment happens. We need to think about those things too.

And that's where AI can be really helpful at parsing out like complex data about a person and, you know, sort of identifying what is most likely to work for this person, which could be a bridge to the LLBM, the large behavior model. But I don't want to just talk for like 4 minutes you guys say.

Well, this actually brings up a maybe semi interesting question for me is what if you put a negative incentive on reading of the e-mail or you know, time spent on the e-mail as opposed to just a small incentive, really disincentivize that outcome and then really, you know, outweigh the attendance at the mammogram for example? Yeah, it. Happen. So I'm going to give you a really, I'm sorry, this is a disappointing response.

I don't think that was actually possible because one of the things that we're actually learning and and this just, it doesn't matter to us that much because we've never put a huge reward on interaction with the product. But e-mail metrics in particular are getting really hard to measure because some of the changes that like Apple and Google have made to their e-mail clients. So all of those engagement metrics are artificially inflated by some of the design changes they've made recently.

And so we're actually even talking about them less and less. But what we have been doing that I think is in the area of your question, people will reply to us if we send text messages, if they received the intervention through a text message, a surprising amount of people were fly back and this is not with a system command, it's with substantive information. And so we have actually we have in market now, but it's going to be it's an MVP that's going to

become I think really powerful. We are basically analyzing with some natural language understanding with an intent model, what people are replying to us. And we are using that to sometimes negatively reward the agent because sometimes people are responding very negatively. You know, they're they are upset that they're being contacted or this is at the campaign I'm thinking of as a vaccination intervention. And vaccines are always a little bit of a hot button for some people.

And so there's some people who are replying like with conspiracy theories and political commentary and we're basically negatively rewarding the agent in some of those cases to a small amount. And we're also taking actions in those cases as well. So we we can send replies that are appropriate to what the person said if like they ask a

question. In the case of somebody who's really angry, what we do is we suppress the messaging because they don't wanna hear from us. You're not gonna get through to that person. No, no. And it's like, why are we spending the 10th of a set to piss this person off even more? Let's not do that. Why won't you get the vaccine? I really want to implant this.

Yeah. So I'm really excited about that because I think that our bidirectional capabilities are going to become a really critical part of what we do. And that becomes a way for us to also gather information about a person's experience and more quickly identify what barriers they're actually experiencing and better train those agents. Like what is the right type of messaging for this person for this behavior? Can we talk a little bit about

barriers? So you mentioned you do a lit review and I know you have some a lot of theory that's really oozing through your models. We talked a lot about Com B. How does that work with the sort of top down theory driven systematization versus the bottom up like talking to health systems and clinicians and trying to understand what did patients actually say? How did those two come together? Yeah, that's a challenge. It's not a challenge.

It's, it's a fun part of what we have to pay attention to. A lot of it is just early documentation. You know, we, we put all the lit review things and I, I mentioned we have a document called Strategy Matrix. We use Excel to build it. It's very glamorous fancy. But the initial version of that is fairly large because of course all of the lit review papers are using different terminology or maybe you're slightly, you know, defining a

barrier differently. So we do some organization and right sizing of what we find in the literature. We're recording what we found from the client research and we're trying to basically bucket it together with what we see in the literature. And what we typically wind up with is the clients experiencing things that were already known to research for the most part, but there's usually a handful of things that are really specific to them.

What we tend to do is we include them in our design process. And then when we develop our content, we always, even if somebody comes to us and they want to use an existing intervention, we go through a tailoring process and then what we call content activation to make it live in their environment. And that's an opportunity.

You know, we have system variables, we have the opportunity to tweak and change some of the messaging and we can infuse some of those client specific things in at that point. And what we typically try to do is just make sure that we do it in a way that is consistent with

the logic model we've built. So what that might look like is if we already know that convenience is a barrier for somebody, we've already identified, you know, the BCT in the messaging, but we're working with a retail pharmacy that really it's important to them that the patients know that they're open more than 9:00 to 5:00. We can build that into the existing message that already uses that sort of approach, but now it becomes very customer

specific. And if it's something that's really unique to that client, we can just build that out as their own, you know, bundle of content that is activated for them, but doesn't become part of our core library. And our CMS, I won't get into it, but our CMS allows us to sort of this is the master content library, the overall intervention, but then here's the iterations of it for various clients. When you're going into a new intervention, are you starting at 0?

Say we know nothing about this patient. We are going to send every patient, you know, some randomly selected message from the library, and then we'll learn to get to, you know, their optimal message. Or can you already start from somewhere and say, well, I know that this patient lives, you know? 20 miles from the clinic. So that's likely to be a barrier to actually attending the appointment. Which of those? Both, both and sometimes the

other places as well. So we have some clients that we work with and it's a net new use case, new intervention and for whatever reason they are unable or unwilling to share very much data with us. And in that case, it is essentially, let's send something randomly and start to learn from scratch about these individuals. And I know that Chris, my counterpart on the AI side is probably would probably jump in and correct me. It'll be like, it's not really from scratch yet.

But for my, let's just call it from scratch sometimes if it's a new intervention and it has similarities to other interventions, we're able to leverage the training of the Asians for those other interventions. So a good example here again is vaccination. We have nudged all different types of vaccines. We've done COVID, flu, RSV, pneumo, shingles, and there are. We've done the research to know that the barriers for receiving those vaccines are not completely overlapping, but

they're a lot overlapping. Covic is the one that's kind of different, but the others are a lot overlapping. And so we have been able to basically use agents trained for other vaccinations to get so that kind of jump start. And then you specifically mentioned home address. And I know you didn't do this on purpose, but that's a piece of information that's particularly useful to us because on my team, Sarah D dot, my VP of Behavioral Design, has her PhD in health

geography. So she intimately understands space and place from a behavioral perspective. And then on the AI side, we have a team member, Rajiv Vatsavai, who is also faculty at NC State, and his expertise is geospatial transformation. So between the two of them, we actually can get a ton of mileage out of somebody's address and specifically identifying things like proximity to where the healthcare behavior is delivered, but also things like, you know, are they near green

space? Are there food deserts depending on what's relevant for that intervention. So we can sometimes do transformations of variables that we receive, or that we have access to derive much more meaningful information from them. Wow, awesome. Those are definitely specialties that I did not. I'm very glad that they exist, but I had no idea. I know it's kind of wild too, that we didn't deliberately set out to assemble this super team of geography people. But. We did it. Really cool.

Yeah, yeah. That's cool. And I guess I wanted to in some ways turn the tables a little

bit of what we've explored here in terms of we've talked about kind of from a behavioral science point of view, what we can do with AI and the benefit that we can have in our interventions and so on. But maybe as a hypothetical, because I think a lot of behavioral scientists is kind of exploring like, what is my value in this new evolving landscape?

And you know, if you then look at the kind of Euro illyria and the thing you described, you know, what would be the cost of removing behavioral science from that equation? Like how do you see you come in as a behavioral scientist and adding value and what would be there if you were not there? Obviously I have a bias. I don't think I would work without us. No, I really do. Actually, I, I think it would be pretty bad and I'll give you a

couple of reasons why. So the first is we have actually done tests where we do our behavioral science messaging without the AI. So, and there's been, you know, various reasons that we've done that, whether it's deliberate or, you know, we were tweaking something. And but the behavioral science messaging by itself has a huge

impact compared to control. So, you know, I, I think that having that behavioral lens on the content that we send out to people, and I actually believe that the logic model behind it is also really important. I've seen other studies, other interventions where people create intervention outreach that's very strong, you know it, and it's what I might call a kitchen sink behavioral science approach.

Like how much can we put into this messaging to, you know, just convey all the different tools and reasons why somebody might be able to do this thing for us? Any given message really only has one BCT in it, one behavior change technique. But I think the fact that we get very specific and spend a lot of time, you know, just really making sure that comes through extreme.

We go through a quality check process where we have blind raters review all the items and determine, does this adequately operationalize the BCT and does it do it in a way that's distinguished from other BC TS? I think all of that means that the content by itself is just a huge value add. But then the other pieces, as I mentioned, there is no universal learner. That's a thing that Chris says all the time. You really have to train your AI against subject matter expertise.

And in our case, we are doing that against, you know, behavior science and healthcare expertise.

We use an example in our sales presentations that I think is, you know, it's a little bit intended to be provocative, but we actually have seen healthcare organizations who hire former executives from companies like Amazon or Disney, you know, these companies that are amazing at engaging people and they bring them in. And I, I so far have not seen a major success story in improving patient engagement in a healthcare setting with that expertise.

And I, I think it speaks to the fact that you really do have to infuse your approach with a deep understanding of well, how human behavior works, but also how does human healthcare behavior work in a particular system that provides barriers, limitations, enables some behaviors. So I, I, I don't think that the product would work without infusing that type of expertise. We, we might be able to get it to do something else.

It we might be able to get it to do something that's still like better than random, but I just don't think that. I think the delta between what we can do today and what we would do without VISI is huge and negative. And if you think about it, that's where most companies are, right? Lacking the behavioral science. Well, or or the AI, yeah, I mean, it's funny and and again, like we're we're a little bit jerks about this. But you know, Chris and I have been to conferences together.

We've locked the exhibit floor and we'll see these companies that are like AI decide and between the two of us, we're like, come on. Yeah, that's not real AI or that's not real D side. I know. Exactly which ones you're talking about. Yes, yes. And that was, again, it gets back to why we wrote that scoping review and sort of the, you know, the dual meaning of BS there, because we wanted to really know for sure what is out there in terms of AI.

Obviously everybody's gonna market in the way that they think is gonna be most appealing to their customers and things like that. But some of them are really doing AI. We wanted to know who they. Were all right. It is time for our quick fire

round. Are you ready? I'll tell you what it is. Don't worry. So this is a game that we've developed. It's called to AI or Not to AI. And so it's your job to determine whether the activity that we present to you is well suited to AI or not. All right, let's begin. So first one, negotiate a job offer to AI or not to AI? I think you can use AI in this. I wouldn't directly interface the hiring folks with the AI, but I, I could use it to like revise, you know? Hey, this is what I'm thinking

of replying to this e-mail. Can you revise it? Can you make it more succinct, more professional or getting stats? Can you tell me what a good salary might be for this role based in the city? So I say AI has a role in this. So really as a as a partner, not a replacement. Correct. How about file taxes? That makes me nervous. I think that the risk of being wrong is too high. I say no AI. OK recommend a mystery book. Absolutely yes. Write a mystery book.

No. It we've done a lot of work looking at the quality of content and I think the short form content can be pretty good, but anytime we ask it to do something long form or repeated, it's boring. The Sign. Behavioral interventions for a mental health app. No, I will not design A behavioral intervention for a mental health app without a clinical partner. About Taylor Cat memes to one specific preferences 100% how personalized will it get?

Well, so I have a cat named Jackie Daytona after What We Do in the Shadows. So Jackie Daytona, human bartender. That's so funny, by the way. I love that. Thank you. But I've tried really hard to get like Dolly mid journey to give me Jackie Daytona, human bartender, but he's really a brown tabby cat type things and they're very they're not where I want them to be. We're going to keep working on it all. Right. Maybe some promise in the future.

Yeah, I'm now just thinking about what we're doing in the shadows and I completely lost track of of thinking about this quick fire around, but OK, next one device and investment plan. Yeah, I think it could. But I this is again where I would want to review it and then go execute it. I wouldn't want AI to devise the investment plan and then execute it. And last one, identify patients at risk for diabetes. Well, yes, we we actually do a little bit of that. So yeah, I think that's a great

use case. Awesome.

All right, it's time to ask you the question that we end with all of our guests this season. Amy, what is your most controversial opinion about AI? I think AI is not that good yet. And I also think that we are unnecessarily afraid of it in the wrong places. So I see a lot of people who are like, Oh my gosh, AI is going to replace us. And I just don't think it's good enough to do that yet. And like I say, I'm on LinkedIn a lot, mostly because I'm told that's a thing I should be doing.

Obviously, notice a lot of people write their posts with ChatGPT. And so first of all, I can tell, you can tell if somebody actually wrote their post this way, because like I said, ChatGPT might look great in isolation, but it has its own tics and mannerisms that show up in it's writing across different pieces.

And like once you know it's voice, it's kind of easy to see it. And so I think that people overlook the fact that, you know, I can tell the ChatGPT wrote that it doesn't mean it's bad, but it's no replacement for your own voice. And then even things like LinkedIn fired a lot of their content writers and now they do that thing where they ask people to contribute to articles.

I'm assuming there's some kind of AI behind that in terms of generating like which prompts which people see and how they frame those to you. And I'll tell you, I don't actually don't read those social articles the way that I sometimes used to read the ones that LinkedIn staff writers wrote because there's just like a a difference in kind of quality and organization that to me as a reader leaves me unsatisfied. So I just don't think that AI has nearly reached its capabilities.

And I think that the human in the loop is likely to be important forever, forever. Maybe not in every case, maybe not in every case. I mean, we certainly have seen things where AI has been able to, you know, manage supply chains and computational sorts of things. But I just don't think AI is going to replace humans in nearly the way that some people think it will. Yeah, I guess time to the first thing is that I'm talking about

memes as well. We're talking about cat memes before, but a non cat meme that I've seen quite a bit with imagery is basically first there's this person who says like I haven't seen a lot AI generated images lately. And then you see them like with like a Oh my God face. Like I haven't seen an AI generated images lately. Yeah, Yeah, it is. It is getting better.

A. Little bit to your point, like maybe there's some subset where it's like really well done where you don't notice it, but a lot of it you notice. And especially, I know also you get recruitment stuff and I I get people applying for things. And I understand why people use AI for writing applications and various things. But it's also like I can see AI was used here.

Like, yeah. So we just hired several roles on my team and one of them was a writing role, a copywriter role, and that was the one where we saw a lot of ChatGPT generated application materials. We didn't see it for the behavioral science roles for or we didn't notice it for the behavioral science roles. I'll say that we might have seen it, but one of the ways that it was most evident was people

would put their work histories. Obviously that's what you do on a resume and they would have things in there. That was like 2012 to 2013 worked on precision nudging messaging. And it's like, I don't think you did. And I also don't think a human being would make that mistake, you know, Weird. Yeah. And And the other thing, again, because we're reviewing lots of resumes, you start to see

patterns across them. Yeah. I even think this was not AI, but when I was a grad student and I was teaching, you know, we'd always get people who plagiarized on their papers. And almost always the way that I caught them was they weren't the only one plagiarizing from the same source. Like I vividly remember one night reading a paper view, like did I just read this paper or looked and there were actually 3 copies of essentially the same

paper in my pile. So I think that's a lot of where AI gets caught is when multiple people use it for similar purposes and it isn't different enough. Yeah, it feels different to them, but then to the one who's receiving all of the AI generated text, Yeah. Yeah, I did use AII, gave it like my LinkedIn bio, which I've never liked and had it edited for me. And I, I what's on there now was edited for me by ChatGPT, but I

wrote the source. Yeah. Well, I think like that's in the end, I think it will often goes back to like the importance of the understanding of the context. And I think that's really is true for AI as as it's been for us as payroll scientists before.

And yeah, I guess to wrap up, it's such a gift to have you on the podcast and maybe it like as a little bit of a thank you as always. I, I was going to message you this, but I had kind of randomly this experience going to an office for a client. I was going to start working with a kind of digital health company and the product manager, he welcomed me. And then in the, in his office, he's like, I've, I've read this payable science book.

Do you know about it? And he showed me your book and I was like, oh, thank God he's showing me that book because there's so many books that you could have shown me and. I'll be like rolling my. You would have just been like, oh. Yeah, but I was like, OK, we're going to be in good place. Let's get. Started. Yeah, I guess thank you for all the work you've been doing and recently now also sharing the work with AI and so on as well. And thanks for coming on today.

It's really fun to shut. Yeah, and I, I have to thank both of you too. I can't tell you how many people I refer to have it weekly. I just think it's one of the absolute best resources out there and especially for people who want to learn more, it's so accessible. And you know, one of the few things that I got where I am always clicking links. So thank you for the service that you both do and I'm honored to be able to be a small part of it. And that's a wrap.

You've been listening to the Behavioral Design Podcast, brought to you by Habit Weekly and Nuanced Behavior. Sam and Elaine tell me this season is packed with incredible insights about behavioral design and AI, so be sure to subscribe and share the podcast with your friends. Though you might want to keep it away from your enemies. In case you haven't noticed, I'm an AI voice. Yep, pretty crazy. Quite the improvement since last season's AI outro, don't you think?

If you'd like to collaborate with us at Nuance Behavior, where we use behavioral design to craft digital products with Nuance, e-mail us at hello@nuancebehavior.com or book a call directly on our website, nuancebehavior.com. A special thanks to the amazing Dave Pizarro for our show music, and to Mei Chen Yap and April English for their help in producing and publishing this episode.

Thanks again for tuning in. We'll be back soon with another exciting conversation where behavioral design and AI intersect. Happens to Mugatroid. Oh. It's been hard. The Dunning Kruger effect is real, y'all. I'm living it.