How'd you like to listen to dot net rocks with no ads? Easy? Become a patron for just five dollars a month. You get access to a private RSS feed where all the shows have no ads. Twenty dollars a month. We'll get you that and a special dot net Rocks patron mug. Sign up now at Patreon dot dot net rocks dot com. Hey Carlin Richard here. As you may have heard, NDC is back offering their incredible in person conferences around the world. DC Porto is happening October sixteenth through the twentieth.
Go to dc Porto dot com to register and check out the full lineup of conferences at NDC conferences dot com. Hey, guess what it's time for dot net rocks Danish style. Yeah, in an airstream, I mean of the most American of American trailers. You can imagine an are in an airstream. So this I guess it was n dc Copenhagen at one point, but they rebranded it dev Festival. Yeah. Yeah, and it's very festivally tense. It is. They have the Boston Dynamics robot here, one of them.
Yeah, it looks like a dog and it walks around, looks around. It's got the arm on it. Yeah. It doesn't do anything threatening though, like you know the one. It doesn't even backflip. Like I said, you can to do a backflip. He's like, no, no, that's Stlas. But it's very Uh, it's very cool. We talked about not making it non threatening too, because apparently they've now made Atlas smaller, like instead of being human heights, it's quite short, and I think it's
all part about making people comfortable with these machines. Yeah, because they are creepy and they're hard and heavy. Yeah. I picked up the battery for that little spot robot. It it's it's a good ten pounds. Yeah, yeah, it's crazy. Anyway, great show. Really excited to be here. Yeah, very very excited to be here. Grishmo Jenna is here and we're gonna be talking with her in a little bit, but first it's better no framework roll it all right? Man? What do you got? Was
something that Joel helan Uh told me about. It's a it's a it's a language called wing and you can get it at wing lang dot io, a programming language for the cloud. Kind of reminds me of Pollumi a little bit. Oh, I'm not really sure how it differs, but it combines infrastructure and runtime code in one language, enabling and I'm reading here enabling developers to stay in their creative flow and to deliver better software, faster and more securely.
And just looking on it, you know, here, here's a little sample clip. Let bucket equal new cloud dot bucket Grand brand. That seems pretty powerful, right, Yeah, bucket dot put so very cool, And there's a local simulator to stay in your creative flow with minimal context switching and immediate feedback. Like I said, it looks good. I haven't run it, but you know, Joel seemed to like it and it looks pretty cool.
Yeah, that's awesome. And there is a video there, so if you want to just watch the YouTube video, you'll probably get a lot more information about it. Wing lang dot io go get it. Who's talking to us? Richard grabbed a comment off a show eighteen twenty two, the one we did with Billy Hollis back in December talking about high level design, and of course Billy's one of our great thinkers in the UX space, and there's lots of good comments on the show. This particular one is from par nine
one one. I don't know what that means. Is that an x or Twitter account? Possibly? Yeah, it didn't map to one, but he says I always love a conversation to discusses workflow and design. As a developer who's heavily into windforms development, ensuring design compliments workflows and vice versa is important and the hardest part of developing a solution. One of the best ways I've worked on on how to improve this is to actually do the role of the
solution is designed to assist. He's shouldn't find repetitive tasks that you can automate, So I think what he's really talking about here is do the work, like what is the workflow? Where can way through it? And the UX can kind of be dictated from that. So spending your time with the app, not just thinking about it abstractly. Yeah. Yeah, no, substitute for that fewest number of clicks, use numbers of change of hand position like to the mouse and to back again. All those kinds of things make a
better UI. Yeah, it's always the same. Right. You talk to the customer and you say, if you could write this program, like, how would you want to operate it start Where would you want to start? Where would you want to go? What are the generals? What are the details? And how do you want to get through that? And often people don't believe you. They like they think that they're stuck in a box of a form or this or that. But you say, you know, if
this was a car, how would you drive it? Yeah, Chrishma is shaking her head, nodding her head. Yes, she's been there, no question. And you know that part of that is watching somebody work, like, look, how do you normally do this? And all? And you realize those little jumps outs even after you've built the app, going back and watching people use it, you know, a few months later going wow, that's not what I always think. Yeah, right, hey, par thank
you so much. Here a comment at a copy of music Coby. It is on its way to you. And if you'd like a copy of music Coby, I read a comment on the website at dot net rocks dot com or on the facebooks. We publish every show there and if you comment Darren, we're reading the show. We'll say your copy music cobuy. And you can definitely follow us on Twitter or x as it's called now, but that's
fine, but the real fund is happening over on Mastodon. I met Carl Franklin at tech Hub dot social and I'm Rich Campbell at mastodon dot social. Send us too, and you know you could send us a comment by two if you like, you can and we will respond. We will respond, We read them all. Okay, let me introduce Chris Chrishma Jenna is a data scientist with the UX Insights Team at IBM in San Francisco. I should just stop there, because that's amazing. That's awesome, the full amazing full
stop right. I mean, you're the only data scientist in the org. She supports eighty plus user researchers and designers and uses data to understand user struggles and opportunities to enhance product experiences. She's delivered fifty plus talks and workshops at multiple conferences around the globe, including piicon US. Grishma is extremely passionate about encouraging, mentoring, and empowering people, especially women and students, in the
world of technology. She's been recognized as a Fellow by the Python Software Foundation and also serves on advisory and leadership boards for nonprofits and other organizations. In other words, a whole lot of awesome. Thanks for going with us, Thank you so much for inviting me. Yeah, what did you think about the comment that Richard read there? Very interesting? I think right about the workflows and just having the usocentric mindset, so that something we focus a lot
at IBEM as well. You must have heard that IBM's one of the frontiers and on the frontier of the design. So they came up with this Enterprise Design thinking framework, which is all about making the process user centric, coming up with something called an empathy map, which tries to put yourself in the use of the users and be like, Okay, what is this user thinking,
what is the feeling? What is this user doing, and what is the current you know, state the ass scenario and then hopefully going to the TV which is the future scenario where there are hopefully no use of frustrations or struggles and it's a very seamless end to end EXPERI yeah, because of course my instinct as soon need to say what is user feeling? It's like I'm expecting frustrated lunch. Absolutely where does the data science fall into this? Because
that's what grabbed me, was this idea. Yeah, you know, so much of UX design seems intuitive based rather than is there a data approach to us? A science there is? Yeah, I think this is definitely something that's a little novel And a lot of people ask me, Okay, you're a data scientist, but on the UX side, isn't that what's happening, what's what's going on? Like, shouldn't you be on this quantitative side with the logical, practical thinking. Yeah, and this is this is definitely a
very unusual role. I will say it's not a stereotypical role. But where data false is that data is the language that customers used to talk to about. So I think there's a huge, huge potential there to actually mind all of that data, all of that feedback that customers are giving us, whether that's quantitatively through the instrumentation of the products of qualitatively, well you know, they're on a customers about called saying yes, office sucks, I'm going to
go to the next competitor. You know, I could have done this with Excel. Yeah, my brother could do this in access. You're you're reminding me of Mark Miller's Science of Great UI talk where you know, the first time we talked to him about what you mean science of UI, Like you said, it seems like into would be taking over, but he talked about you know, there are certain things about the way your eye sees things and the way your brain reacts, and and that can be other way. Contrast
is perceived as importance. The things that are higher contrasts tend to be more important than lower contrasts, And so that can translate directly into rules that you can use for a better user interface, a better UX experience. And so that's exactly what I thought of when you said, you know your data science around UX. There's a lot to it. There's a lot to it.
But I also see we're good. We're getting good at instrumenting stuff now, the open telemetriies and things, the lists, like we have the tremendous amount of data we can pull from people using the app, not even talking about the information of putting into it, but how they interact with the interface, And I wonder how well we can use that data. Like I've worked on a bunch of that, and we end up with doing heat maps where it's like, oh, their mouse is always over here, and it's it's rarely
over there that always do you go to eye track. I haven't dealt with eye tracking data myself, but yes, I know that some of the teams use the eye tracking data. But again, like you mentioned, right Richard, with a lot of instrumentation data, where are the clicks happening, where's the mouse activity happening? How much are they strolling through the page? Are they skipping over some demos because they're like, yeah, yeah, I already know this stuff, like you know, just just get me to the good
stuff. I know all of this basic stuff. But I would say it also really comes down to two things. Number one is what data are you exactly capturing? How are those metrics defined? Right? Because what might be a conversion for my product or for my perspective might not be a conversion according
to you. So I think just having that level of standardization of Okay, this is how we're going to track a log in, this is where you know, this particular workflow starts, this is how this particular workflow ends. What happens if there's some drop off in the middle, or you know what conversion means someone who gets to the end of a work yes, exactly right, So whether that's a purchase order going through or somebody putting you know,
starting a workflow again? Now does that mean do you start a workflow? Doesn't mean ending the workflow? Where do you exactly when a motion points? I mean we usually think about conversions from an e commerce perspective, Yeah, but I love the idea of applying that to an internal business workflow. That'd be a great number to have. I've never thought about having. Is how many times have people started down a form? Yeah, and then a band
abandoned it? Yeah? And if you could see a pattern that data is that they consistently abandoned here, yes, which might be they need additional piece of information that interrupts them enough they have to leave the app to do that.
Like now you've clearly got a feature waiting to be built. Absolutely, And I think the other thing is also the amount of data are capturing right that that plays a huge role because unfortunately I've seen too many of these instances, especially now given that AI and Jenny I chat GPTY is pretty much do you know all people are talking about executives coming and say, Okay, we're going to build this genitive AI model with deep learning in machine learning and all
of that. But when you go to look at the data. There isn't a lot of data. There is not even good quality data available to do those modelings. So I think having some realistic expectations really help speed up the process. It also strikes me, especially when talking about the large language models and even the general mL stuff, that's a pretty big hammer for what is not that big a problem if you're thinking about it, well, like I
just wonder. I mean, it's cool, it's cool hammer, but it's like, really do you want to That's like it's almost like, hey, we just rewrote this and Ruby on rails, our problems go away. But the more important thing is like you're going to leave me alone for six months while I do that, Like if you pull out the mL hammer, you're
really just getting a three month pass. Yeah, when actually studying the data meaningfully could get you to a solutions faster and simpler yea, or at a minimum, give you better inputs for that machine learning model in the end. Anyway, Yes, completely, I agree. I heard this. You know. An ology that someone used is that machine learning or deep lunning models are often the ferraris when you might even need like a bike to I'm talking a
pickup truck, a two wheeler will do without a motor. Yeah, so I mean that I've found these days with telemetry specifically, it's either way too little and you never you can't get anything useful lot of it, or it's so much it's just unmanageable. Yeah, how do you try and thin the data out? There? Is there a tooling approach? I mean, if it takes too little basimbly, nothing we can do. So we kind of
got to go with it too much. So now what Yeah, I think the kind of approach I leaned towards is start with things you are absolutely confident about. We become up with two metrics that you say, okay, the conversion rate and you know, maybe the number number of logins per month or
you know something like that. Just come with two metrics that you're absolutely confident and that you can manually check that, Okay, what's being instrumented, what's being captured, is what we are expecting, and then you feel confident. Maybe keep building onto that sect thing through a bunch of obvious and errors for a log in, yeah right, the speil log and failed log at an incomplete loggin. I don't think it's more than that. And now can we
see them all can we now see a number behind that? Or would you start with the happy path? Depends depends. So the way I like to do it is that let's pick up two metrics in the instrumentation phase that we are confident to it that we can check. Once we have that in plates
lists, start thinking about what future questions we would want to answer. So I might not have the data today, but three months down the line, I want to see how many users are abandoning on a particular workflow, right, are converting to a particular workflow, all right, and then start those that reverse engineering of okay, to be able to answer that question. The kind of data I would need is the abandons, maybe the Paige speed load.
Maybe you know how many demos they are watching, how much time they are spending on this particular product, And then go back and see what other metrics that you could come up with actually start tracking all of that data.
Now, when you're talking about data science and in this I know you mentioned AI a little bit passively, but do you use predictive analytics on this data to try to figure things out or any other kind of you know AI if you will, Yeah, yeah, okay, So a bit of a disclaimer, I'm not really sure where the lines between artificial intelligence and data science, you know, blur, but I always feel yeah, I mean data science is really at the very very lowest level of AI. It is all statistics,
it's data sensors, machine learning. So just for the purposes of our conversation, I'm going to interchangeable use or just maybe stick. Do you know data sciensing machine learning? Yes, Predictive analytics is definitely something very popular. I think one of the most most abundantly available use cases is predicting if a customer is about to leave your product, So it's basically predicting customer attrition. So you can maybe start looking at you know, has their overall usage of
the product decreased? Have they opened a lot of support tickets recently? Maybe those support tickets have not given resolutions to them right, or maybe the average time taken to handle its support ticket is increasing. You also have a lot of feedback surveys that they give, right like how happy are you with this experience? Would you recommend our product to a friend or a colleague, and
then just looking at the trend of those goals. So that is definitely one case where you know the predictive models can help and alert the product team that here this customers at a high risk of actually, you know, terminating the contract. Yeah, so so better go ahead and maybe have like a touch point of okay before the customers actually talking about canceling exactly. The idea and the idea that you would contact them and say, hey, you've been having
a time can we help you? Like, what can I do? Extra? Yeah? Why did you post this on Twitter's angry Sweet? Yeah? Just call support? Yeah, you know I have. I think I talk about this the show that Bloody Dishwasher, the Melee. Yeah. I kept breaking Yeah, and I couldn't get regular support. Regular sport wouldn't respond to me four weeks. But if I tweeted and tweeted yea that day. Yeah.
And at one point I'm like, listen, I'm not that mean a guy, right, I just didn't get anywhere I would Is there a better way to do this? And the and the person on the other end's like, literally, nope, this works, just use it. I'm I'm in that limbo period right now with the HVAC system in my studio. Oh man, yeah, they's a new one. Well they I don't want to take
up too much time with this story. But they sold me a twenty four thousand BTU unit for a six hundred and eighty square foot room that's insulated to our twenty R fifty R fifty okay, really freaking insulated. Yeah. Yeah, you don't have to pump very much cold air in there. Yeah. So it cools off in about five seconds with this thing, and then it stops, and so the humidity rises to sixty five seventy percent, Yeah,
and it gets unpleasant. And so I had to beg with them to you know, get somebody out here to fix this, and they finally decided to get a smaller unit in there, but then they wanted to charge me more. That's like, wait a second, you screwed up, and you want to charge me more. Yeah, And so now we're going we didn't get anywhere with their subcontractor, and now we're going to the actual contractor, who company who we bought it from, right and there, and they are unresponsive
right now. And if they come back with I'm sorry, you're gonna have to pay it, then then I'm going to unleash the kracking. Yeah, do the thing, yep, yep, you just need to send a tweet to them, right, if there's to start, but begin the public humiliation, right, let it begin. Well, we'll keep you posted on that.
Yeah, no kidding, but I mean you still your language to me strikes as external customers reacting to product, right, I mean, and that's fine, Like those are all good things, and it is interesting to synthesize data from your tech support side as well as perhaps a sales channel. Maybe you're doing some monitoring at the social media level, to say, do we have some sentiment analysis going on? Coordinate this information, but then to add
the utilization of the software. Yeah, Like this is not just a person being randomly grumpy, right, it's this is where they're falling off on the workflow and they do that consistently. Like, this is not a whining person. They're trying to get it done and they're abandoning for whatever reason. Like that that is such actionable data. I am you are watching this customer try, Yeah, Like shouldn't you do something about that? Yeah? Yeah?
And I think one case where data science comes in really helpful is that maybe as a front front facing you know, you're facing the client and you're having these conversations, maybe it's a product manage. Maybe it's a develops having a conversation with a very frustrated customer, right saying, Okay, you know this isn't working the way it's expected. I'm not going to do a bunch of
issues. And that's a data point that you have. But when you look at the data perspective or quant perspective it you get that scale, right, So now you start to notice this is not just one isolated customer, it's actually ten customers, hundred customers, right, thousands of customers that are doing that. And I think that helps to also remove that bias, right, which it could work in both ways, not just that customers the pain exactly.
I guess the next question you asked when you talk about like a failed workflow like gus, well, how many succeed? Yeah? How often do we succeed? How many other abandons do we have? I really like that you take the personality element out of it to some degree, and it's like, here's a set of data and you have found with someone willing to communicate with you about the problem, don't make them the enemy. Here's all these
people that aren't communicating about this problem but are having the same problems. Yeah, exactly, yeah, and it works two ways, right, Like the user researcher could maybe be in a conversation and then notice that, okay, the customer is struggling with the product, and then they go to the data science team or the analytics team and say, okay, we saw this one person. Are there other people who are struggling with this? And then you
notice that you know, this effect is that scale. And another thing it also does is like often product managers or you know, a client facing teams here from really important like it's called the hippo the highest paid person's opinion, and it's usually the highest paid customer, like we hate this feature. When are you in unveiling? You know, XOY is that feature and they're just going to be you know, maximum attention to them because that's where the money
is exactly, all right. But then when you combine it with the data, you can actually realize that is this person's problem? You know, actually yes, is it unique to the customer or is it a little more pervasive where other customers are also facing this? And then it's probably a future worth
including on the road. Oh man, now you've now you make me feel like it's a superpower, like if you applying data science to all of these kinds of problems so that you're always talking from more than the immediate issue in front of you. Exactly. My automatic reaction to a person's having a problem with this feature is to be able to apply some data analytics, say how
many other people? How many six? What are the other scenarios? Like, I mean, even think aboufter our past scenario where you've got a customer talk and now you've found out this other customers failing, Like lots of people exclude themselves from serving, but they don't want to do that. But to call that customer say hey, you know, I'm from such a such a company and we'll see you're struggling with this thing and we're trying to address it.
Would you help us to understand what you're trying to do, Like, I think you'd get good response from that. Yeah, I mean, you're going to hum the fine line of being creepy. You know, we've been watching you and we see you're not having a good time. Better to say that you're a hired psychic. You've discovered that they're the spirits. Have told me about your unsatisfaction. But I know. I do think it's an interesting part of this is to just have that reflex of let's can we quickly gather
more data, like do you see an opportunity to build up tooling? Like I'd really want to make that mechanism simple. I don't want to have to call you Krishma as the expert to say, can you give me analysis on this? Yeah, I wanted to be a button the moment it pops up. What is the IBM system that you talked about earlier, You're sort sort
of hinted at it. What is that play? Yeah? I think we have a bunch of different tooling and framework and to be honest, one of the issues we're facing a standardization because just the software unit that I work with, we have around eighty two hundred different products and all of them have absolutely varying degrees of data maturity, right from like we haven't really put this product into production, or we only have like two clients, all the way to
we have thousands plus, you know, users every month, and this is just a huge amount of data. So I think it's it's hard to generalize what that looks like, which is why where I need to have a personalized approach for the team I'm dealing with, the product, I'm dealing with, what is the exact objective or the problem statement that we're trying to find automation.
That's that's very interesting because that is definitely something that I'm trying to do as well as unit I've seen a lot of other company needs do as well. Is this concept of anomally detection, right, right, So throwing all of the data into a model that will help you understand if a particular user activity or a particular user is you know, I'm showing up as an anomaly. They're doing something different, they're doing something unexpected, maybe they're running into
a bunch of issues or errors. And then that also goes into with incident management. So a lot of security teams do that to see it, right, maybe there's a DDoS attack happening, that's anomally for you, so that's an incident, right, So using kind of the same principles, So that
is definitely one way of automating. But again that only comes into play when you have a highly mature team in terms of the instrumentation, in terms of the data, in terms of understanding how many users you have, yeah, and what those scenarios look like like it does feel like something I'd want to build into a tech support client. So not only am I pulling up the anomaly, but then provides analytics run the anomally say this is the four hundred
times this has happened this week. Yeah, right, and with this many different customers, just to give a sense of urgency and scope to that. Like I said, I love the idea of being able to resist the hippo because they're asking for a fitture that's distinct to them. The majority of customers are fine. Yeah, And at the same time, when you have a problem, show up to see, okay, well this is unique to that
customer. I can approach it differently. And I'm not saying like ignore it but so much, but recognize it's an unusual work Yeah, as of course to it's a usual workflow that often fails, and maybe you can get a shape for it. But also you know, maybe it fails ten percent of the time and ninety percent time it succeeds. Like now you've got a different shape of how you're going to approach it. Looking at the problem, it feels like it's time to take a break, So we'll be back after these
very important messages. Hey, we're back. It's Carl Franklin. This is Richard Campbell, and we're here with Grish Magenna and we're talking data science and UX and I'm very fascinated by this UX sort of a science framework that IBM has. Well that is there any one of those things that you would use at the onset of a project to help design the US or are they sort of things that you work with after the fact, Like this last example you
gave, seems like you have a mature team and a mature project. Yeah. Yeah, I think again going back to like at the start, it's more of let's start thinking about what are the questions you want answer and what are the metrics you need to track, So it's probably at the earliest stage of data collection and just deciding what those metrics should look like. I think one thing I am trying to do as well is when I'm working with these
mature teams and royalizing there are sort of inconsistencies with the data. I'm actually trying to create this guide of Hey, I worked with this team that had great data maturity, but we still had inconstituencies, we still had issues figuring out, you know, what was going on. So as a heads up to you before you even start instrumenting, learn from our mistakes, take take our failures into account, and here are the recommendations that would make you know.
So that is definitely some sort of you know, activity that I'm trying to do earlier in the pipeline. But yeah, I think, I mean, that's that's all you can do, right because you can speculate. But at the end of the day, when the data starts full flowing, and that's when you can actually do some expiation and and that it's an understand is this good enough? Is this good quality data? Does this make sense?
Or do you maybe need to you know, modify a few things. And do you have a sort of general knowledge database of things that you've gleaned from projects working? You know, when somebody says, you know, customer says that we're thinking of doing it like this, and you say, well, you know, based on our experience, that approach hasn't really worked well and years why, like, do you have this sort of general knowledge that you're
gathering that you used too? Yeah, I do have it unformally, but formally I think this reminds me of what is my primary responsibility is to create this insights repository of sorts, which is where all the user researchers are coming in and they have findings, and more often than not, all of these
findings are stowed away and like you know, drives and folded. And let's say somebody leaves the company two years later, product manages like, oh that thing was done two months, two years ago, where is the token? I find? You need to talk to ten people? So we're definitely about knowledge management. Yeah, it would be really cool to upload that to a chat chypt like things that you could ask it questions directly. So what do you think about this? Well based on past experience, Yeah, exactly.
But then also am I other risk of automating my job away? I don't know. I mean I've always had the experience that there's we're never getting to the bottom or to do list anyway. So knocking automating stuff, you're just going to do more faster, We're just going to do more things. But you have worked with some interesting customers I saw. I haven't seen the top, but I've seen the abstract for the top where you talked about folks like Airbnb, and Spotify and so forth, like can you dig into any of
those scenarios. I don't want to get any any trouble but here, but it's like that just sounds cool. Yeah, nom, So those are those are case studies. I wish I could claim that those are the projects that I haven't done. Those are case studies, though, but very interesting ones. So I think Airbnb and Spotify does a really good job of having their ux research teams, design teams and their data science teams work in very tight
conjunction with each other. Yeah. So, like I mentioned, right, you could have a user researchers that says, Okay, this customer, this users having something odd, let's go to data scientists. But conversely, you could have data scientists are looking at that aggregated level of data and saying, hmm, this is kind of unexpected. We didn't really expect such low conversions. To check the metrics. The web pages are working fine, the instrumentation
is okay, what's happening? Can you use a researcher go and talk to the customer and try to find out what's exactly going on. That's where the quantitative and the qualitative aspects of it. This is this is the title from your child, right, the US data analysts they should be friends, Yes, they should be best friends friends. Okay, well, because you get this problem with ux of it only being intuitive and a dodo. Yeah to the data science folks are going to come in, are going to help you
beef up your case for work absolutely and vice versa. There the data folks are going to see these anomalies in the arc, like the larger data sets, and be able to come back with have you looked at and maybe turn up some other issues exactly exactly, And I think especially as more companies and executives go into this data driven decision model, right, it really helps to
have that evidence. So you can say, my instincts or my intuition based on all the research I've done, say that we should, you know, modify the workflow, we should introduce this feature. But here's the data to back it up. We spoke to our users or we have looked at instrumentation data. So here's the evidence that I can you know, help prove across a point with Because anytime you change a working data flow, you're impairing its
productivity for some period of time. Like people know how to use this, so the improvement has to be big enough that thing initial impact of the change is offset by the net long term net benefit. I guess an interesting ROI question. It's like, hey, we're not going to change this because it affects so many users and the inkroll improvement doesn't appear to be substantial enough. That'd be an interesting place, like I don't know if any organizations I've ever
dealt with where they were that mature. Thank you. See, this is a design improvement, but it's not a big enough design improvement. So I think we're going to do it and until we can find something large enough that it's worth disrupting the workflow flow yep, yep. And even then, I
think communication is really important. You need to tell the users here, this is our latest updated version, and this is where now this menu has actually gone over to the right side, and these are the new features that have been on wheals. So I think that communication and almost change management, is it really important so that the expectations of the users are you know, kept
kept in check. Yeah sure, and yeah, I certainly. I talked to folks who have client fatigue that stuff, especially in the SaaS products and cloud stuff. It changes too often, right, And I spend enough time on the sytems. It's like I get tickets all the time, and I can see the wave of oh they updated, decline again this week, and I'm getting all these tickets like somebody move my cheese? Right, and you're back to what the heck happened? How is it different? Here's how we
deal with it. Talk to you next week. Yeah, and that This is another case where data science comes in really handy, because then you can start doing AB testing right and actually look at the data. Rights this change actually useful? Does this make sense to users? Or are the users are frustrated and they just keep going back to like, you know, take me to the old layout, take me to the old format? Can I turn
it off? The first question? Every time? And I love you know, AB testing isn't always obvious, right, Like how do you know which to pick A or B? Like? Why was it better? Yeah? You know, Okay, you serve them fifty fifty percent of the time. Like what's the metric that shows it was a superior outcome? Higher completion rate? I guess would be the one? Yes, yes, definitely. I think you also need to look at the demographics of users, right, did
you do the branching based on where the users are based? And if so, were there any cultural differences demographical differences that contributed to that? So I think just it's definitely a very complex topic. How do you have those confounding variables? How do you make sure they're bearable to attracking, are independent of each other, or at least as independent as they could an experimental sets. Now you also have to look at things like is there a geographic event happening
right now? An earthquake, a flood, a heat wave, not that that would ever happen, Nah, a forest fire? Looking at you, Richard. Yeah, this comes down to really, how do you how do you separate A and B? Like? Why? Because my instinct as a developer is just gonna make it random. But nothing's actually a random So what do you mean when you say random? So, I mean, at a minimum, if you're thinking webish here, I've got a cookie, I'm going to stick a given cookie to A, and the next new cookie I create,
I'm going to stick them to B. Yes? Is that sufficiently? It's not random at all? Actually, it's a cyclical measure. You haven't I mean, you're ignoring all the other factors that good enough. I don't know me the data can tell yeah, yeah, how much do we know about that cookie to actually say well this is who got it? What was
a separation on them? Yeah, I don't know that I've ever looked at the data that way, Like Ashley dug In and said, did we make a good A B test from a client selection perspective before we say hey A out A outcompleted B therefore A is better? Not even making me think too much, christ Man, It's all about the biases, right, selection bias, confirmation bias, recency bias. Everything I've thought about it is a lie.
Yeah, well even in fact that I'm just selecting new cookies so my existing users don't see it at all, Right, like that that could be in effect as well. So many opportunities to lie, yeah, or to just or to select a truth yes in advance ways to filter? Do you ever have to tackle those kinds of problems like how are we going to separate to request these request streams to make sure or we get a good AB mix?
Like what does that even look like? I don't think I've actually done a lot of AB testing on this current role, not not so much. But yeah, again, I think I'm challenging more with that volume. And you know, data quality is shoes right now. So hopefully in the future
I'm hoping to get to those good, nicer problems to have. But right now it's a lot about education and awareness about how the instrumentation needs to happen and what other things that's possible, because oftentimes it's also just the black box perspective of AI, right, like, oh, it can do anything and everything, but we know it's not true to start from. I'm also thinking from the friends perspective of often it's just a developer decided to do that.
We've decided to do an ab testing group of developers there, it's like you should be calling your data science people at that moment to talk about how we're going to discriminate these Yeah, let those experts get into the play and even know that they're doing that, because that can give us a make the get a better result if we work at it a bit. Exactly. Yeah, And I think even having these multicultural, diverse teams all hell because there's a
lot of communication messaging involved. For example, I recently came across this website that says that each color has very different connotations culturally, like read in some cases might mean danger or something negative, but in other cases it's actually a sign of prosperity, right, well, and then the Chinese us right, yes, exactly exa and the Chinese dragon is a very powerful positive thing, and European culture it's a negative, right yeah. So yeah, where you're
using that stuff matters as well. And certainly words, right, I mean it's some words that translate to other languages to mean embarrassing things. Yes, we see that over and over again. It's the famous Nova story, right, yeah, Nova, it doesn't go ye no, Chevy Nova, don't call a card that in a Spanish speaking control mistake, or the slogan Coke ads life when translated to Japanese. I think it was our Chinese one of
the languages Coke will bring your ancestors back from the dead. Yeah. I think the most recent one was the video came out with their latest emmal model or one of those framebooks, and basically translated to Spanish, it meant bottoms. That's why you see like all these nonsense words as names of products totally like that, because they're just they don't exist otherwise boid. It helps, like, for example, work just Fine, which is the sound beforehead slap,
also is the name of my production company. So you know, we've been digging in purely on the telemetry side, all of this very quantitative data, but there's a whole qualitative side, right Doing hot surveying well, I've come to appreciate incredibly hard to do. But as soon as you you know, you want people to type something or write something, now you got to
analyze it all, like what do you do? Oh? Use researchers are actually pretty good at this, but unfortunately it's in manual efforts, so scaling it up can be very time consuming for sure. That's another place where I've kind of loved this use case is to help analyze surveys or reviews or you
know, Twitter complaints or whatever social media mentions that you have. So what I like to do is, you know, run all of these through Python scripts that I've developed and then try to understand what are those top teams,
keywords and sentiments coming up. And while I will be the first to admit it's never going to be a hundred percent perfect, it's usually a really good starting point that then I can handle over to the user researchers and they'll get a lay of the land almost like Okay, you know, there are a lot of mentioning about us, or there's a lot of negative sentiment attached to
this new version that we had. Let me go in and manually dig in a bit and see, you know, what are those more insightful things that I can come up with. So that's another you know, symbaetic relationship we have. It seems like a multiple choice survey kind of thing is the way a lot of people do that kind of stuff. But I mean there's there's problems with that too, right, Like what if you don't provide the right option, you know, and then you have other and now you give them
an Irish text. Now you're back to text again. So what are some of the other problems with that approach, the multiple choice survey kind of thing.
I think there's always the order that the options are presented in, right, especially if you're not really paying attention to the survey and you're just like it's like the first option and just you know, I want to quickly complete the survey, So that that is definitely a bias that can creep in as well, and if people just don't want to do it, they will just pick random things exactly exactly. Yeah, I think language really the way you
write that option it is left to it's subject to interpretation. Right. What I might interpret as option it might not be what others interpret as OPTIONE. So having that, I always like having those open ended text responses because then
the users can add in a lot more information. They can also give you a context that hey, you know, it makes sense if I'm doing X, y Z. But actually I would like to choose option B on a regular basis because I do another workflow way more often than I do option A. Brian McKay works for a company Roster, that does AI analysis of text of comments that people leave and determines whether they're relevant or not and actually turns them into data. And they're using GPT. Yes, yes, that is
That is definitely a very interesting use cases. How do you identify if one this topic or this comment is relevant and two is this actionable? Right? Even if it was relevant, you could just have the comments saying this product is awesome, I love it, or this product sucks I hate it, but there's nothing really actionable from there. And oftentimes some of the best. Some of the best ideas for the products come from the users themselves because they
are at the center of this. They use this day in and day out, and they are probably the ones that are going to say, hey, have you thought about this feature? Or your competitor already has this feature? You know, I would love to see this coming in here. Sort these comments by actionability. Yes, that's a cool concept, and there you like, that's I really wanted to get into this. Where would I use machine learning? What is the right thing to do? That's something? Sort these
by action ability? How do you get the tool to assess that? But I guess this is what LM's in theory or could app Yeah, yeah, yeah, given given the right domain knowledge, because I think a lot of this, you know, enterprise products especially have so much technical juggy. Yeah. Domin knowledge is really really important, right, because maybe what means workflow to us, like in general language, might be a very very specific activity
that's happened. Defining addiction what do we call it? A glossary of vocabulary right at the front of a project makes so much sense. I got bit by this once. I had a customer that we developed an app for. I'm not going to say who it was. But at the end of it, they basically said, well, where's the the feedback page. I think it was feedback or maybe it was a comment screen, right, and I'm thinking we did a fill out form, right, But what they meant by
a comment screen was a live chat. And they were like, oh no, this we said this in the steck, we want to comment screen. I'm like, yeah, but that definition was never defined anywa. To them, it meant a live chat, which hack of a lot more software, a lot more than writing a comment and having a store exactly. Yeah, yeah, surprise, yeah, yeah. So when you say comment screen,
what do you mean exactly? And it's it's one of those things that we all take for granted because you know, we thought comment everybody knows what that means. You know, you leave a comment or an it's a contact page, a contact page, contact page. So when I say contact page to you, it's fill in this form. We'll get back to your later, that's right, not a live chat. Yeah. So, hey, it
pays to go over every fine detail when we're doing that. Well, I'm picking back to our comment at the top of the show with par about you know, watching people in the workflow, are actually going through the workflow. Yeah, Like if you've done that and you've gotten to the contact, yeah, it would have been and now we chat back and forth. You mean
chat screen. Yeah, Oh well it's a different thing entirely. I certainly on my radar to do more shows around building a large language model sort of domain specific, because that seems to be we don't need existential conversation with software. The generalized ones are amusing. We can debate the value of search on that, but the idea that it understands a particular domain and can help organize large amounts of data. Effective summarizer, a prioritizer like those seem like very
practical implementations of this. Being able to sort comments into ones that are actionable. That's a huge amount of time that you could really nail down. It's just an interesting metric of is it relevant, you know, does it talk about a product in a way, and then you know those are all elements that come down to this concept of actionable ability. Yeah, but it does mean we're going to be training our own I'm hoping the tooling gets better.
Like it. This still seems hard to approach it, does it does? Yeah? And again it really goes back to the amount of data. How do you define actionability? Do you have a ground source of shoot that says, these are the comments that are actionable because you know, these keywords were used, or maybe they were really long comments, which again there's a correlation right to having longer comments has a lot of more information in there, versus
these are the ones that seem actionable or maybe aren't even relevant. I think this one time I was doing some reviews analysis and there were those automated you know spambots, you know, putting in really random advertisms. I'm like, no, that is not relevant at all, and I don't even know why it's in the data set. So I think that's definitely going to be an interesting thing to deal with. Yeah, yeah, interesting times and yeah,
definitely new tools emerging. Still, we're going to get better at this. All the more reason to buy pizza for the data team. Of course, I'm a big believer in you know, folk city together, and so anytime you can do this cross team, it's like, how do we help each other? Like you really got me thinking about the US team wanting to have access to more data before they act on those problems. I'd do the same
thing, but with barbecue that total yea and vice versa. That you'll hope the data analytics team that's looking at telemetry and so forth are I think routinely just saying hey, here's what we're seeing today, because often they may not even it may not occur to them that what they're seeing is unusual. Yeah, so they just show it to the US to periodically say the US folks,
here's how we see how your apps being used. Hope that fits with what your expectations were, because it might just kick off a conversation that finds something for you lose customers over it. Absolutely, and that toll triangulation of that qualitative plus quantitative or UX resocier or plus data scientists coming together and giving you those different plus spectives is so so powerful, right because it almost feels like as a data scientist or as UX resocial you're looking at a very small
part of the picture. But then when you start talking to different teams and you know more, get that vital perspective, that's where you can see the entire picture of oh, okay, this is what's actually happening with our users in the product. Awesome. Yeah, so what's next for you? Have you done your talk yet? No? I am busy all day on Friday.
That's what we wanted to do this on Wednesday. So because you've got a busy day on Friday, I looked at your schedules like, I don't know, did you lose a bet, Like you're working the whole day. Yes, I have no idea why that's been scheduled that way. But you know, you enjoy the conference for a while. I guess, I guess, yeah. I would rather get out of the way any other conference and then have more time. But well, it's Christmas. Has been great talking
to you. Thank you some excellent ideas. I really appreciate it. Thank you so much. All right, and we'll talk to you next time. I'm dot net What is it? Oh? Yeah? Dot net rock Nice. Dot Net Rocks is brought to you by Franklin's Net and produced by Pop Studios, a full service audio, video and post production facility located physically in New London, Connecticut, and of course in the cloud online at pwop dot
com. Visit our website at dt n et r o c ks dot com for RSS feeds, downloads, mobile apps, comments, and access to the full archives, going back to show number one, recorded in September two thousand and two, and make sure you check out our sponsors. They keep us in business. Now go write some code, see you next time. It is harder than my Texas