You're now experiencing data with Brian O'Neill. Experiencing Data explores how product managers, analytics leaders, data scientists, and executives are looking at design and user experience as a way to make their custom enterprise data products and analytics applications more useful, usable, and valuable. And now, here's your host, the founder and principal of Designing for Analytics, Brian O'Neill. I’m going to try to withhold my sarcasm here in how I deliver this today.
I was just thinking about maybe doing something a little bit opposite. So, I was trying to think about what kind of reasons would be out there that you should not invest in user experience design or user interface design. If you’re running an analytics SaaS company or an AI product, anything in these complex data tools and data product space, when should you not spend time and resources on design help?
UX design is inexpensive relative to the cost of most engineering and data science work, however many times, the perception is that design work is expensive. And if you’re considering an investment in design, there are probably lots of reasons out there that you know from this show and probably elsewhere as to why an investment in product and user experience design as well as user research can help you get ahead. So, work with me.
I’m going to flip this over today, and why and when are you not ready to work with a designer, right? What are the signs that maybe you’re going to waste your time, or it’s just not a good fit for your team? So, I want to share some of these ideas and also give you some counterpoints to these. So again, trying to look at this just from the other side of the coin, looking backwards, and I hope it’s helpful to you.
So, first one, you want short, easy wins in the form of small UI improvements or dashboard changes that a non-designer or an analyst or, you know, an engineer, can make quickly, and anything that takes more time feels like something that you just can’t afford to do. It’s just too expensive because it’s going to take too long, and you’re constantly racing to get, whatever, your product iterations out the door, or if you’re working on an internal tool to deliver something back to the stakeholder.
So, one of the counterpoints to this is lots of teams seem to have the money to build it right the second time, and not the first time. So, you have to decide, is it worth it to invest in that the first time, versus all the sunk cost and time that goes into doing it right the second time. And again, some of this depends on how important the product is.
Are you working on a revenue, you know, generating solution here where, you know, lengthy feature development has a penalty associated with it in the form of burning resource there, and also the sunk cost of you’ve got more engineering code to manage, and you’ve got more product debt, so you might have added complexity where complexity was not needed, and so that further contributes to the product degradation. It is true that most impactful design generally creates more work.
It creates more product work, and it creates more engineering work because humans are messy, and while there sometimes are these magic small GUI-type changes, almost every project has a collection of low-hanging fruit type fixes to it, as well as more strategic work. But it’s true, design will create more work. The question is the outcome versus output thinking. Which game are you playing?
Are you focused on delivering outcomes, or are you simply trying to push out features, outputs, deliverables, things like that, on a schedule? So yeah, if really you’re focused on just short, low-hanging fruit, easy wins, and you want to limit everything to just small GUI improvements, and think that just coming in and tweaking the UI is going to do it, you’re probably right. It’s not worth investing in hiring a designer, or hiring an outside firm, or anything like that.
You’re probably best just getting someone internal to do it, or hiring a contractor, or something like that, and giving them some assignment work. You’ll probably have just as good results with that. The second one, another reason not to invest in design, right? Well, you have a strong design taste yourself. You’ve talked to users before.
Maybe you’re head of the product, and you are spending a good amount of time with users—which is great—and maybe you’re pretty confident that you can perform all the UX research yourself, and probably do enough of the design to get something over to your engineering team, and they can then take your style guide, or design system, or whatever bootstrap GUI that you’re using, and apply that over to your low-fidelity work, and you’re basically rolling at that point.
So, let’s talk about this strong design taste yourself. One of the places this sometimes comes from is when we see design done well—particularly a really nice UI or user experience—that is a tool that we’ve been using ourselves, it’s easy to perceive, hey, I could do that because when you see the end of all the product journey that happened, that’s all you’re actually seeing. You’re not seeing any of the journey, and it probably should be feeling intuitive and relatable.
And so, this gives us the impression that I could have come up with that solution because it seems so obvious. There’s a bias here at play because getting to that point where it looks very simple and intuitive is often quite difficult to do. And I think this problem actually increases with data and technical tools, analytics solutions, things like this. It’s much harder to even get to the end of the journey, so I think you should check this bias.
It doesn’t mean that as a list—just because you’re listening to this show that I have an opinion about, whether you have a good design taste or not. You may have a really good design taste there, but I think you should have a healthy amount of skepticism when you see what you think good design work is as to whether you’re assuming that you can leapfrog to that place.
Most designers can’t just leapfrog to that space, but we have experience about where can shortcuts be taken, where can they not be taken, what’s this deep sense of empathy we have for somebody else, and wearing that hat at all times when we’re doing this work on behalf of clients and stakeholders and things like this.
So, just beware of this, but if you feel like you have this really strong design taste and you can handle it, and you feel like you’re getting really good results doing what you’re doing, by all means, continue doing that. There’s no reason to get outside help. Just realize, the destination that you’re looking at isn’t necessarily showing you what the journey looked like.
The third one is that GenAI is going to change everything, especially with LLMs, that can, you know, directly query and produce all these different one-off needs that a user could ever have, right? So, why do we need to get design help? I mean, this stuff is—like, basically, it’s a magic box; you type stuff in, and it just comes up with the answers, right?
Well, I think most people listening to this show you probably know this is an extremely hard design and data challenge as, you know, an LLMs interface may seem to theoretically support infinite queries, it’s a blank box inviting just about anything to be typed into it, even if your model, you’re using a small… language model that’s been trained on a very specific data set to give very specific answers.
Another way to think about this is, you’ve exposed all of your model attributes to users for full control, like if you’re, say, a predictive analytics solution, or something like this.
And as such, you might feel like there isn’t really a need to make it more explainable because you’ve shown them all the features that go into the model, and maybe they can even play around with all of these, so there really shouldn’t be a need to get outside design help because you’ve made the model explainable or the application. The challenges on this side are that we’re assuming that exposing features actually gives end-users a sense of trust.
What this doesn’t account for are some of the human factors, right? You might be telling a very highly-trained specialized area—let’s say it’s medicine or something like this—and you’re saying that the most prominent features that go into the prediction are three different variables that this community of medical professionals has never seen before or seems to not correlate to anything that they’ve been trained on.
And these people were not involved in the process, you didn’t work with them to design the solution; you just found out that you were able to show a high degree of predictability based on the training data, and it’s based on these features.
And it may be right that the system has come up with some sort of medical discovery that the team doesn’t know about, but if we’re talking about producing an application or service or solution, a product that someone actually needs to use to get value out of, it doesn’t really matter how right you are with the technology. It doesn’t matter how good at the AI you are because you’re dead in the water from a product perspective.
And this is especially true if you’re building something that needs to generate revenue because there has to be a sense of trust and belief in the solution as well, and we’ve all seen the challenges of this with LLMs, in particular, with them hallucinating and making up all kinds of responses to things.
I also have this feeling of fighting with these LLMs when you’re trying to constantly retune and re-prompt in order to get it to produce an answer or a response that you want, and you’re unable to get it to respond in a way that makes you feel confident that it understood the query to begin with. And then you start to have all these questions about, is the answer not in there, or am I not prompting it correctly?
If you think that most of this is just an AI problem, then yeah, I would say, don’t bother to invest in design work. There may be times where simply getting the right answer and showing a high degree of predictability, for example, with you know, with predictive analytics, you may be doing well enough just getting a high degree of accuracy in the predictions. Only talking to your users will tell you the answer to this. I think this is a really difficult space to operate in right now.
When we’re not talking about probabilistic solutions anymore, this is just a really hard space. So, you have to decide, is it something worth investing in outside help with, or can you just keep doubling down on data quality, getting access to the right training sets, tuning the model, this type of thing. Number four, technical users leveraging data want to access the CSV and the raw data. Give me the Excel download.
They know how to use Excel or other tools, and they don’t want things dumbed down too much, and you might think that design is to come in and try to make it simple and dumb it down. Or maybe you see yourself as a platform provider. This is especially true for, like, self-service analytics, where there’s often this feeling that we need to give people access to the raw data.
The counter to this one about users not wanting it to be dumbed down too much because they’re very technical—maybe you’re serving the finance department or a scientific department, or maybe you’re a data platform providing data to analysts—this assumes that the goal of good UX work is to quote, “Dumb it down.” That’s actually not the way I see the goal of user experience and good design work, right? Design is about, at a minimum, making it useful and usable, if not also delightful.
And in order to do that, we need to understand the people that are going to use it, their tasks, what a current experience looks like, such that we can actually posit what a future outcome would look like. What would an improvement to this person’s life look like?
And if the answer is to simply keep dumbing things down, well that’s probably not the answer because there are indeed tools and solutions out there that need to be complex, they need to provide a powerful amount of flexibility for those users to be able to do their job, especially in an enterprise context.
So, if you have worked in the past with designers that keep wanting to overly simplifying things there, well, what’s happening there is they’re probably coming at this with their heart in the right place and this sense of empathy. They’re looking at this interface and seeing how complex it is and how it could be simplified.
The ultimate focus here should be on aligning the product with the needs of the people that you are trying to serve with it, which doesn’t always mean simply reducing tool fatigue. That’s often a big part of it, but it doesn’t mean you need to remove all your features and, you know, remove access to the download and all this kind of stuff.
Usually what’s happening there is, there’s a gap between expectation about what users want to do with the information—and I’m really talking about, like, more that the analytics application, dashboard, tooling, BI, these kinds of applications—the reason they’re asking for this a lot of times is there is an unmet need, and there’s a fear that the tool is never going to be able to provide all the possible views and experiences that these users want.
And my general feeling is a lot of this is, it’s a lack of exposure to what it’s like to be those customers, those users. We don’t know enough about how they do their work, and their feeling is, well, if I just have access to the raw data, I can get it. I can take it from here. And these are opportunities. There are opportunities here to figure out how to make the product better by spending more time with them at that point.
When they’re downloading these raw data files or requesting this kind of information, that is a huge goldmine of opportunity that UX research can help you with. So, going back to this idea of when is it not the right time? Well, if you really believe that hiring design is going to dumb down your interface in the wrong way—or your experience—and your tool is known for being complex, and super powerful, and it’s enterprise grade and all of that, then yeah, you should not.
The counterpoint being, that might be a problem in the type of designer that you’re bringing in, or a firm, or hire that you’re making because they may be oversimplifying the problem space, thinking that simply simplifying is the answer to all design and interface problems. Number five, you do things with Agile and you work iteratively, so you can always make the AI, or dashboard, or the product iteration better on the next pass.
That is a reason not to work with design because you’re working with Agile and you’re doing iterations. So Agile, and doing things the product way, and being outcome-focused, those two things do not necessarily go hand-in-hand. And almost every company that I’ve worked with that’s doing anything with Agile always tells me, “We’re not doing pure Agile. We have our own version of it.” I really don’t care if you’re doing Agile or not doing Agile.
What you’re really looking for is, is there a repeatable process to delivering user experience outcomes and business outcomes from the work that you’re doing, however you go about doing that. So, which race are you playing? Is it about speed to outcome or speed to output?
And with a lot of the more religious Agile teams that I see, really what they’re doing is there’s a giant backlog of feature work, and they’re trying to stay on top of doing just the minimum amount to make sure that they’re shipping something on time. And so, the game is really about outputs delivered on time.
A concept of what better means is often undefined or vague at best, often because there’s no feedback loop, there is no way a way—interest or resources to go and test whether or not the recent iteration that shipped actually worked or not, and a lot of times, the product team is disconnected from how a UX outcome helps contribute to a bottom-line result. So yes, don’t invest in design if you believe that you’ll eventually get there, doing things the Agile in the iterative way.
It’s not a good use of your time. One of the other things with this to be mindful about is that it’s very easy, in this Agile space, to fall into this trap of you see this giant skyscraper, which is the product.
The end of the product looks like this beautiful blue glass building, and there’s water outside, and the underground parking, and it’s got a food court, and there’s, like, a concert bandstand in front of the office, and your Agile process is about building one floor at a time, and then building out the food court, and then the bandstand gets installed, and then the parking lot is carved out. A lot of times, this is still what’s happening with Agile.
We’re not doing iterations; we’re doing increments of work, and it’s very easy to just get lost in that shipment of features on a schedule.
And some of those, and I’m being maybe a little bit extreme there, I’m not saying that these are completely disconnected from any type of financial benefit that should come on the back-end of that, but it’s very easy to just start playing the outputs on a schedule game because project management, well, that’s one of the easier things to manage in a large business, is risk and project management stuff, where risk is really tied to, are we on the schedule or not?
So, decide which game you’re playing, and then you can decide whether or not to invest in design there, right? Number six, you lead a technology, like, a foundation team, such as a data platform, that has to get done, no matter what. So, it’s best to start with a healthy platform before worrying about the application and user experience layer. Okay, so why is this one of the reasons you should not get design help?
And again, I’m positing here that, because you lead a foundation team, such as a data platform, and that’s going to have to get built no matter what in order to have these downstream products or applications built on top of them, you don’t need design help. My counter to this is that, really, the user experience should help inform the design and UI or UX of the platform, and what types of services it needs to support.
And this is to help de-risk the platform development and ensure that you’re not spending a lot of time on aspects of the platform that really aren’t going to produce any near-term value at all.
So, integrating user experience design, which often comes in the form of making sure your architects and lead technologists are aware of the product-y work and the user experience work that’s happening way upstream, this can really help save projects from expensive wrong turns that—it’s just very expensive to change platform directions once you’ve really gone into a full investment and a direction there, and you find out that it’s not really enabling the product to do what it needs to do.
It’s expensive to do that. So, the very product way to think about this is—Marty Cagan calls this, like, ‘time to first value’ or something like that—that’s a very different way to think about this, right? That’s going to force us into a just-enough platform to make sure we’re showing some value, and that it’s worth continued investment. Should we feed some of that value back into our next investment, or do we need to change directions?
These days, I don’t think you usually need to build a massive foundation before any type of value can be shown. There are ways to work in smaller phases and be very outcome-minded here. Of course, there are going to be exceptions to that. This is a general guidance, right? So, number seven, designers don’t know about complex analytics, AI, or statistics, et cetera, et cetera, so how are they possibly going to be helpful to your team who is probably quite technically proficient?
I would say there’s some truth to this, and that is a good reason not to bring on design, especially if you’re thinking about bringing in somebody to do fairly low-level surface UI changes, look-and-feel, this type of thing. There may be some work that they can do there; it’s probably not going to move the needle in a giant way, unless there are some very obvious low-hanging fruit type problems.
However, there are plenty of design firms and independent designers out there who have been working in complex enterprise software space for decades, so I think this really gets into how much experience you’re looking at, as well as the exposure to your domain or industry that the designer has had.
Particularly true if it’s more junior; not so true probably if it’s senior, so I think having an open mind that you probably don’t need another person coming in—particularly a design or user experience professional—who is primarily going to be a heavy, analytical-type thinker. Part of the reason you may want to bring in this perspective is because they are not approaching everything from that standpoint.
As designers and creative professionals, we deal with the qualitative gray area all the time. Things are, they’re not black-and-white in the world of design, and having a balance of these perspectives can be really valuable in ensuring that really technical products actually connect with humans on the far side where the interface hits the eyeballs, so to speak. So, be cognizant about the type of person that you’re hiring.
Obviously, it’s probably more expensive to work with someone that’s more senior, but it’s only more expensive if you’re just looking at it as a, you know, a consulting project cost, or a salary cost, and you’re not connecting that back to the value that you’re getting, and the speed that you’re moving at. Are you going to produce results on a faster basis, or are you simply looking at it from a labor perspective?
Obviously, the goal is to get better outcomes and outputs from that person such that your technical team’s work gets to shine sooner.
The other thing about designers not knowing about complex analytics or artificial intelligence and stats, et cetera, this is a very new space for a lot of people, including the end-users that are working with these kinds of tools, and so you have to decide whether or not there’s value in bringing somebody onto the team who has experience putting useful products and solutions in front of people, and figuring out, how do we make this new thing that never existed
before, how do we make this something that’s indispensable to the customer? And obviously someone that has a track record—or an agency or a firm—that has a track record of being able to take complex domain spaces and solution spaces, and turning those into something that somebody wants to pay for or that they can’t live without, that skill set, even if it’s not specifically in your industry, might be very valuable, especially if low adoption is something that you’re struggling with.
So, domain skill, too. My general advice to my clients and things like that is, you know, ramping up on domain skill. Domain skill is really valuable. Knowledge of—and I’m talking about, like, for example, are you working in healthcare or financial services, things like that—there’s definitely value in having someone that has practical domain knowledge.
There, they know the lingo, they know the vocabulary, they know all the risks, and the legal compliance, and all the different stakeholders that may be involved. There’s definitely value in that. I tend to think that there are, especially when you go with a more senior resource, if you’re bringing on some design help, the more senior they are, the faster that ramp-up process will be because they’ve had practice ramping up in a short amount of time into new domain spaces.
And we’re looking at patterns. Especially in the analytics decision support space, things like this, there are often patterns that are not specific to one industry.
However, if you’ve worked in that industry forever, you may see that as this is a very specific to healthcare kind of situation, and our whole industry is, like, screwed up, and you know, I’m thinking about the US and all this, and so we have to play the game within this specific constraints, and the way this bubbles down into the product or the application is X, Y, and Z. That whole thing, even if their knowledge is not specific to health care or medicine, they may have had
that same situation in a different industry or domain. And this is something I’ve noticed just being focused in this space for a long time that it’s funny how sometimes you can almost just swap out the industry name and insert a different one, and some of the same patterns of problems and possible solution types exist across these things.
And I think some of that just gets back to the common ways in which end-users need to do their work or achieve their tasks and their goals using software—particularly in the analytics space—around making better decisions, and how much information, and what kind of information. There are patterns here, and so finding resources that have had enough exposure such that they can see these patterns and bring that to bear, there are people that can do that.
But if you’re really shopping by price or just looking to, like, dabble in this, and bring someone on for the first time, and you go junior there, you’re probably not going to get any of that particular value, and you’re going to end up talking about fonts, and colors, and design systems, and fairly low-level surface things that may not move any of your big product KPIs forward.
So, you have to decide whether or not you believe that there are, you know, designers that have focused in technical space, and whether or not that’s really what the problem is or not.
My suggestion is, bring in a more experienced and seasoned person for a small engagement, and test the waters that way, as opposed to say, you know, hiring a junior person and adding them to a data science team, and then hoping that they’re going to, kind of, eventually grow into a more strategic partner to the product side. I think that’s a much more expensive bet. It will take a lot longer to pay off, and you’re probably going to have some frustrating early phases to that work.
I’d rather you test that out with the best possible solution you can on a small basis, and then reevaluate at that point.
The other challenge there, you know this is—I’m going on a tangent now a little bit about staffing and hiring designers for technical software product teams—if you don’t have a senior person there to mentor and guide a younger designer who may be new to the analytics or AI space, or if you’re in a very technical domain space, it’s going to be very hard for them to show their full value to the organization because they don’t have anyone above them who has walked
that path before and knows how to navigate the organization, and wherever the strong product power is, which often in technical products, this is in the engineering group, and not necessarily in the product management side. And so, if you really want to leap into something better, and not just kind of incrementally, step your UI forward, having a first hire come in that’s quite junior, it’s probably not going to show the best ROI.
I don’t have a ton of data on this, so this is heavily just an opinion from my experience, but you have to decide, do you believe that there are designers that can soak up the required data or analytical knowledge or AI knowledge or not? Is that a skill set that they can have?
And even if they have a shortage there, are they compensating that for other skills that your technical team currently lacks, and are they bringing to bear skills that you may not even be asking about because the problems that they can contribute to are not even being discussed or asked because the current team cannot see those things.
So hopefully, through your discovery conversations, if you’re thinking about hiring a designer, or a firm, or anything like that, you’re starting to get some of that signal through those early conversations. And if you’re not, then I’d say, move on and look somewhere else. Or keep doing what you’re doing, if you believe it’s just going to create a tax.
Number eight, retraining your machine learning models is expensive, and you won’t have time for that, for sure, and you really can’t go ask for more money at this point for your machine learning application that you’re building or whatever, so what’s the point of bringing on design when you know you’re not going to be willing to change all of that? I’ve heard this before. This is the sunk cost problem.
I’ve seen this many times, especially in large enterprises, where you may have hundreds and hundreds of people working on a product, or an application, or solution, and it’s been going so long, it’s over budget, and it’s taken way longer than we thought. It’s very hard to stop the train, and I would probably say, like, you’re right.
Hiring a designer to come in and re-steer that it might have results, but it means that the leadership has to be really open to stopping what they’re doing now and potentially pivoting and making a significant change, which might be, letting go of all the previous requirements and accepting that maybe the strategy and direction is not right.
If they’re ready to do that, then it might be a good time to bring somebody on, but if that’s a decision that the organization is definitely not willing to make, and you’re under this pressure, which is, this thing is going out the door no matter what on September 1st.
Whatever we got is launching, and this kind of very date and output focused mentality, then yeah, I would say it’s probably not worth bringing in somebody because, at best, they are probably—it’s not that they’re going to be disruptive, but what they advise you that might need to happen in order to improve the user experience to get the business value that you seek—if that is indeed a design problem that you’re having—the change required
to do that might be more significant, and the tendency will probably to kind of pick at it and try to do little incremental steps towards it. But if you’ve got all this weight behind the ship, and it’s like this massive cargo ship, and it’s already up to its 30 knots cruising speed and re-steering that sucker is really tough because of all the momentum and inertia that it already has, yeah, I’d say it’s probably not a good investment.
And unfortunately, you might have to wait until you ship that sucker, and you wait for the results to come in. Does it get the market penetration? Does it get the subscribers, the subscription revenue that you’re looking for, whatever the business revenue side is, the ROI that you’re looking for? Sometimes it has to ship and fail before the leadership can accept that it’s time to let go of what we started out.
And I hope that there’s less of this happening, and that we’re accelerating our feedback cycles so that we’re hypothesizing and learning in faster increments such that we don’t build in so much of that debt and sunk cost because it’s so hard to let go. The longer we’ve been working on that same strategy, and direction with something, it’s so hard to let go and admit defeat, and admit—it’s not even defeat. It’s accepting that maybe it’s time to stop. Just a quick story.
I was listening to a podcast with, I think it was the head of Google X, and they were talking about how one of the things that they celebrate there—and they’re really focused on innovation and really tough tech type problems over there—when the leader of a project inside Google X decides to hang up that project, and stop work, and pull the—I think it was Toyota that had the factory you could pull the chain, and the whole production
line would stop—when that person stands up and says, “You know what? I don’t think we’re going to see the value that we want in the time that we want to get it, and it’s time for us to stop working on this project,” that person is celebrated in the org. They’re given a large bonus, apparently. Everyone stands up and claps for them.
And I think that’s fantastic because it’s celebrating that this group has learned something, there’s probably some really great knowledge that’s been accumulated during that effort, but they’re making the smart decision that, hey, these resources could be better deployed elsewhere, and it’s time to let go of that. So, I know that’s a tough thing. Every company talks about innovation and all this kind of stuff.
Rarely do I see that type of courageous leadership and opening up the field for teams to be able to raise their hand and state that without fear of retribution, or how they’re going to look, or the social pressure that will come with standing up and saying, “My idea here is not working the way I thought it was going to.” So anyhow, that’s my little tangent on that. Designers don’t know about complex analytics and AI, so how are they going to be helpful to your technical team?
And is that a reason not to invest in design. So, I think I summarized that one. Number nine, you have a robust change management program or training program associated with this product that you’re going to deliver, and your users and customers are both willing and interested to take time out of their schedule to learn your tech, to learn your new product, and how they need to change the old way that they’re doing things.
Or maybe your audience of users is so small that you can just kind of heavily customize solutions for them; you don’t have hundreds, thousands, or even millions of users to deal with. These are kind of in the same space. This gets to do with, are you having to figure out what’s going to work for thousands of people, where you can’t give everybody a custom solution, or walk them through it or not.
I think that’s fine if your audience is so small that you have direct access to them, and you can basically get rapid feedback from them. And it might be it’s only a handful of people use this solution, but it has big impact. That’s fantastic, and you may be able to do a decent job with those rapid feedback cycles without hiring a professional designer. The counter to this is, you know, most new tooling, new applications, product, et cetera, is a tax on people, or it introduces risk.
And I’m sure we’re all aware of, like, right now, what’s happening with AI, and specifically GenAI, organizations are trying to leverage this in a heavy way. And there’s a lot of people saying, “I don’t want this stuff. I don’t want GenAI.”
There’s fear about its displacement power that it’s going to have, and so even if it’s great from a technical standpoint, and being able to project how the business might benefit from it, there’s always that human factor, which is, are they going to be willing to embrace this new technology or not? I generally feel that most people don’t want the new way.
Most people are not living on the edge, particularly in the enterprise software space, where if people are working in a large organization and using your product, they probably don’t want to introduce unnecessary risk. It’s much safer most of the time to just repeat yesterday, and not to embrace something new.
I mean, a lot of organizations, you go all the way to, like, the large banks, and financial services, and certain things in healthcare, there’s a huge amount of risk there, and generally they don’t—in my experience, that’s not a place where they really want to invite a culture of embracing new products and services, new ways of doing things for which your software or your application—or AI product, or whatever it is—brings.
And so, if you’re going to rely on training or change management, and you believe that you’ve had a successful track record of that, then, yeah, I guess you can go ahead and replay that training program that you do, and hope that the new product or application that you’re working on in the data space will be well received, simply by having access to training, or maybe it’s direct access to your product team, and they’re responding to those changes in a rapid way.
As you’ve probably heard on this podcast, the majority, I’d say the majority, of my audience, are not generally building solutions that are only for a handful of people. They generally have hundreds, if not thousands of users, potentially more, and so they can’t rely on, quote, forced training or offered training because a lot of people aren’t going to take the training.
They need the change management that this new data product introduces that as much as possible, we want to roll that into the user experience such that it doesn’t feel like a dramatic change is happening. We reduced the fear, we reduce the risk, we build the trust. We want to design as much of that into the solution as possible, so that we don’t need large amounts of training.
Training, unfortunately, often means that it’s not intuitive, which means you can’t just walk in and start using it; you actually do need someone to show you how to adopt this new process, or this new application, how it fits into your workflow. This is very much not how most design professionals think about it. We’re usually thinking about, how do we get rid of the manual? How do we build in as much of the intuitiveness as possible into this solution?
And often that starts with our user research, our user experience research, and our exposure to end-users, such that we know what’s it currently like, what’s an outcome look like for them today, and what does a better outcome look like tomorrow? Like, how would we know if we improved something in this person’s life such that they would invite this product or solution into their environment because it’s better than what they have now?
It’s a different perspective than looking at it as a technical problem. So, you have to decide; should you bring somebody in if you’re going to have robust change management, or your audience of users is so small, should you be bringing in user experience design or not? It might be a reason not to do it if that’s how you roll stuff out, and you’ve had a good track record of doing that to date.
Number ten, the last one here—thanks for going through all this journey with me today—you cannot quantify the value of a UX design or UI design investment in your product to your boss, let alone your own data product, when it’s not commercial or revenue generating. So again, this is getting back to feeling that the primary value you have to show, it’s a dollar value, it’s a financial value, and you’re not sure how to figure out, is this going to be worth it financially or not.
And you might also be struggling with the I’m not sure—you know, we build a machine-learning recommender, which is built into the front-end of the website, which shows insurance products, and so we’re powering a bunch of the intelligence there, but it’s not the product; it’s a part of the product, and you’re maybe not even sure what your data product overall, what its contribution is to the business, let alone what the design part of your data product is, back to the business.
So, it feels hard to know how to allocate funds to this, whether any funds should be allocated to this. And maybe it feels squishy. Maybe you know, like, this application is important to us, the interface is complicated or it’s complex, at the same time, we’re really not sure how to quantify that. So, I would say first of all, you’re probably correct.
You should not work with an outside designer, or design—an employee is a little different than a consultant; I would not advise you to work with somebody if they cannot help you understand what the value is that they’re going to bring to your product or your project that you’re working on in a way that you can understand. So, that means using your KPIs, your domain-speak, your language. That should be something that they’re able to do.
If they’re just talking about how many people, how many hours per week do you need this to start, and it’s all about the inputs and the labor that they’re going to perform for you, and it’s not about the downstream outcomes that they’re going to bring, then I would say, you’re right. You probably should trust your gut there and say, “I don’t know what the value of this is going to be.
They’re not talking about the value of this with me, I’m sure maybe we’ll get some type of happiness increase, satisfaction increase, from end-users.” But if budgets are tight, or you need to go back and sell this, maybe you need to acquire the budget from your management, you’re probably right. Either it’s not the right time to bring anybody in, or those are not the right people to bring into your project.
This is a tough space, and quantifying design is not just about the ROI, like, did it increase sales? Did it save cost? This kind of thing. I know a lot of times in the analytics space, we want to boil everything down to that, but only measuring the dollar contribution of design is really the wrong lens—so this is my big counterpoint here—there are times when we can do that.
I mean, there’s large e-commerce companies, for example, they might have design staff, a whole team working just on something like a checkout flow, and doing very small A/B testing and design changes to figure out whether or not they can get a bump in conversions or whatever. And you can argue at this point, now we can actually heavily attribute the value of design to dollars because it’s so tied to it, and it’s not a messy problem.
Unfortunately, most of us listening, and most of you listening to this show, and the people that I talk to, our problems are much more complicated than this. So, I think what an analytics product manager, an AI product manager—especially one that maybe hasn’t worked with user experience design before—needs to accept is that there are other ways to measure the value of design’s contribution to your product and to your team and your organization.
So, for example, let’s say that you have a mission-critical internal data product, it’s used by the most senior executives in the organization, and you and your team made their day, or their month, or their quarter.
You saved their job, you made them feel like a hero, or you gave someone that’s maybe in middle management some power or perception of seniority that eventually moved them up and got them—maybe their ability to use your data for decision-making enable them to actually make some really important decisions that had huge payoff for the organization.
What is the value of giving them that experience and making them feel like those things, having those feelings—those stakeholders who are champions for your work—and feel like, oh my gosh, this saved my butt, or this got me the advancement that I was looking for. And maybe they didn’t even tell you until afterwards that they were looking for some ammunition to get to pull the trigger on an initiative that they believed in, and they needed your data to help sell that there.
What is that worth when a key customer feels like you have their back? Ideas that spread, win, and so if these people are spreading your idea, or your product, your solution, around and talking about it because it enabled an experience like that, I think there’s a lot of value in that. You have to decide what it is, but that’s often one of the reasons that design really has an impact.
And sometimes it’s one that we didn’t ask for upfront because we didn’t really think we could put that as a goal on a project, and it became a surprising outcome of the project. This can be an intentional outcome of the project, but trying to simply assign a dollar to value to that, that might not be the right point.
I think we can all accept that if our boss—you know, if you have a [laugh] boss or management, and you’re able to make their day, there’s going to be inherent value in that for your team, whether it’s in the form of more funding, or getting left alone so that you can bring solutions back to them and ideas because you’re now a trusted ally. There are other forms of value here than just assigning, did we save money or did we make money, right?
The second one under this number ten, this ‘just the facts’ mentality. It’s really not enough here to just give the facts. Rampant focus on data quality and information alone, it’s really incomplete, as it doesn’t factor in users’ emotions or feelings that they’re having when they’re using your solutions.
If you’ve ever battled with an LLM—I think we talked about this earlier—but if you’ve ever fought with an LLM to try to twist it into giving it what you wanted, and you simply can’t get it to, well, the model may have passed all of the test criteria that you gave to it, but at the end of the day, you might have a bunch of users who are really angry and frustrated because it’s not helping them get done what they wanted to. They’re looking at this thing as not being helpful to them.
Their feelings about it are that this is either attacks on me, it’s frustrating, it comes across as if it’s going to—wow, I can type in anything and get all these insights back, but when I actually try to do it, it’s really hard. That’s all about their feelings and perceptions, and it doesn’t matter factually how technically right the solution is. You’ve heard me say it ‘technically right, effectively wrong’ space.
Another example, just thinking about other forms of value here, you know, we buy insurance all the time. So, we’re spending money on something that most likely will have zero economic value this year because we’re actually trying not to have to file claims most of the time, yet this industry does very well because the feeling of security matters. That feeling is worth something. Quantifying that feeling in dollars, well, if we’re literally talking about insurance, you can quantify that.
It means that the value of feeling secure is something greater than whatever the cost of the insurance plan that person bought was. So, you can put a dollar value on that. Not all solutions we can, but what’s important here if you’re not selling insurance is this idea that insurance does have some type of inherent value for it, and it’s about the feeling of security.
So, if your solution can build feelings of confidence and security, does that have some type of value to your organization and to your end-users? How would the world look like in your space if you were bringing those feelings forward through your application or service, right? We talked about this fourth one, under number ten, did the idea spread?
Did a key group of people who adopted your new, and you know, AI-based product, or whatever it was, did they spread the word to other departments and colleagues? Did they champion the change? And did they say, you know, “This really is better than the old way we were doing things before?” Did it improve their status? Did it make them look good?
I think these things have a lot of power because all people want to feel like the champion, and if you’ve ever been that person who’s becomes such an advocate for something because you saw what it did for you, and you want to share that with other people, there can be a lot of value in that. If your product or application is the thing being shared there, there is value in that.
And so, if the design helps contribute to this feeling that this is worth sharing, and I’m willing to spend some of my social capital to go out of my way and say, “I really think this is good for you,” and it’s not a sales pitch, it’s not a marketing campaign from the product company, it’s an actual end-user that’s sharing those feelings, this is another metric that design can bring to a project. Qualitative perceptions of trust is the fifth one here.
So, if people trust your new AI product or analytics solution, in part because the system is really clear about what it can do, what it knows about, where the data came from, and it’s just enough product at the right time, in the right format, and the right experience, how valuable is that to your stakeholders and end-users, particularly if, say, you know, embracing AI is a goal for your organization, and you’re able to give them those feelings of trust and security,
and are willing to embrace change, what is that worth to the organization? Is that worth anything to you and your team, as a product, to have that trust. Especially if it’s a zero-to-one situation, where it’s the first time you’re rolling out something that’s disruptive, potentially, or a big change from how they’ve done things in the past? What is it worth to have those qualitative perceptions of trust?
And if people trust that both you and the AI solution is not out to take their job, and rather that your solution was designed as an aid to make human work more efficient, or to give them the time back to work on higher value challenges, what is that worth, especially when we compare that to a large investment in AI, and the end product or application that was built, which was technically right, is feared.
It’s imposed with change management, and it’s sent down from above, and said, we are now doing things this way. Get on the bus or get off the bus, but this is how we’re doing it. That’s the opposite to building the trust into the solution, right? Now, we’re imposing this thing that people are already—a lot of end-users that are non-technical are already concerned about here.
What would it be worth if that was not their perception, that this was perceived as an ally, a way to augment that human work and not replace them, or replace their job, or give them that sense that eventually, hey, they’re not going to need me and my team anymore because of this solution. Anyhow, this was a long episode. There’s a lot of stuff in here about reasons not to invest in user experience design help for your analytics solution or your AI product that you’re working on.
I hope this was useful to you. Please let me know. Leave a review for the show. You can head over to designingforanalytics.com/podcast. I’d love to hear what you think. You can also record an audio question or comment there for me right in the browser. You don’t need any plugins or anything. So, I’m always curious to hear if these episodes that I roll solo are helpful to you. Anyhow, that’s it for now. Thank you for listening to Experiencing Data.
We’ll be back with some guests soon, and I wish you all the best. We hope you enjoyed this episode of Experiencing Data with Brian O'Neill. If you did enjoy it, please consider sharing it with the hashtag ExperiencingData. To get future podcast updates or to subscribe to Brian's mailing list, where he shares his insights on designing valuable enterprise data products and applications, visit DesigningForAnalytics. com slash podcast