Deep Dive Into Experimentation With Lea Samrani (ex Bumble, Badoo, Uptime)

And um, I think you're a lot less likely to actually put your money into this product.

I'm Christian Vasile and on this podcast, I bring you world class product and design leaders who found ways to shape products, companies, and entire industries, and who are now sharing what they know with you and me. My hope is that we all get to learn from the experiences, ideas, and stories shared on this podcast. I'll see you next time. In the process. become better designers. Today, I'm chatting with Lea Samarani, who's one of the best, if not the best product manager I've ever worked with.

Since we've worked together, Lea has become a consultant and is doing great work for her clients, helping them scale and grow their products. Today, we're talking about how design can build better relationships with product, what kind of designers collaborate well with product people and we go super deep into talking about experimentation. I hope you'll enjoy this one. Lea, very excited for our chat today. Welcome to Design Meets Business.

If someone is looking over your CV, they'll see some cool companies there, Badoo, Bumble, Uptime, and more recently, a lot of different startups that you're advising. But if they look even further, they'll see some great results you've helped these companies with. Whether that's increasing conversion rates or increasing MRRs or helping them remove other barriers for growth. And I know you're a big fan of experimentation, moving fast and taking calculated risk.

Just the type of product person that I love partnering with. And I'd like to dig deep today and see what designers could learn from you to replicate some of these successes that you've had with the teams you've been working with. But before we begin and go into all of that, please tell us a little bit about yourself.

So it's been very interesting over the last few years. I've worked with a lot of different apps, lifestyle, dating, health, fintech healthtech as well, like very different category, but all of those products have something in common and it's always a trying to build a great experience for the end user. And my job has been to find what this great experience is for the end user, which will also make the company money and grow and be sustainable over time.

So that's what I've been doing for the last few months. It's very interesting. I learned a lot. It's very exciting. And I hope I can keep doing that for the next years to come.

sure. Recently, you've moved from more permanent roles to advising, right? And you're working with a lot of different companies now. How is that? How has that changed your daily life versus just being on one company?

You still need to be able to dive in very deep into subject matter, even though you don't have the same level of information as you would have if you were in house. There's a bit of balance as well, because very often as an advisor, you're responsible for the strategy, but you're not actually executing. And, we very well know that anyone can have an idea, but the execution is actually what makes a difference.

So you don't have that much control on the execution part of things, which tend to be very important for the end results. So it comes with a very different set of challenge than being in house, but you do get to meet a lot of people from very different backgrounds. You get to work on completely different kind of products, different size, different problematic.

I've learned so much about things I knew nothing about, like recently I've, I've helped a product that was growing fish on Mars, what do I know about growing fish on Mars? It's a, it's just very wide range of product on, and it's amazing. I feel so, What's the word? I felt so lucky that I get to experience that. I

And I think learning all of these puts more tools in your tool belt You said that as a consultant it's harder to control the implementation of it and the execution, which as you also said, it's probably the part that matters the most. So how do you then keep track of the execution and try at least to control or to shape that in a way or another when you're not there every single day.

And this is what we're going to be doing this month. You break that down into a smaller chunk of work that are easier to go into execution. And if you work closely with those people, you can give feedback on a regular basis and be much closer to the project. Now when the company is, bigger or maybe more spread out, there's more stakeholder, it gets way more challenging because you can only affect a certain area of it. So in that case is really more around process.

In my experience where you try to shape a bit more how people are working with each other and how the company process are enabling you to apply that strategy rather than the actual execution of. the strategy you suggesting

If you're a designer out there and you think what does a good product person do? Help us understand what's that ideal person like?

And for me, what sets them apart is that They had a very good understanding of the strategy from the business perspective, they had a very strong understanding of the customer, the end user, the person that will be using your product with a lot of empathy for those people. But they were also very practical people. So they had that capacity of taking all that uncertainty and all that knowledge and turning it into very small, applicable decision.

The second thing they were really good at was ensuring that everybody in their team understand why we're doing what we're doing and so that allow them to create a product mindset across the entire team. And so suddenly you're no longer the only product person in that team. You're not the product manager. you are the one shaping your product mindset. So the entire team become a product manager.

And I think we see that more and more recently, like you yourself, you're not a designer, you're a product designer, right? Product designer, frankly, can eventually replace product manager and they should because it's almost the same set of skills with an added hands on design experience. So I think that's a really great product manager that I've seen now in most company, that's not the case.

In most company, product manager are project manager they do a lot of admin, they just move things around. They don't get to really decide what they're working on. It's very often stakeholder led or sales led or finance led depending on what company you work with. And then it's more about you trying to push that vision to life or quite often as well as just trying to keep your team happy or explaining why you stakeholder have decided something when you don't really know yourself why.

You just kind of that middle management person and this is probably the worst Application of product and I think that's what gives product manager a bad name because it's not It's it. Yeah, it's you just a product manager by name in this case, but not really in function.

So then what it does is that it forces you as a designer to borrow some of those skills that the product person otherwise would do and do them yourself. And we talk a lot about being more business savvy as designers on this podcast. And I guess for me, anytime I look at a good product person, for me, they're always just very business savvy.

And if we're able as designers to learn those skills I don't think it's a matter of replacing, as you said earlier, but if you work in a smaller company that doesn't have a lot of resources and has a design team and an engineering team, but doesn't have resources for product team, then I think as a designer, you can fulfill some of those duties yourself, because as you said, they're very complimentary skills.

So, One of the things that I believe in is that in this triad, engineering, product and design, one of the aspects that sets good teams apart from lesser good teams is the relationships that they're able to build with each other. I think that the strong relationship between design and engineering, strong relationship between design and product, those oftentimes push teams further than if those relationships are not good.

So I wanted to talk to you having worked with a lot of designers in the past in different teams and different size of companies. How do you as a product designer build good relationships with your product counterparts?

If we all, the three of us, and actually I would argue the four of us because I do believe analytics has its place there as well. If the four of us are aligned on. Why we're doing what we're doing and who we're doing it for, then the rest kind of flow.

So you all have different skills and you all have different background, which allow you to see something from a different perspective, but you do that with the same objective for the same end user and to try to solve the same problem and understanding why you're doing it. And so that allow you to have the same foundation. And the second you have the same foundation, everything else naturally flow. Then there's respect that comes with a job there's mutual respect.

You can't, or at least I can't go to a designer or an engineer and did you say like, why did you do this thing this way? Do it that way. Instead, there's respect. I have respect for the skills, for the experience. So if you're questioning something, it's more around take me through that past. Tell me why you decided to do something a certain way so I can understand your point of view rather than essentially saying what you did is a bit icky. Let's, do it my way.

And then, it's just a human side to it. Like, how do you build relationship in your life? There's just basic stuff, right? Be kind to people, talk to them, try to have a decent like normal human relationship. If it's outside of work, sometimes it's even better. That used to be a bit of a bit easier before. Right now, you have to make a bit more of an effort to try and talk to someone outside of work.

But those are just like basic human thing that at the end of the day, when you get along with someone, it's also easier to work together and you don't have to agree. Frankly, I rarely agree with my engineering counterpart or my designer counterpart. We very often disagree, if anything, and that's very healthy. I think it's very healthy because that's how you get the best of each other. But the disagreement is always done within respect of understanding each other's skills.

And within that foundation where we all on the exact same page on why is it that we're doing what we're doing and who are we trying to do it for? So the disagreement is on the what, and that's okay.

Oftentimes what happens is that someone comes in and gives their subjective opinion. Oh, I don't like this because I don't like pink or whatever, right? As a designer, that's really hard to deal with that because that's a personal opinion, right? You don't have anything, any data to back that up with. You're just like, I just don't like it because I don't like it. So in that case, I think disagreements can sometimes lead to a bit of animosity and a bit of conflict in the team.

How do you encourage designers to deal with product people who give them feedback based on, personal opinions. Is there any way to push back? Is there any way to ask further questions to understand where that feedback is coming from? What do you think designers should do in that case?

It could also be because they had that vision in their head and that vision is just something they're struggling to get away from because that's just what they see and what they think. So in that case, that's a bit harder, but you can again, try to understand what is it they're trying to achieve with that vision. So it's not necessarily saying why would you put, you know, I don't know, that CTA on that page, but more around what are you trying to achieve by putting that CTA on that page?

What do you think that will enable the user to do? How do you think somebody is going to feel when they see that? Or what action do you expect somebody to take when they see that? And I think that allow you to understand a bit more what they're trying to achieve and then find some sort of way to deliver that, which can be a mixed vision, it can be yours, can be them. It doesn't matter that much. Sometimes it will change. But it's really trying to understand that.

And very often we come at it from a different perspective. When you look at something from a design perspective, you do want it to, I suppose you do want it to look good, right? To look nice. It needs to be beautiful. It's art. It's a bit of art, right? From a product perspective, I don't think you come at it from that art view it's way more around the practicality of it. I want to know if something's going to be easy to develop, to release. I want to know if something has been tested before.

And so we have a bit more certainty around it rather than maybe something brand new that's maybe a bit more risky and maybe that's not really the right time to take that level of risk. I want to know if this is going to solve the problem that we're trying to solve in the most efficient way. And if that's not the case, then, there's definitely going to be some sort of a discussion there.

And the last bit is a bit around the overall look and feel and the thing, I mean, that should be fully owned by design really, but there are some case scenario where the product manager is going to provide feedback on that. And that could be either because the designer is maybe a bit junior and so need that kind of support. Or because the product manager is just too opinionated, so on the one hand that's good, the person can be a bit junior, can use a bit of feedback and develop.

But I do get concerned when your product manager knows more about Or has a a better eye, I would say, than your product designer, like that's so rare, if that's the case, I'd get concerned. And on the other hand, it's just setting like this healthy boundary from design to product, which is really again, like going back to, what is it you're trying to achieve? What do you expect the person to do there? How do you expect them to feel? And then go back to your work and see if that achieve that.

You care about how it looks, especially people earlier on in their careers, care more about how it looks than the business part of it. Because frankly, design education doesn't teach you that. So that comes with experience. So if you're earlier on, you're much more likely to just put something forward that you just like how it looks as soon as you understand what other people care about. And that's not only product people, by the way, that's also engineers. They also care about.

If you come up with this fancy, crazy animation that looks good in Figma or whatever, and then you show that to an engineer and he or she thinks this is going to take me days. Can we have something simpler? It's not because they don't like it. It's because they care about being able to ship fast and they care about not having to write a lot of code that will create tech debt and all of these other things. And it's the same with product. They care about does this solve the problem?

Is there a risk here that we have mitigated? So I think when you present work from a perspective of what does this person care about and what should I talk about when I present the work covers what they care about. So if they care about Does this solve the problem? In your presentation of the work, talk about how you think this solves the problem. If we present work like that, I think that can also resolve, pre resolve, ahead of time, some of these issues that might otherwise arise.

I'm going to ask you a question that you might or might not have an opinion on, but what is an ideal design partner for you as a product person?

And it's not, I heard like a couple of times like this is designing by committee and I really disagree with that for me, this relationship is so important and bringing those feedback so early on, it allow you to avoid this big clash at the end, it completely avoid the actually you didn't really get my vision, or we went a different direction, or you have to justify why you're doing what you're doing. just because you have that relationship early on.

So having that person that's very communicative and actually is happy to get feedback early on and share their idea. And it's not pixel perfect. If we're talking about an idea, I'm not going to look at the details. Like I know not to look at the details because you told me it's just an idea. And so I really like that. I think that really get us to a place where at the end of the work, I'm always like, wow, this is amazing. I also have a preference for product designers that understand data.

It is sometimes a little bit challenging when you work with product designers that care a lot about something. And I get why they care about it, but that thing they care about will be seen by two people and then as a product person. At least as a business person, what the work we're trying to work on is something that need to impact a lot of people, a lot of users. Unless, you know, we're like that huge team. Perfect. I can spend time on like that tiny bits of the floor. That's great.

But often that's not the case. So if you're going to spend a lot of time on something that's going to affect two or three people, this is something that I'm going to question. But if you're someone that understand the data and understand how you work, affect the end users. That's never really a question because you would, from a product design perspective, you'd come and say, actually, does that really matter? Like only two people are going to see that. Can we move on?

So I think that's quite interesting. And then the last bit for me is around copy. That's a whole other subject really, but in most organization, I think what you'd see is design gets done and then copy is almost like just placeholder. And then he goes to a copy person and then do the copy the end result doesn't really make a lot of sense. It's a bit patched up.

So there's ways of working when you can do like content design where you know the copy is very integrated very early on, but you can also have a product designer that's thinking about what are the messages we're trying to communicate there and how should we communicate that and what's the best. Form to do so when they're designing something. So we never really end up with, I don't know, space that don't really make sense or steps that don't make sense.

And I guess that can be brought back to a bit of the psychology of what we're doing, who we're doing it for and why we're doing it right. And not just thinking actually I have to design three screens, but what is the intent of my screen? What is it trying to achieve and how is the best way for me to communicate that both visually and verbally?

So yeah, so those are like the three skills, you know, over communicate, feedback, early relationship data awareness, and if you don't know what the data are or how to get them or what they mean, that's what the product manager is here for as well, or your product analyst is something you can easily go and ask and then the last one is really understanding, what are we trying to achieve, but not just with design as in, How is my work communicating to the end user and how is my work solving the

problems the end user and integrate copy as well as visuals?

What I've learned, and I remember I was at British Gas back then, and we had this very complicated analytics platform and I was swimming in the dark a lot. And I went to our analytics team and I said, can you just please explain this to me? And what happened is that they've realized that someone in the team was actually interested in their work.

They ended up for weeks, just bringing analytics to me because they finally thought, okay, we have someone in the design team, a partner in the design team who understands the value of what we do. So then again, that's also going back to building relationships is being interested in what the other person is doing as well. So that's one of the thread that I think it's worth talking or it's worth highlighting again, the other one, I am a big believer in the fact that designers should write copy.

I will never move away from that now. With that being said, you don't necessarily need to write absolute perfect copy. It doesn't always need to make perfect grammatical sense and all of that. But if as a designer, you write your copy as you design, then it will feel like a better experience. And then what you can do is to bring over a copywriter or someone who.

Who can just polish that for you, but I always find it strange when you design something and you put lorem ipsum in it, like the content is part of the design, the design without the content is doesn't make sense. So for me, you have to write your copy and if you need someone to check it afterwards, make sure it's, tone of voice, whatever fine, but at least the direction of the copy, I think that's important to write there.

There's another thing that we haven't talked a lot about, but you are a very big fan of experimentation. And I like to pivot a little bit into talking about the experimentation and your background in it. And then how can designers fit into that framework or into that way of working a little bit better than perhaps they do today.

So let's talk experimentation. What are your thoughts there and what sort of successes have you had experimenting in the past?

And then I've been involved now in. multiple project where we try to brought experimentation to a new company that haven't adopted it yet. And I'm always amazed at some of the early results you get when you start experimenting with a company. Because it's not just about let's A, B test. It's really about let's change people mentality into thinking everything we do is a hypothesis. And nothing is just set work.

And when you start thinking that way, when you start thinking everything we're doing is a hypothesis that we need to validate or invalidate, it completely change the way you make decision. It changed the way you think about your work. It tend to make you go a bit faster, at least I believe, because If you do it well, you start thinking like, what's the smallest way for me to validate that? And it's just validation and validation doesn't necessarily need to be an A B test.

It's better if you can, but some company don't have enough user or, they don't have enough engineer to build it. And then validation can happen before the product or you can also even just fake it with with copy or with ads or anything else that you can do before the product. But that mindset becomes. Everything we do is a hypothesis that we need to validate in some shape or another.

And then you end up setting your team in a way where they work on like much smaller timeframe every week or every other week you start, releasing, pushing new things to life and you start judging your work with a different lens. So it's not Oh yeah, we finished that project on time. We're happy. We built that big new feature. Let's launch it. It's really about how many people have adopted it. Is there a repeat usage of it? Are people more retained once they go through that feature?

What are the feedback on it? Can we also get user feedback on it, like qualitative information? That's also part of experimentation. And then if you have the luxury of big data, absolutely fascinating to see how often something that's You know, being requested or you thought was so important is actually not received like that by user at all. And I've seen that so many times. I've seen like the most requested thing and it are being adopted by, 2 percent of users.

And this user actually were not more engaged or more retained than other users. I've also seen things that are very controversial from a business point of view or from a team perspective. Something that we never wanted to do or we thought was so annoying or. Sometime even stupid, frankly, and actually when you launch it, the results is there and it just works. So big data is very powerful, but also understanding why things are moving a certain way is important.

So experimentation for me is one way of validating things. It's not the only way. You get your big data, you understand how something have affected your user behavior. But then you need to qualify them. Why has it affected them that way? What is it that happened? Qualitative there is also very powerful.

And that's why experimentation allows you to do it, just allow you to validate how what we're doing is actually aligned with our user base and or our business target, really. And again, 90 percent of the time it's going to be wrong. It's a big celebration when a test works because it's not that often. And if you test are working really often. It's because you're really early on in that journey, and you're so far from being an optimized product that pretty much everything you do works.

So I've seen that as well when, uh, when teams have really good hypothesis, they have a really good understanding of user problems. They have a nice, discovery tree. They put hypothesis, good design, go and put it to test and then it works. Let's say 70 percent of the case, right? This is because a product is really not optimized. It has a good market fit, it's a good product, but it's really not optimized and your tests are working better.

The more your product become better, the less likely your tests are to work. And if you were not to test you could think about it as in 90 percent of the case, we are going to take the wrong decision. Now, one of the problem that comes with testing, and that's something I've seen quite a lot, is people think, I built this test, it's kind of an MVP version of something, it's launched, it worked kind of okay, let's move on to the next test.

And that's wrong as well, because experimentation is a way to validate a direction, it's not an end result. So that feature that you just launched or, that user journey or like tiny validation, it's telling you to go into a direction.

It's not the end results, and it's not something that you can just let there and then move on because if you do that, then you find yourself with one of those products that have tons of feature, but all of them are half baked and none of them provide a really good user experience. And that's also very important, we've seen a lot lately, I think of a pushback around like the whole concept of MVP, even around, does MVP really works?

If something is not well done or well thought through, can you even use it to measure results? And then for me, an MVP is very it's just a way of validating that you're in the right direction. And sometime just feature adoption can be a way of you thinking we are in the right direction. People want that. Now let's think about what that feature should actually look like. And then let's test the first version, the second version, et cetera, et cetera. And so we go into the.

The details of that user flow, but that's not something I've seen a lot because team are pressured for, delivery for pace, because there's always a lot of idea, a lot of project, a lot of OKRs. I haven't seen very often team that have that kind of space where you can think here's a hypothesis. Let me validate it first outside of product, talk to users or do copy test or an artist on email, whatever. Okay, this is a good direction. Let me now build a validation inside the product. This is my MVP.

It's a user flow. It's not a feature. It's an actual user journey, but maybe a simplified journey. We test it. Okay. It has a good number. What is good means as well. That's something you need to set before you start testing, but it has good results. We seeing, I dunno, high adoption, higher retention rates higher engagement or monetization, like whatever is it we're testing for. Now, there's a step that comes there again, which is how do we evolve that feature now?

How do we evolve that user journey into something that's actually the end product or some sort of an end product? And that step is usually missing with experimentation. And that's one of the reason I believe why team are pushing back against experimentation. Because they see it as this like feature factory of half baked MVP that just, quickly get to life and then disappear, but nothing else happened.

And then The fact of the matter is that what you're building is not an MVP, it's just an an MP, it's just a minimum product, but it's not necessarily viable. I mean, To me that, that part in the middle of the viability of an experiment is really important because if you half bake your MVP and you put it out there and people get this feature in their hands and they think this is absolutely crap because you haven't executed it well enough.

then they're going to drop, they're not going to adopt that feature. You're going to look at the analytics and think, Oh, it didn't work very well. Let's go into a different direction. When in fact, the problem might be the fact that your MVP was just not well enough put together. I've dealt with that so often. How do you, ensure there's a balance between how fast we move and making sure it's a viable experiment that we're doing?

Maybe that doesn't make it into the first version. If there's like backend infrastructure needed, maybe that doesn't make it into the first version, maybe we fake stuff. If there's a lot of logic, again, can we fake some of it? And then, if it works, then build the logic. So it's a way of making our work a bit faster and a bit simpler while achieving the same end results. Because me as a user looking at it, I won't know that you fake that logic. It will look real to me.

I will only know you faked it if I use a product a lot for a really long time and I see that actually doesn't change. But my perception is that there's an actual logic built into it and you will see from my reaction, from my usage of the product, if this is something that's worth pursuing or not. But it's not problem that is easy to solve because it's a very common problem. I agree with you. That's something we see a lot. Aaron.

And one of the reason I think we see that a lot is because we don't understand what success means when we start designing something, when we start building something, even before, when we set a hypothesis. What would success looks like? And that's something that we need to think about before we build it, because it's going to tell us if what we're building is actually worth building. Is it going to answer a question or actually, is it just going to open more question?

Because we have no way of understanding if that's a good thing or a bad thing. You know, You learn something and increase something by 2%. Is that good or bad? does that mean you should spend more time on it or not? Have you achieved the maximum you could achieve there, or is that just showing you that direction is worth pursuing and potentially you can increase it by 50%? And this comes from setting a very clear understanding it comes from setting what success looks like.

From the hypothesis stage and as a product designer, if we go back to that's something that you need to understand as well to do your job, how can you design something? How can you create something if you don't understand how success looks like? How are we going to measure that? It's part of that thinking as well. That needs to come into the role early on.

If you don't understand what problem you're actually solving, it is then your responsibility to say to your product manager, to any, any product counterpart or business counterpart what problem am I really solving here? with this design before you actually go and open Figma. And yeah, so that, that's sort of part of your discovery that, that is an absolute requirement.

So talking about running some of these experiments, how do you come up with all of these ideas and all of these hypotheses for what experiments to run?

it can be anywhere in the product or any kind of behavior, but I'd like to understand how What part of the product is the part that is used the most and which part is not or how long someone is using it for so we can draw this little bit of a journey and understand where is it that we should be focusing on because it will affect the majority of people and a lot of product that tend to be very early on once a product is a bit more mature it tend to be much later on and the later it is the harder

is it to get results because it take longer to get there. But assuming we have this, fresh product and we realize our day one retention is really low or conversion to payer is really low we are gonna try to understand why. So we're gonna look at this data to see, you know, are, are people registering? Are they dropping during activation? Are they reaching our aha moment? Do we understand what the aha moment is actually, are there many of them?

There's a, the whole concept where actually it's not just one action that you need to do to get value from a product. It's a bunch of action. It's a bunch of repeated activity. So let's try and understand that first and get that from the big data. Now that we understand that. I want to talk to people to understand why are things the way they are. And there, there's two schools. There's one school, which is, let's not talk about product at all. Let's do like more job to be done interview.

Let's talk about people about what matter to them? What are they pull and push and what drive them into the day to day decision making? What scares them? What are the anxiety and what are the current behavior that we might need changing? And this really allow us to have a bit of psychology of users.

The second thing is way more closer to product where, you actually do proper user testing of your product and you see how people are interacting with it and understand it closer to the thing you're building, how they're interacting with it and, what's happening there. Majority of the time there's huge incomprehension. That's where we start. People don't actually understand what your product does what to tap, where to click, they don't understand any of it.

They're just like this very confusing experience and a lack of consistency between what they've seen maybe on the ad that brought them there versus what they're experimenting on the product and generally like a lack of understanding about what's in it for them. And then on top of that, we ask them a lot of stuff. We ask them, give me notification permission, give me location permission, give me your money, give me like whatever is it we ask them a lot of stuff.

So we ask a lot of commitment from them, but we're not giving them a lot and start all put together make them drop off. Plus a lot of, other things like life, like I don't have a lot of time, I have other products, I have other things to do, et cetera. So then you go and talk to people and you put two and two together and come up with a bunch of hypotheses around what you could be testing and those hypotheses, they don't need to be small hypothesis. It can be big things.

It can be, you know, we believe that if we build a new product, actually we will solve that problem. And once you have those high level hypothesis, you go in and you try to think about how can I validate that? And then that essentially leave you a list of ideas that you can go on and test. And you can classify them, there's like different way of classifying them. Like people like to do high impact, low effort first.

I think that's arguable, but anyway, like you classify them and then at that stage again, you need to set like what success going to look like and then you go on and you do it.

It's the hypothesis is that if we give people I don't know, a way of pausing their membership rather than cancelling it, they are more likely to take that, therefore, we won't have to go after them to convince them to resubscribe or something like that. The idea for the experiment comes because you have the hypothesis, Doesn't come first. So I like that and I just wanted to, highlight it again. Last question I have on experimentation. You might join a company.

And they are not necessarily the mindset of experimentation. They don't understand what it is. They don't understand what's the value of it. They think it's too risky. They just want to put big features out there and then just run the company like that. And perhaps there's a place and a time for that, but you might come in and you might say at the point in the company we are at right now, experimentation is what we need.

How do you then convince people who are risk averse and they just, they just don't understand that world? How do you then at least start convincing them and start showing them the way that experimentation is the way to build a better product?

A vast majority is, they're doing it on MVP and non MVP as well. When they're releasing new big feature, new big experience, they're doing it in a way where they are I'm measuring the results and usually you do that by split testing or by releasing to only, one country first or one part of the population first. Especially if you have network effects or, anything like social media or dating, then you even need longer time to get results.

So those companies tend to do like a percentage of the population for a very long time. So that's like how one way of going about it. Find a bunch of successful company, put it together, show an example. That tend to work quite well. Secondly is looking at what we've done in the past.

Even without experimentation, there's always a level of understanding on things that we may have built and released and failed with, like you can go back and see that we spent eight months building whatever, and actually we can see that only 1 percent of people are using it, or we can see that whoever is using it has no difference in retention, monetization, like whatever are those metrics you care about versus other cohort of people that would assume you have decent data, which often those

companies don't have, but let's assume you do. So you can go back and you can literally say, actually, if we were to have experimented with that we would have saved the company X amount of time, X amount of money, and you can quantify that. You can quantify that with people time with investment, with revenue with actual growth, which is something that really helped putting things into numbers, especially depending who your stakeholder are and how they think.

The third part is convincing the team to, to do it as a test. You experiment with experimentation, if you may. And for that, you, you need to possibly use the simplest possible tool out there because we don't want to have a big, you know, engineering integration.

Maybe we already have some things that are used for release that we could use to like split test or maybe we just do it outside of the product first as well, to see if it works, like on ads much easier to test cause it comes with a tool. And that first test, it's very important that it has results. It can be really good results or it can be really bad results. That doesn't matter, but it needs to have a very strong results.

So you can actually show the value of it because with experimentation, you know, when you do it a lot, you'll see that a lot of the time you don't really get any results at all, right? There's no significant difference. And usually that means there's no impact. And then it's up to you to decide if that's a good thing or a bad thing and then what's the decision there.

If your first test have no impact, it's going to be really hard to convince your stakeholder that, you need to invest the time and effort to do that when actually shows no impact. So that first test you're doing, make sure that it's something that is going to be strong. So that is something very top of the funnel that will have a lot of people that go through it.

So we get those numbers really quickly that the change is not tiny, that it's big enough, not too big that you can't, understand where the change come from, but big enough that it will have an impact. And ideally the impact, we want it to be on something that's very tangible. So I love to do monetization tests because you get really quick results and you can very say, essentially we make twice as much money if we do that versus that. And we only know that because we did a split test.

And I think, yeah, I think those are like the three things that you could do that works quite well at least what I've seen. So take external example of very successful company and how they work look at your own ways of working in the past. Tangibly measure how much you've lost by not experimenting or by committing to a decision that now retrospectively, you know, was a wrong decision.

And third thing a validation of the experimentation framework by doing a mini test, out of the product on something that's small enough that it can be built quickly, but big enough that it will have an impact and a very tangible impact

And, we talk a lot about the value of design from a perspective of business. We can move metrics, we can reduce costs, we can do all of these things through design, but you also believe that the UI of it, which is something we don't often see that it has business impact, you actually believe that the UI of it, that UI side of design also can have a business impact. Let's talk a little bit about that.

And I actually, I found that a bit funny because it's from the private industry. We used to a certain standard from the private industry. When you look at public industry, you're very different, but that's a topic for another day.

that means that there's a certain level of things that you have to be doing before we even talk about, what your product is solving, how good is your product, et cetera, there's just like a minimum standard of entry that include things like bugs, product stability, product speed, and I believe you also include the UI, the general look and feel of your product.

I believe consistency in UI and quality of UI is directly linked with trust and it's how you create trust with your brand, with your product. And trust is a necessary step for people to come and use your product to pay for your services. Let's say your UI is a bit more maybe lifestyle driven, like Airbnb, for example, or Apple. I do think it has an impact on the end user as well.

I think it has an impact on how you can price your product, on how you can position it, on who you attract, on how people talk about your product, so on virality of your product. It's harder to measure because UI is usually something that when you change in a product, you can't necessarily change as a test. And it's not just within the product. There's a lot of brand that comes around and a lot of product perception that comes with it. But it's easy to measure the other way around.

If you look at a product today that have What I would call a shitty experience, really, that have, a really poor UI, like a lot of different UI. It's just, it's very inconsistent. It's hard to use. There's stuff popping in there and there. Automatically, as an end user, you lose your trust with that product. You think this is a bit spammy. This looks like, you know, the internet of the 2000. Like, This is not something I'm okay with today.

And I think you're a lot less likely to actually put your money into this product. And that's something that we did test and that we can test because we see that product, especially in the B2B world, I think in SaaS products, you really see that, right? Like the one that I'm doing really well right now, like the Miro of this world, they have really good UX and they have really good UI as well.

I think naturally you're going to think that's a less valuable car because it's not taken care of as much. I think it's slightly similar, you open up an app and then whether consciously, probably us as product people, we do this a bit more consciously, but non product people, they do it subconsciously.

I believe they look at it and there's just something that feels good about a product that is crafted well, that is well thought out, where the UI and everything is, the details have been implemented well, I think when you open an app like that, it just makes you want to, sometimes makes you want to give their money just because it's, it seems like a quality product.

And if, if they're putting so much care in the UI, you also think they're probably also putting a lot of care in whatever is behind it versus. It's like getting on a plane and the plane is dirty. We think what else is not working on this plane? Are they also not taking care of, the engines or the landing gear or whatever it may be? Lea, we've been here for an hour, roughly, and the tradition that I have on the podcast is at the end, I ask every person the exact same two questions.

I'm going to change the second one a little bit for you because you're not a designer. That's fine. But the first one is the same. It's

a what is one action that you think led to your success that perhaps in one way or another separated you from some of your peers?

So I really actively try to learn and absorb as much as I could from other places. And then I would come back to my workplace with some sort of a plan about how can we action some of that, I would always make something with it as in. think I remember the first time I came across the whole Northstar concept. I was at a product tax conference in Lisbon many years ago.

And it wasn't necessarily the first time I came across it, but it was the first time that I thought it was explained in a way where it made sense to me and I could see the value in it. And so I came back to work and start thinking, how can we apply that? How can we shape that in a way that would help our business grow?

In a way, doing that, I think, enabled me to be maybe not ahead of the trend, but like following up, keeping up with the trend and trying to constantly bring my place of work in that shape of constant innovation and new ways of working and adopting this thing as much as possible. looking back at it now, I had no understanding of that back then, the first place I worked in, in, in product for me was it wasn't a big company. It wasn't a big name.

And I didn't necessarily think that we were doing things well. I didn't necessarily think that we were great at best practice. And then looking back after everything I've experienced. We were actually pretty good. you know, And the main reason for that was because me and my team, everybody there was actively learning from everything else we were exposed to and trying to bring it back to a business.

And we had the space within the business as well to change things like we were empowered to actually put those things into place and change things and take those risks and that shaped my career. I think that accelerated my career a lot.

And that allowed me to be where I am today, where actually now I'm still learning a lot, of course, and I think I'm, I will be learning till the end of days, but I'm also in a place now where I can almost, teach and pass on my knowledge to like the next generation. And that, that's pretty cool. That's a nice, like circle of life situation.

The last question is, what are we not talking enough about when it comes to products?

because, we talk a lot about the psychology of the person and how we can impact that person's life and how we can fit with their life. So it's not really a number, but we think about it as something we need to get on our product. It's a measure of success, right? The more user you have on your product, the more engaged they are, the longer they're there. It's a measure of success to how good your product is.

And sometimes I think when you step back and you look out the impact of your product on the world it's astonishing. And it's not something I had the occasion to think about very often. But when I do, I'm always amazed at the fact that now tech has an impact on the world, it goes way beyond our own, number and KPIs and metric. We have an actual impact on the world. We're shaping the world of tomorrow. And that's very scary because there's very little regulation in what we do.

And I I can give you a very tangible example of that. I worked on a product that had 600 million people on it. 600 million people use a product. So when you make a decision, when you change an algorithm you're actually impacting 600 million people's life. I've worked on products that have a huge health impact where the product actively prevented people from committing suicide, for example. I've worked on products that shape people's careers.

So the entire workforce of one generation Somehow has been impacted by the decision that we made on how are we going to present the job? How are we going to build recommendation? How are we going to help someone build a career? And a lot of it is like algorithm based, is UX based it's tiny day to day decision. But if you look at it, if you step back of it, it actually impact people life in a way will have a butterfly effect on the entire world.

And in many way I think working in tech, working on product the way we do is extremely powerful and with that comes a certain level of responsibility as well. And that's not something we talk about very often because I don't even know that we realize how impactful our work is on the world really.