We're doing a special in-episode feature with our friends and sponsor Swarmia on engineering effectiveness. Is engineering a good investment? Is it helping the business move forward? elements make a great software engineering organization and how can we get there from our current situation. Stay tuned for later in the episode, Otto Hilska, founder and CEO at Swarmia, breaks down tactics to enhance your strategic decision-making.
unlock productivity, and drive better business outcomes. A good relationship is not about how well you work with each other, but how well you fight with each other. And you know how well can you disagree with each other. And that's the entire goal of a product trio. They shouldn't agree. They should look at it from different lenses. But then how do you have that argument and come to a consensus is like the hardest part of kind of managing it.
Hello and welcome to the Engineering Leadership Podcast brought to you by ELC, the engineering leadership community. I'm Jerry Lee, founder of ELC. And I'm Patrick Gallagher, and we're your hosts. Our show shares the most critical perspectives, habits and examples of great software engineering leaders to help evolve leadership in the tech industry.
This episode is all about product innovation, and we get into tons of different frameworks and practices with Samya Bhatnagar, chief product officer and co-founder at Involve AI. We cover the product trio. and how it enriches problem solving, different frameworks for healthy communication between engineering, product, and design. We also discuss the value priority score, gaining alignment, measuring success, and incorporating bottom-up decision making.
Let me introduce you to Samya. Before starting Involve AI, Samya co-founded a startup right out of high school in New Delhi, which focused on using technology to reduce gender-based abortions in India. She did her undergrad in computer science and went on to earn her master's in computer science from the University of California, Santa Cruz, with a focus on natural language processing. She's a Forbes 30 Under 30 alum.
a winner of the Stevie Gold Entrepreneur of the Year Award, recognized as the 50 most powerful women in tech by the National Diversity Council, and is on the list of top 100 women in technology by AI Technology Magazine. Enjoy our conversation. with Samya Bhatnagar. To kick off our conversation, Samya, just wanted to first off say welcome to the show. Thanks so much for joining us. Thank you. I'm so excited to be here. Absolutely. So we're here to talk the broad umbrella topic is
the product trio. To begin our conversation, before we get super deep into some of the frameworks and things that we wanted to dive into, I was wondering if you could Tell us first, like, first off, what is the product trio? And then I'd love to learn, like, where does your passion for this particular topic or space come from?
Yeah. So talking about the product trio, maybe I'll start the other way around. I'll talk about like the passion of, you know, why I feel this is so important. And then I'll go into the product trio. I feel context. of the problem space is something that a lot of product managers and a lot of product and engineering leaders seem to miss. Our entire reason for existence is to solution. and to find solutions to problems. So the minute we get a problem from any which way,
our natural inclination is to immediately jump into a solution and start figuring that out. And what I realized and why I'm so passionate about the product trio is... If you have not framed the problem correctly, no solution on this planet can be the correct solution. Einstein once said that if I had one hour, I would spend 99% on the problem and 1% on the solution. That's totally correct because you need to contextualize the problem.
contextualize the frame of reference, contextualize what audience you're building it for, and basically the validity of the problem in itself. Every problem is not a fire and every problem doesn't have to be solved right now. So the context around the problems... is something that isn't talked about enough.
We as product engineering and tech leaders, our entire job, and that's what is kind of like projected in every way, shape, or form is your job is the solution for things. Like here's a problem, go find a solution. So that's where... I think the passion comes from where a lot of the times I feel we're set up for failure because of like all of this subliminal messaging and organizations talking about you need to find a solution.
well, is the problem relevant? Should I even, like the first question is, should I even find a solution? Does this problem deserve a solutioning? So that's kind of, I think, where this... irreverent passion essentially comes from. And the product trio is essentially a way for getting multiple points of view at the same time. So you have a designer.
You have an engineer and you have somebody as a product manager, like somebody in the product space that essentially contribute towards the product trio. And the reason for this is... As an example, if somebody talks about, hey, every time, like for example, for Netflix or Hulu, they say, hey, every time I search for the horror genre, I can't find anything that I like.
The engineer will start thinking about querying. Are we getting the right material? Do we have the stuff in our database? That's his or her point of view. The designer will start thinking about, do they even know how to find it? They'll start thinking about, is the search bar appropriate? Are they able to even look for the right thing? And the product manager might start thinking about,
Do we have enough of this content? Are relationships good enough to be able to get the right kind of content that this user wants? What is the demographic of this user? So you're essentially... contributing multiple people to the same problem and nobody will look at the solution the same way. So essentially, you're really enriching the opportunity space around that problem. I think highlighting that last point is
for me, clarifies a lot of why it's so important to have those three points of view collaborating together so closely. Before we dive in a little bit deeper, what's the origin story behind like your interest and your passion and your curiosity around that space? The short answer of that is like falling on my face. If you're the only one as a product manager who is tasked.
with solutioning, you can never find the right solution. And I've seen that time and time again. And that could be in the way of, let's just say I had a user interview and I came up with a solution. I showed this to...
If it's an assembly line, let's say I show it to my designer, the designer would think about it completely differently. And in fact, a lot of the times completely invalidate my idea. And I go like this time or then if the designer and I collaborate and then I show the idea to my engineer.
like an assembly line, they'll probably say, this is not even feasible. What are you thinking? You're telling me to press a button and a rocket goes into space at NASA. That's not possible. So there was so much of the idea. The idea of making it frictionless started with friction. and essentially failing multiple times on trying to find a solution and not being able to find the optimal solution. The other reason why I think it became so valuable was no matter what you do alone, it won't...
impact the long-term business North Star metric. Your goal is not to release a feature. Your goal is to improve DAUs. Your goal is to increase stickiness or your goal is to increase partner revenue. is not just to get that feature out the door. And I think not having the collaboration of the three essentially pillars that would make that happen can never work. And I think the most important thing to reduce that friction is
Number one, we're all in the same page. It is not product design and engineering. A department is PD&E. like we're all in the same boat going towards the same you know destination it's the same goal and we all just need to collaborate so we don't call ourselves as like product division the design division we call ourselves as bdne My VP of engineering, Justin, is the partner in crime. I think the leaders have to be completely aligned and not have friction. And that's not an easy thing either.
because everyone's looking out for their own teams. So trying to find middle paths, being extremely collaborative, even when you don't want to be, is extremely important. And I keep talking about this thing that I read that you can be three kinds of... You can be the innovator, which is like a Tesla. They're like, I don't care what the users think, take it or leave it, you know, for the Tesla car or whatever. Then there's like a mechanic.
which is I would kind of think like Salesforce. They understand the problem of the user, but they don't take solutions. And the third is called the mother, which is like whatever the user says goes. And the biggest thing that I kept falling into is When you don't have the trio in place, the product manager always kind of starts going towards the mother aspect, that whatever the user says goes, because there's no force that's balancing out the product manager.
manager is the only person who's aware of the business metrics which is most commonly revenue So the pressure of if I say no to this customer for this feature, sales is going to lose revenue or we may not get this new prospect or this customer might churn is such a huge pressure that the product.
vertical alone takes the brunt of without like the engineer going like this is not possible this is going to break four other features or we're diverging from the roadmap or the designer saying you're like breaking the entire vision of what we're thinking about so that's what i started seeing with myself We had a customer that was a huge Fortune 50 customer. And unfortunately, we started barreling down the route of whatever you say we'll do.
80% of our revenue came from that customer. There was so much pressure to be able to manage. Then we had a pivot in the middle and we moved to Involve AI. I learned because I had my ml team as a part of the product trio and we had my app team and a designer so it became a
quad at that point and we had another customer similar to that that was again a huge part of our revenue but they were pushing us towards something that we didn't want to be like completely changing our product vision and we actually went ahead and said We're going to fire this customer. Wow. I know this doesn't make sense, but this is.
The long term losses are not worth the short term gain. So kind of learned from that. Well, I think that's a really powerful story. First off, framing the dilemma of how you can be three different types of companies. And if you skew towards one, the forces. or the different roles that sort of could be at risk of pulling you in certain sort of tendencies or directions and the risks of those. I think it's an interesting place to jump from as we transition to the conversation for...
how to optimize the product trio in this relationship. Because you've helped set the stage of like, okay, we're considering the structure of your company and how it's organized. We've got PD&E. We've got the need for...
engineering and product to be really aligned at the at the highest level, as well as within the product trio. So when you're thinking about like how to optimize all these different relationships, what's what's a good place to start? The good place to start, I think is the product roadmap. which is actually the foundation of everything. That's the foundational document. It is owned by product.
but collaborated on extensively. And complete transparency into strategic initiatives and what we're thinking and the scope is extremely important. Other thing that I feel to optimize for the trio is a very deep understanding of what the other person is doing.
a lot of empathy for the other role. So it is not easy. I know there's product and engineering is at loggerheads where they say that these guys keep increasing scope, whereas engineering, we go like, these guys just don't finish whatever we're giving them. It happens everywhere. It's very natural about heads. But I feel this.
concept of empathy is so important and it comes dumb down. Like if the leaders of engineering and product and design have empathy for each other, it'll trickle down into the teams. And what we started doing was we started showcasing the work that we're doing, even if... it's completely technical. So we started doing these very small 20-minute weekly or bi-weekly demo days. We have two pods in engineering, so essentially it is bi-weekly, but we get to see something out.
a weekly basis so engineering start shows us things like even if it's something like penetration testing like It has nothing to do with product. It's a technical roadmap item. If you don't showcase that this took so much work, we had X items that were pending. This is everything that needed to be changed. It was like this much of lift. There will never be empathy because then product will go like, what do these people do all day? Because there's no visibility and transparency there.
So showcasing everything that you're doing and then products start doing the same thing. We do this thing called the product roundup where we started showcasing and then design becomes a part of it. Here's everything we thought through. Here are the user tests we did. Here are the user interviews that happened. Here's the strategic initiatives. Here are the experiments that we are running. Here are the ones that we validated. Here are the ones that we invalidated. Like just talking to you.
all the trains of thought. We're building this product requirement document. Can you look at this? Is the feasibility, right? But showcasing that, because I think product and what they do is like a very, it's very cultish.
Anybody outside a product doesn't quite know what they do, but they think they're valuable in some way. Anybody who's not there, it just seems like this shiny thing. Oh, I'll just sit and think in a room all day and that's what we do. And there's this kind of like, it's a bad rap. But opening up that box and saying,
listen, here's everything that we're doing and kind of building that empathy that everyone's working hard is really important. And that's what started with the lessening of the friction. A lot of empathy, a lot of communication, and a lot of transparency about
everything that's happening in a very, very rigorous manner really helped. I think what you mentioned about the couching and under showing the work in a lot of ways, like the actual work, but also the thought process and the considerations around it helps.
really reveal sort of the invisible labor that goes into a lot of different things. And I think like the very specific one of if you're in engineering, like even demoing penetration testing or something like that to show that how that can be innocuous to people outside of the the org, but the workload.
accomplish that is huge. So I think that's really great. One quick question. So you shared a little bit of examples of engineering and product and design was sort of within this, this like showing the work theme. What does that look like for design? Design works very lock and stop with product. But the one thing that we found really helpful is getting design and front-end engineers really interconnected helped a lot to build that relationship as well.
What we don't want to do is consider design as a part of product. Design is doing like their own thing and design is collaborating with product and collaborating with front end. So usually what happens is a product is the first one. Like after we've had our user interviews, we at least try to have one interview a week.
That's like the minimum. It's not acceptable to not have an interview. So if by Wednesday, we don't have an interview lined up for the week, we will like, you know, gorilla market. to be able to get an interview for that week. So the product trio is on a lot of interviews. They're really helpful for contextualizing problems. My product is the first one who kind of takes a stab at it after taking design.
and generating feedback. So basically we get on an interview, we find an opportunity, which is increase performance. I'm going to make something up, like increase performance at the particular page.
And then there's a lot of considerations around it. So engineering comes up with their ideas of the opportunity, which is very, very engineering focused, which is like, can we implement lazy loading? Can we create a UI view that can make it easy? Design is thinking about it in terms of of how do we change around the UI, I'm proud of just thinking about like, do we even need this or not?
But once we have all of those considerations, product takes the first stab at it. And once that PRD is ready, like the PRD is very focused on key actions that we got. And then it just gets passed on to the design team. And design comes up with their own.
considerations, shows it to engineering and product. Once we're all in agreement, that's when it goes into user testing, which design runs again, which is like what's broken, what's not broken, how can we fix it? What product helps really in that.
particular situation is personas who are you building this for what is your demographic what are their needs and where engineering comes in is in terms of feasibility which is like can we even do this or not or can we change this up and the feasibility will become way lower and then design runs with user testing.
That's great. Related to the question of how some of these things are scoped and determined, I would love to shift the conversation towards your perspective on how you approach or determine what is... the right thing to build. And so I was wondering if you could share a little bit about what that looks like or how you think about that particular challenge within this space. We started with drawing sticks. Whoever gets the shortest one.
feature just never makes it. That's how the best problems are solved. That's great. Exactly. It's like a game of probability. Some people have worse luck than others. This was a very iterative process for us as well. Again, it goes back to our initial conversation about... contextualizing the problem space first. One problem does not need to be solved in one way. What I saw was actually the first solution is...
always the worst one. So the biggest problem usually comes with contextualizing the problem. You have everybody giving in so many ideas, but there's always one solution in everybody's mind. They always say, hey, we need to solve this this way. And that's 99% the worst way possible because everybody's first idea is the worst. There's three things that make up creativity.
novelty of the idea, diversity of your ideas, and then the fluency of your ideas. How many solutions can you throw out for the same problem? So what we all do first is we pick the opportunities base first, which is like, what is the opportunity that we can solve for? So we never build a roadmap based on features. We build it based on opportunities.
that there is an opportunity to increase the level of insights that a data analyst gets from our platform. And how we kind of measure that is it's based off of a framework. And we've created a pretty... internal framework. We saw the rice framework and the value effort score, nothing worked for us. So we kind of created something called as the value priority score. And the value priority score, again, we wanted to bake in.
the biggest pressure that product managers and product people get, which is like traction metrics, business metrics. So we've divided it into five segments. One is that this is going to help. bring more sales through innovation. Nobody has this feature. Second is this is going to help sales because there's competitive parity. The same things for customer success. This is going to help retention because it's an absolutely new feature and it's really innovative.
Or it is a retention feature for competitive parity. And the fifth one is enterprise readiness, where a B2B system, like the admin preferences and role, you know, creating roles, that's really important for us as well as SSO. stuff like that so that's what contributes towards the value part and the priority score essentially aligns with our strategic goals for that quarter
So we are a very OKR driven company. So whatever the OKRs are, we go like, which one of our OKRs will this impact? That becomes the value priority score to be able to actually surface what opportunity we want to focus on first. That's great. And what happens when you have to say no? We say no first. Tell us more about that. Well, it started with my dad, who is very genetically predisposed to say no to everything.
I'm just missing a mustache and a bald spot and I'm going to turn into my father. We are very rigorous with prioritization. And that is something that I feel me and my... VP of engineering, we have that in common, which is with everything that you say yes to, you're implicitly saying no to something else. So we have to be very careful with what is happening. There's a difference between saying no. in a way that still makes the person feel heard.
without offending someone i was gonna say what is your approach to that i would love to i would love to hear how you approach that because i i'm terrified of saying no so so there's a different way for internal for internal folks i just say no like there's nobody like
different, like we give them a reasoning as to why. But we've done a few things that I think have really improved our experience in saying no. The first one is we released what we call as a product portal. So our roadmap that is kind of quote unquote public. is it's not based off of timelines. We call it now, next, future. There's no timeline associated with it, but essentially a priority, which is more important for people to kind of look at.
So number one, that really helps with just getting visibility into what is getting worked on. When it is something that a customer requests for, that product portal is like a game changer because what our CSM does or what we do is we open up our portal and we say, okay. Tell me what the feature is. And then we write it out in front of the user. We don't have to say yes or no to anything, but we're basically just hearing them out. That's what they want to do.
In fact, we go a layer deeper, which is instead of calling that a feature request, we take it a little back or like, okay, let's talk about the problem. What are you not able to do with this? If there was a workaround, how are you working around this problem right now? And why will you lose if you don't get this feature? What will happen? So these are like the three questions that are really important for us. What is the urgency of this?
And what would the loss be? How are you working around it right now? And what is the actual problem? Like, I get it. I want a job down over here or a button over here. Great. What problem are you actually trying to solve? What is the outcome of this that you want to achieve for your business? business we see that most of the times that just you're getting hard and just venting saying this sucks and that sucks and everything sucks is
actually very, very therapeutic for the user that's frustrated. That's step one, which is you might not even have to go anything beyond like somebody just getting an escalation point to just talk about how everything hurts and everything bad. The second aspect is showing a log of it somewhere is really important. So instead of saying no, you can actually just add it to the future because the future is not defined in your roadmap.
You can showcase that to the customer and say, hey, as it starts coming up, we'll let you know. Usually our cycles are six months, always put an error rate of 30% to it. Well, we will get back to you in that time. But look, it's already here. in the future and that's all that's needed they just need to be heard because 90% of the times they have a workaround 90% of the times things really don't suck you're just frustrated because you'd rather find a better way of doing it
And then if it's actually valuable, you take it still through that value priority score. It might be a great opportunity, but you have to find that first. You need to validate whether that's important or not. A lot of the times when something actually holds merit... I go back to my other users because we do, we follow the continuous discovery framework. So we go back to our other users and actually like bring that up going like, is this a problem for you?
And if multiple people do say yes, then yes, this problem has merit. But if it's like a one-off, we won't change our product roadmap for that.
We're taking a quick break for a special feature deconstructing the topic of engineering effectiveness with our friends and sponsor, Swarmia. Otto Hilska, founder and CEO at Swarmia, discusses how to drive... better business outcomes, breaking down how tracking your engineering investment balance and time spent enhances strategic decision-making and unlocks productivity in your engineering organization.
The one way to improve business outcomes would be to track investment balance of your teams. So how much are we spending on building new things versus keeping the lights on? And the conversation you should be having is what kind of reactive work is affecting our... ability to focus on the more important things. What are the root causes and how can we fix that?
So you can segment and analyze your engineering work in multiple ways. One of the ways I like to do it is based on this balance framework that a former VP of engineering at Dropbox introduced. And the idea is to look at how much are you spending on
new things, which is what everyone expects from engineering? How much are you spending on improving existing things, which is a function of all the features that you built? You're going to still have to maintain them later. Then you have improving your own team's productivity. which is things like refactorings and those technical improvements and finally
there's the mandatory category of keeping the lights on, because if production environment goes down, we're going to go and fix it no matter what the feature priority was. When you look at these categories, you can see themes where keeping the lights on ends up... And if that's the case, well, that's a very simple result of taking technical debt for a very long time. And what you have to play with at that point is the 20% of your capacity. And that's likely going...
to keeping the business stakeholders happy and building a couple of features. But it's going to take the team forever to get out of that situation. So what the team can do, oftentimes this is a call for staffing. So if the team scope is too big... then they just need more people to realistically get out of it. They also need peace of mind to invest in the right things. So if there are things that need to be fixed and the root causes of those issues, then...
They just can't ship that many product features right now. maybe you can move some of the responsibilities to another team etc but these are the types of discussions that you should be having and in reality most organizations are just saying that well this team is not shipping anymore we're not getting the features that we wanted so leveling the discussion from that.
to looking at the different segments and how do we invest in these teams and how much headcount does any team need. That's a huge improvement and it allows the developers to get their voices heard as well because most likely they have an idea what that improvement project under that improving productivity.
be when you're able to focus on the right things first of all it's a huge motivation boost for the developers because everyone loves seeing their own software being used by someone else who is actually getting value from it If you get from investing 10% of your time on the right things to 40%, that's a 4x improvement in the team's ability to ship valuable things. And that's perfectly achievable. And we've seen that in multiple organizations.
takes in data from all the development tools, including Jira, Linear, GitHub, etc., combines it with survey data from your developers, introduces a bunch of feedback loops where developers see how did they just do with the feature that they shipped. Well, in case of business outcomes, we give you a view of your whole organization's time spent on different categories of work, like new things versus keeping the lights on.
We give you an idea of how much money we're actually spending on these things. So what does it cost for us to deliver this big UI redesign that goes across all of the different teams, for example, that allows you to have a better quality conversation about what is the ROI. on this initiative and all of that happens automatically
Swarmia helps everyone in your software organization, from the CTO to each engineer, systematically eliminate bottlenecks that slow you down. If you want to learn how Swarmia can drive better business outcomes, improve developer productivity, and enhance your... Going back to saying no.
If the default is no, how do you then determine we're going to pursue this? So the default is no until it's yes. If somebody says build this next week, the answer is no. If somebody says this should be a part of the next release. no matter how small it is, which is like, oh, I just get logged out. Can you increase session time for me to not get logged out? The default is no, because even these small things take up space. So it's literally like, you know, you have a glass jar.
To be able to optimize it, you first fill it with rocks, then you fill it with pebbles, and then you fill it with sand. If you fill it with sand first, there's no space for the rocks. The default for even these small things, just because they say, hey, it's really small. it's still no. However, if somebody has given us something for a future consideration, that is never no. That answer is, let us put this to our framework, validate it from the market.
And then I get back to you. The important thing is external pressure always deviates you from the product vision, which is like, you know what you're building six months out most. People, especially in go-to-market, might not know it. There's a real thing around recency bias. Just because you lost one account for the lack of this one feature that no one else wanted, because it hurts bad, you think that this happens really often.
But actually, that's why, like, especially for go-to-market teams, getting them to record close lost reasons is so important. Because you can go back and say, okay, this feature seems to be, you're saying this is bad. Let's go back to the close loss reasons in your CRM to see how many deals do you actually lose to it? Is it one? Is it three? Is it nine? It's 20.
If it's 20, of course we're going to take it in. But if it's one, no way. At least lose three deals in succession because of the feature, and then we'll think about it. That's such a good point, because I think the recency bias thing comes up a lot, because it's especially if it's if it's like a closed deal, like
that is emotional, especially for the people that are there, like being told no, like, sorry, we're not going with your product or service or whatever is, especially if you're, you know, very attached to having built the product itself, and like you care about it, that's that's really hard. And so tracking the deal.
And then helping figure out, is this actually recency bias or is there more statistical relevance for this? I think it's really good. How do you define success for the product trio? So you've got...
the value priority score, you've got the default no, and then helping a defined framework for how to get to a yes. So then what does success look like? How do you think about that? So when we're thinking about thinking through a feature let's just say from an opportunity we found a solution that we all agree on and we're starting to build something out the most important thing that we're all aware of And again, this goes back to framing the problem is you're focused on the outcome.
What is the outcome you're trying to achieve in the real world? It's not just getting number of sessions. I hate when somebody says, this will increase number of logins. What does that mean for the business? It always has to translate back to traction metrics. direct or indirect manner, but it has to. So it's very important to be aligned on what outcome you want and how can you measure that outcome in a single KPI. Like it's very important to measure what matters and not everything about it.
Now, one of the most important features that we recently released, which actually has been extremely game-changing for us. So we're an AI platform, which... essentially releases a magic number to tell you with AI to tell you, hey, is this customer at risk or not? And we started seeing this really big issue with trust in data.
Because they're like, you're giving me this magic number that I should base my job off of. You know, how do I know the validity? How do I know if it's right or wrong? And explainability of AI is a problem that has not been solved by computer science. So we were in like a really tough situation. where we're like, okay, now what should we do? So we released something that we call as the health score ingredients, which is kind of like the credit score for AI.
So you know how you have your credit karma page that says your credit score is 650. Here are the factors that are contributing to it. You don't have enough accounts that are cycling. You don't have a long enough credit history, like whatever they tell you. But it kind of builds a little bit of trust into that.
So we literally took inspiration from that page and built that up because it's something that users identify with. They see their credit score very regularly. So we thought that might translate over in terms of trust. Measuring a metric for trust. is really, really hard. Totally. So we started with a primary and a secondary metric associated with it. So the primary metric that we measured was... Something like just that feature NPS, that do you think that this is valuable?
Do you have more trust in data? So that was a very qualitative measure that we got. And the secondary measure that we put, because again, we wanted to take that back to the business metrics. And we're like, well, if you have trust and data, that means maybe the feedback, the bad feedback that we might get about a health score not being accurate, because it's different than what you think. And that's the entire goal. You think about the health score differently. You don't know enough.
about it and the data is telling you something else, that's actually a good thing. It's not a bad thing. So we started measuring the declines in people giving us the, is this prediction accurate? Like a no as feedback. because that translated to business metrics and retention values. Measuring one or two things that encompasses everything in terms of like how you want to move the business is extremely important and being aligned on the outcome. The outcome was...
Can we build trust and data? We didn't even measure how often do people come to this page. We didn't measure how often people are downloading a report versus session time on it versus like how long people are on the page. Because that's the mistake that I think.
a lot of product people make, which is you measure everything that you can measure. And tools have made it very easy for us. You just, you know, highlighted, you hotspotted, you're like, we want to know heat maps and how much are they scrolling?
So what outcome? It doesn't matter, but you like knowing, but then what? And a lot of the times that might misdirect you. But if somebody is on the page for a very long time, does that mean that you've built trust in the data? Not necessarily. So measure what... impacts the outcome is the most important.
I want to dive into this example a little bit more because I think the whole creating the development process and determining to create that trust score in addition to that with the whole feature set. What was the conversation like within the product trio here to... align towards that pathway with the different perspectives? What did it look like to bring everybody together and to gain alignment towards that decision? I feel like this was one of the most contentious.
feature sets for us because it was very abstract. Like building trust, like I want to build more trust is something so abstract and can be taken in so many different ways. So it started with, we actually get led our ML. part of the trio take the lead on this because we're like
What are the ways in which we can explain it? And it started off as we can show them a shop analysis. And I was like, people don't even know what a shop analysis is. I don't know what a shop analysis is. So it started with them taking the lead and actually presenting. three to five options for us. But the really good part is the design team and the ML team worked parallelly without showing each other.
each other's work so the design team came up with like this credit score kind of page which was really interesting and then the ml team looked at it from How would I run a Python script? Like if I had to go into like, they started talking about F1 scores and stuff like that. Like how do I actually test for accuracy? So we came up with two extremely different looking solutions. And the product team actually got involved in figuring out how they would tie to each other.
Because though both were correct and the problem of explaining AI is just so hard because what you're taking is something like so technical and putting it in business context. So value of product was to actually understand. understand the user and what they want to get out of this to build trust. So what we realized is we did so many interviews to figure out there were like three questions that people were asking.
Number one, I think something completely, this score, I feel this customer is doing great. You guys are telling me this customer is at risk. Why should I believe you? Secondly, you gave me this health score. What is it based off of? Like, what does it even make? Like, okay, I agree with it, but why should I agree with this? Can you give me details about it? And third was, what are the key drivers that are impacting it? I feel like logins are great.
but what is actually impacting it? So those were the questions that just kept coming up again and again. So we had to figure out based on that business context, how to marry those two solutions together. So that was kind of like the conversation. And we actually had to release this in beta.
to kind of iterate on it as well. So we knew right off the bat that this will be one of those features that we can just throw out into the abyss. We have to get beta customers. We have to see how they work with it. And actually, the first iteration of the solution, we actually got a lot.
lot of negative feedback about trying to show them benchmarks they said this is like too much i don't even know what this means because we started showing them median values and stuff like that we got a lot of negative feedback and we dialed it back even more we're like okay even this is too complex let's dial this back. And then finally, at this point, we're at a 91% feature NPS and our number of no's on the prediction accuracy went down by almost 31%.
Such an interesting story, especially for, as you said, like something so ambiguous, like how do you develop trust in the explainability of an ML algorithm, like in a way that, you know, the layman who is... is dialing into that page can can really understand like, to me, that's amazing. And also the questions you were asking, like, I definitely jumped to like characterization of, you know, I have like my young teenage cousins and like different like toddlers that I know asking why.
Why? What is it? And then like the teenager is like a rebellious what? Like, what do you mean? Like, what is this? Which like totally makes sense. So it's like when you can answer those questions, it simplifies. It was definitely the five why principle that we were kind of getting hit with. And a lot I'm like, do you really know about the foreign model? Like, why are you asking me so many questions? Like, that's my question. Like, why are you asking me so many questions? But...
It was definitely very difficult for us. Like now in hindsight. When we think about it, we go like, yeah, of course, that's the way you should solve it. But it was a good four month, almost killing each other kind of like process. But that's like another part that I feel so grateful for. I feel like a product trio is working well.
when they can have a really animated, almost like angry discussions with each other and then walk away like nothing happened and just take it in their stride. So I feel comfort is built with uncomfortable conversations.
A good relationship is not about how well you work with each other, but how well you fight with each other. And you know, how well can you disagree with each other? That kind of just, you know, put light on the fact that, yeah, if they wouldn't have been like at each other's throats, we probably...
wouldn't have had such a good solution that first shows that you have a bunch of passionate people who want to solve the same problem but they're looking at it from different lenses and that's the entire goal of a product trio they shouldn't agree they should look at it from different lenses but then how do you have that argument and come to a consensus.
is like the hardest part of kind of managing it. Like you just illustrated like the most beautiful style of conflict, the one that ends up in a better place and everybody feels in the like the most integrity with their relationships afterwards. How do you do that? Like, is there a frame? that you'll introduce? Is there a magic secret to help create that type of discourse within your teams? How do we get the product trio all in that healthy state of conflict?
I've tried two ways and I think both of them work. It depends very much upon your leadership style. I can switch to both. I've tried both and it's very mood dependent for me. The first one I think is having a mediator.
Like when you know that things are getting really, really heated, it's very important like for someone to translate, even if they're saying the same words, but translate it for somebody else in the same tone or paraphrasing it for somebody else really helps. Especially when... things are getting like really out of hand so we decided like having like a third party mediated whether it's me or someone else who's not a part of the trio even though we don't have contacts that's not the
point of this the point is to be able to facilitate a good discussion until they don't get used to facilitating it themselves a new product trio will always always fight more because they're just not used to each other's working style and they shouldn't be
So having that mediator helped a lot. The second thing I did try to do, like there's some people who are very, very passionate anyway, and there's just a personality diet. So giving them a framework to be able to talk to each other with, I think is... really helpful you know the kind of structure that i run with and i read it somewhere and i can't remember it now but really helped was talk about here is what i saw it should be able to be recorded on a cctv camera
Like you don't put, you don't embellish, you don't say anything. It's hard facts that a CCTV would kind of record. Then you say, my story around this is, or I hypothesize that. abc is happening and whatever i disagree with you on or what i feel that you're doing is And it could be like if two people are combating each other and something, I feel that you are completely disregarding point A, B, and C. And just talking about it in that context.
helps remove the emotion and the passion out of it because you're like going back from a very emotional brain to like you've got an alligator brain and a logical brain so instead of snapping at each other you go back little bit of a logical thinking because you're forcing a framework on someone that dissipates the passion out of it while still keeping the important meat of the material in mind. That's such a great framework.
Because I think I definitely err towards wanting to avoid conflict, but I oftentimes feel like I can get to a better state by being able to rely on something like this. So like, here's what I saw. My story around this is this. What I feel is this. It's like, it is very dis... disarming and it helps get to kind of the root of the discontent or the disagreement in a way that is not you stink i hate you and or i think you're dumb and and from a much more productive place
Like, how stupid are you? I'm like, you can't do that. No. I know we only have a few minutes left. I wanted to ask you about bottoms up decision making, because a lot of this, like what I've noticed in our conversation is this, a lot of this is very driven by the quote unquote, the product trio, driven from the bottom up in all of this.
and really empowering people to facilitate these conversations themselves and to really come to decisions themselves and filter those up. So can you share a little bit about your perspective or approach around incorporating bottoms-up decision-making? First of all is everybody should feel that they have the autonomy.
to make decisions. For bottom-up decision-making, I use the concept of task-relevant maturity very, very deeply. So if someone's new on the team, somebody's three months in, will I let them? make their own autonomous decisions? Absolutely not. But will I guide them through the process? So there's a stage for the first three to six months on my team that is complete micromanagement.
I will be on every call. You will shadow every call. You will learn. You will ask questions. After that, you run every call, whatever, whether it's a user interview or a decision. You run with it, but you run it by me so that I'm sure that you're comfortable. And then when you reach a stage where I'm... comfortable, and I feel your task relevant maturity is high, then yes, you have the ability to make those decisions. And that is a very important part that I set with
in the onboarding itself, then no, bottoms up decision making does not start on day one. It'll start when I think you're ready to make it. So that's like one very important thing. This helps me as a leader have more control and have more trust in the decisions that are getting made. So usually if I'm having a product trio, I would always have at least one person whose task relevant maturity is high so that they can help manage the rest of the people with lower task relevant maturity.
And the other part is delegation. So I have this framework of importance, urgency, skill. So when I have an opportunity, high importance, high urgency, then only people with high skills should be able to be managing it. whole product trio needs to have high task relevant maturity. If it's urgency, low, important, high, I'll probably put one person with a medium or somewhat on a borderline of low and medium task relevant majority on it so that they can learn.
Because even though it's like an important thing, you have time and time allows for iteration and making mistakes. And then if the importance is low and urgency low, the whole trio can be brand new because there's not a lot of risk in making that mistake. That helps me really manage decisions. And I think it's really important.
to set expectation that this is democracy until it's not so there are certain situations where yes i do have a veto card and i have an absolute like no no this will not happen card and everybody's real well aware of that, that I will come in when I really see fit. And if it's a really critical, like, for example, if we're doing the raise or something like that.
I just cannot afford mistakes. So yes, I will play my co-founder and CPO veto card there. And they're very comfortable with that. And that expectation is there. Sameh, we have a few minutes left. We've got a couple of rapid fire questions to wrap up our conversation. if you are ready to jump into those. Okay. Perfect. Okay. What are you reading or listening to right now?
I am reading a book called Competitive Strategy. And I just finished a book called Dopamine Nation yesterday. Dopamine Nation. It's really good. I think it helps with building better products because you know what creates... pathways to light up. What tool or methodology has had a big impact on you?
The biggest tool that has had a really big impact on me has been Miro. I started using Miro a lot. And the other thing has been pen and paper, just like literally drawing out what I'm thinking. And methodology, I feel like I... am a methodology at this point. I just love framework so much. So any framework that can benefit the business, yep, I'm all for it. Would love to invite you to a feature framework jam. We have a few people that are definitely like archives of really fun frameworks.
you go toe to toe with the best of them. So we would love to have you involved in that. Awesome. What is a trend that you're seeing or following that's interesting or hasn't hit the mainstream yet? I think BLG is really cool. It hasn't quite hit mainstream yet, especially from an awareness part of the funnel. Like usually PLG is a very sales focused motion.
But I feel like using PLG to improve awareness of your product is something really, really cool, but I don't think enough people are using it yet. And just for the folks that are unfamiliar with the abbreviation, is that product-led growth? That is correct. Two final questions. What has been one of the most meaningful in-person experiences that you've had with your team, company, or otherwise? Doesn't have to be the most important one, but one of the ones that have been really meaningful to you.
I just went a month or two ago to this conference called the She League and I was speaking there. And it is a group, it's a conference and it was like 200, 300 women. And it is a conference for women of color specifically. And you know how to navigate your career. And it was one of the most. meaningful experiences for me because I've never been to a conference where I've come out learning more and just like learning that there's so much courage and so much strength around you.
And you're all in this boat together. I think negativity has a way of alienating you. And it was such a meaningful experience where everybody's going through the same things you are. You feel very connected and you don't feel alone and isolated. So yeah. To walk away with the outcome of a sense of courage and strength is such a powerful phenomenon. That's great.
Final question to wrap us all up, Samia. Is there a quote or a mantra that you live by or a quote that's really been resonating with you right now? Yes. So the one thing that I say again and again is it doesn't matter how far you have to go. It matters how far you've come. Your delta matters more than... distance to your goal. So if your delta is high,
That's all that matters. So people, a lot of the times, don't appreciate or cherish how far they've come. And they're just so worried about where they're going and how much time it's going to take. You just forget about the journey that you've taken. Gosh, such a great way to close. this conversation for especially when I think about like when you are stressed out or anxious about what's going on in your life to just consider
how far you've come to get to this point, even before you're tackling the challenges that are in front of you is a really great way to build in gratitude. Samia, thank you so much for your time and stories. Thank you. I appreciate your time. Mia says bye too. She's like, I'm finally back in France.
If you enjoyed the episode, make sure that you click subscribe if you're listening on Apple Podcasts or follow if you're listening on Spotify. And if you love the show, we also have a ton of other ways to stay involved with the engineering leadership community to stay up to date and learn more about. all of our upcoming events, our peer groups and other programs that are going on, head to SFELC.com. That's SFELC.com. See you next time on the Engineering Leadership Podcast.