From Pixels to Tokens: UX Is Not Enough Anymore

Hi everyone, My name is Patrick Akil and if you're interested in how they build AI features over at Microsoft Copilot, this episode is for you because I'm joined by Stephanie Fisher, Senior Product Manager over at Microsoft Copilot. And a lot has changed the expectations for engineers and data scientists, the focus on delivery as fast as possible and doing that through many, many experiments. I really enjoyed this conversation.

So enjoy. I was wondering because you you mentioned that you were PM before anything copilot related. Yes. What has changed now that Copilot is there with regards to PM? Everything, literally everything is start. Yeah, such a good start. I think like what has changed is what makes a good product, therefore what has changed it,

what makes a good product manager, right? Because you have to adapt your skills to whatever you want your product to be. And then also just, I think the variety of things that you're now doing as APM is so different than it was before. Yeah. So I don't know which aspect. Why? Why did the product change that much from your perspective? So, so I was in, I was building office products before, right? I'm a product manager for worst copilot right now.

Before that I was a product manager for like other office applications in general. And I feel that back then everything was about like a great product was a great UX,

right? Really good user flow. For example, one of the products that I worked on just before Copilot's game was, for example, multilingual proofing, which meant if I'm in Outlook and I'm a multilingual speaker, like I am like, that means that I'm sometimes writing in English. That means that I'm sometimes writing in Dutch, right? And it's really annoying if my spelling suggestions, if I'm writing a, a e-mail that's mixed English in Dutch.

So that's squiggles of the Dutch, for example, you get red lines everywhere. So I was building a product related to that multilingual proofing. And then like the thing that I thought through as a product manager was just simply OK, what's you know, how will a user use it? What is a good user flow? What is the best UX? You know, what does the system have to do? We're talking pre copilot and pre AI about deterministic systems as well, right?

The system just had to work. And as a product manager, my job was to create an excellent spec. And in that spec I had to take all the edge cases into account. And that's what I did. And then a good product was a great user flow, but also a product that followed the spec to AT, right? Right now, I think in AI we're moving towards outputs like it's all about output quality as opposed to UX. So we're moving kind of from pixels to tokens, right?

And no good UI right now in AI roles can make up for the fact that your output quality isn't good. So a great product now is not a great user flow, but a great product is, is your output accurate?

Is your output relevant customer satisfaction? And that also means that as APM, your job becomes really different because I'm going to have to constantly test the prompts that my scientists are using, are building on top of the models and see, oh, how's the output quality on these types of documents? How's the output quality on

other types of documents? Like, And then I communicate back to my scientists and say, hey, listen, guys, like this is, this is kind of the golden standard for looking for this type of outputs or this is really bad, right? But then again, like the systems right now we're building are non deterministic like like we can't plan for the outputs, right. We can build great problems if we don't know what the output

will be ever. So just kind of that whole testing, experimenting mindset is something that's we never did before AI world. Interesting. With deterministic stuff you could aim for like let's say your idea of perfection and then it works over and over and over again. Do you think you can still do that in this non deterministic way or will there always be edge cases that you just can't accommodate?

For I think there will always be edge cases you can't accommodate for like you can make a great problems, obviously. And we have all these evaluations from the science side as well where we can measure, oh, how well is this model or this problem doing in terms of accuracy, relevancy, all these types of things. But then again, like you never know indeed, because it's non deterministic. So yeah, so, so, so you can't, can't plan for everything.

How do, how do you decide then if you're experimenting with, let's say a new AI feature, when to go live? Because the only experience I had was we're working towards an outcome and the outcome was making people more productive. And even if our AI solution was only X amount of percent accurate, I was like, let's do user testing.

Let's measure if we actually contribute towards that outcome. And we also have to test kind of the engagement and adoption and stuff like that. But even if we're not 95% accurate, but we're hitting 80% and people are already significantly more productive, we can go live and like release that. What is your thought on releasing new AI features? I think it's a good point. Like as a like I do have together with my scientists, just certain baselines that we want to hit in terms of metrics,

right? Like we do want the outputs to be above certain accuracy. However, like the point that you're touching on is really interesting because what you also don't know now in using AI products is how people are actually going to use it, right? And the only way that we can know that is by bringing it out to users as fast as possible. So I think as BMS, we're really and like as feature crews in general, we're really going to have to switch towards a more OK

experimental mindset. Like my entire work right now is building hypotheses super quickly, trying to validate those hypotheses and like throwing them away and failing fast and like modifying those hypotheses. So for me the process or or or the time towards shipping at least to internal users has become much smaller just because I want to see, OK, but like we can test everything within our own environment, but then we we can see all the accuracy, relevancy, all those kind of metrics.

However, I've no idea how this will behave in a real world indeed because of the non deterministic aspect of it. So, so like my whole mindset of me mindset of me and my crew is to ship as soon as possible and then see real world usage because only then also you're going to encounter those those edge cases and moments where it doesn't work, right. Yeah. Do you always ship kind of in this phased approach to internal users first before releasing to the wider public? Yes, yeah, yeah.

That's how we've always done it at Microsoft. So we have, yeah, First we ship to internal dog food, dog food users. So those are users who signed up for. I want the earliest earliest features. We call them dog food users. We call them dog food users, then we shift to the interactive Microsoft and then we start shipping to worlds towards external users. But what is really just like anecdotally, what's really interesting as well as before

AI, we yeah, yeah, yeah, yeah. Before our AI world, whenever we were building a product like before we would even release it to those dog food users, we would have a certain confidence in a product. It was almost ready for external users as well. So the only reason why we would then ship to internal users is just to, you know, catch some bugs, etcetera, you know, see if there are certain parts of the

products that aren't working. But at least back then, we had almost like a 9095% confidence, OK, the product is good enough to ship to external users. However, like talking again about that experimental mindset, that is not something that I want to do anymore. I don't want to kind of siloed together with my scientist, you know, try to optimize before we're even going out to actual users. Instead, I think we have to go to actual users as soon as

possible. However, like imagine the shift within our organization that needed to happen. Like, I think a year ago I was working on a pretty controversial AI product, and my idea was, OK, you know, let's just ship it as soon as possible just in order to validate our riskiest hypothesis as soon as possible.

But imagine you're a dog food user and, you know, you're used to seeing stable products, and you're used to seeing products as well that are meant to, you know, shift to actual customers. And then suddenly you're getting something that is like, to be frank, pretty unfinished. I was discussing with my software engineers, okay, you know, what is the bar? What is the bar in this new AI role to ship to dog food users? And we called it the non embarrassment bar.

So the moment that there are not like it doesn't have to be great. It absolutely doesn't have to be great, but if we're not embarrassed, let's just ship it to users, internal users and see what's happening.

Okay, how was the feedback? In the. Beginning. Yeah, yeah, I don't see was yeah, yeah, yeah. Because you mentioned you're a user and you think, oh, gosh, these products are actually going out to users, you know, to external users because that's what you think is happening. Interesting. Yeah. So I got so much negative feedback of people being really concerned. You know, you can't shift this to to user.

And I'm like, yeah, yeah, yeah, I know guys, you know, but this is just part of the new experimentation products. Yeah. The process. I, I had Sam Bacon and he's the CPO of Veed, which is like a really big, I think it's the biggest online video editing tool out in the market. And he says what distinguishes us, and we used Google as an example, is that I can talk to customers like this and we try and do that every week. How easy is it for you to not just talk to dog food users but

actually customers in the end? Real customers I, I think the great thing about working at such a large corporations that we have a ton of structures in place where we can actually talk to enterprise users like real life enterprise users. So we have programs in place where a ton of enterprise users from enterprise companies sign up and they say, hey, we want to develop these AI products together with Microsoft.

So you know we're delivering features to them really early so it can they can give early feedback. So in that sense it is pretty easy. At the same time, like the hardest, like like again, the hardest thing is when I was building products before AI and again, it was very much based on user flow and UX, etcetera. I could go to platforms like user testing. There are platforms that are user testing, right? And just people all around the world can sign up.

It was very easy because you could just like show a flow and think, man, people would comment on it, right? And then you would have at least a sense on how it would resonate with users. However, right now, in a world where it's so much about output quality, you can't do that anymore, right? Just because people can't. Yeah, yeah, yeah. And people can't really test it

out anymore. So. So it so it is hard, but there are avenues simply because we have the benefit of being a large corporate and with our name, Yeah. I. Think it makes sense. Earlier you, you talked about kind of this change in behaviour, right? First delivering very high quality and also going to your dog food users with the assumption that this is basically a formality and it's already there. We're just trying to iron things out. Yeah, in case we miss things. Yeah.

But also from your perspective as APM it it sounds like you made this shift like no problem at all and you're having a blast. Which is amazing. I grew up in, I, I don't have much PM experience and I did all of a sudden become responsible for an application and that also had a Gen. AI component. And because I think I have this engineering background, I always like just shipping and then seeing, OK, do we meet our use case? Do we meet whatever we aim to

achieve? And if that's with 80% or if that's with 100%, I don't mind to be honest. And we can move on and deliver value in other aspects. So I've always kind of had that. But how is that shift then for you? I think I also maybe had the benefit that I was a software engineer before being APM and I was only APMI think for a year, 1, 1/2 year before the whole AI

stuff hits. So I didn't have like a ton of habits or something like that, right. And what I think is kind of ingrained in engineers, which I noticed with myself and also with my team, like, yeah, shipping thinks it's fun, but experimenting is also like so much fun, right? So I could very like, I could very much like, easily tap into that by myself, but also like getting my engineers excited about just running experiments,

looking at scorecards. And that data came pretty naturally, to be honest, just because, like, it's, it daps into some inherent curiosity, I think, which worked really well. That's amazing. Yeah. Yeah, yeah. Has your expectation of software engineers, let's start with software engineers first changed throughout kind of this experience? Yes, yeah, very much I think. I think as a like a, a good, this has always been my vision about product management.

I think very traditionally, like really back in the days, it used to be pretty hierarchical in terms of how you build products. So the product manager sits in like an how do you say the ivory tower and you know, defines what needs to happen and writes A spec and then throws that over to design and design makes some sort of engineer design makes some sort of design, and then we throw that to engineering and engineering just needs to execute, in my opinion, from like PM.

This is like not how you shoot PM anymore. I really see myself as APM that's facilitating all the other people in my team to be brilliant at what they do. And I really think product building is like an equal job between design, engineering and products. And now with the whole shift towards AI, where there's indeed like tons more experimentation, etcetera.

I always also expect from my engineers that they step up their game in terms of experimentation, but also step up their game into what are those experiments that we can run, AKA what would resonate for a user in your opinion as a software engineer, right? So I'm really trying to stimulate them to also think about the user and the product, etcetera, because I also don't think that should be my role anymore. I have a pretty large team,

right? I think I have like 30 software engineers, scientists, designers. It's really big. Yeah. OK. And that means I, I can't make all the decisions myself. You know, I really, yeah, really scary. Yes. And I also don't want to do that because, like, it's not the product. It's not going to be the best product if I'm by myself going to think through what they, what the product should look like. So I'm really trying to stimulate my engineers and my designers into product thinking,

into users thinking. Thankfully they are really great product thinkers. Like all my engineers have a really good understanding of customers, etcetera. But there's also some work to do. So I invite them over to user interviews and those kind of things just so that they can get a better product sense. But because, yeah, I think they're going to need to step up in that regard as well. And they're for for everyone, by

the way. I think PMS are going to need to learn how to code better and design better.

Designers are going to need to learn how to code. You know, engineers are going to need to learn how to PM. Like, it's becoming so much more. Don't worry. So much more intertwined everything. Yeah. Yeah, what, when is the right time to involve engineering? Because I, I only have experimented with this. I did a kick off new milestone, engineering stakeholders all in one room. Let's discuss kind of the pain

points and the problems. Yeah. And the feedback I got from some people, really enthusiastic, really excited from the beginning. Other people like this is too vague for me, like there's no structure yet. We don't know what we're doing. Yeah. I feel like it's a waste of kind of my energies and effort. Yeah, interesting. So then I'm like, OK, next milestone, I, I just do a raise of hands who wants to be there from the beginning?

But then I think it's valuable, but I don't want to put this on people like it's always been this balance for me. Yeah, I think it is a balance like, but when I was a software engineer, one of the things that I used my PM for most is really understanding, oh, why am I taking up this user story again? Because like, if you're an engineer, you're involved so much in the reason you're kind of forgetting the big picture,

right? And I think that is the main, the main difficulty that my engineers have. And I constantly have to remind myself like, oh, I need to communicate to my engineers like, why we're actually doing this? Like what are we contributing to and what are they directly contributing to, not only in terms of product, but in terms of success, You know, so we obviously want to increase engagement, activation, like all those sorts of metrics, like, and how does that trickle down

to them? Because it gives them kind of a sense of, oh, This is why I'm working so hard on this. Yeah. Purpose. My sense is I'd love to involve engineers from the beginning when I have an idea of, hey, this is the user problem, right? So I've talked to a bunch of users and kind of consolidating all of that information. Then I'm presenting that out to my engineers. I'm like, OK, this is the direction that I'm thinking in, but I'm also open towards thinking like what you guys think.

And the same thing with designers and scientists as well, by the way. I think we didn't work with scientists so much before, but they're so important right now in our work. So it's an entirely different type of colic that we're going to have to involve from the

start. Yeah. Yeah. Are they ingrained and kind of part of the engineering team? Yes, it is a separate team with like a separate manager and things like that. But I usually work in we call ADRI Trios. DRI stands for designated responsible individual, something like that, whatever, another three word acronym. Anyway, it means there is someone responsible for every discipline. So I'm obvious, I'm whatever product I'm building, I'm always putting together APM.

That's me, a scientist, scientist and engineer and a designer. And with the four of us, we're responsible for the products. Like doesn't matter. Like I don't care if you're an engineer, like if something falls into your place and I'm not there or I'm busy or whatsoever, you should solve it. Like we're really like with the four of us responsible for the products. And all those the same people kind of over and over again. Or do you switch with regards to the people in those little

teams? Yeah. So it's a, it's a good question. So as I mentioned, I have a team of 3035 engineers approximately and within that we're building separate like all different products, right. So it's usually like well 35 / 4. So that's about like 8-8 different products are being built or something like that. So it's like constantly like different people and working on. Different products and you're the only product person that's there, yeah. Wow, I wish I had to help but I don't.

Yes, that's super. Interesting. Like I've been in organizations where there's teams to the smallest level, 3-4 engineers, then there's also PM there. And then there's this hierarchy also of PMS, and they have one person that's product manager, and they also use terminologies of like product manager and product owner, kind of, Right?

Yeah. And then in the end, there's one product manager without a team that just has a team of product people and they have smaller teams and like, how does this work? Yeah, Yeah. That is interesting. Yeah, we don't have that. And again, like I'm, but I'm outsourcing a lot of my engineers as well. I think like the PM role is very confluent as well. It means so many different things in many different organizations even need like product owner, product manager, project manager, etcetera.

For example, as APMI personally don't run ADO boards or something like that or sprints or whatsoever. Like my engineers are responsible for that. They can do probably much better than I can. I don't add any value to that personally. It's not like within my skill sets. So I'm very consciously trying to think like, oh, what can my engineers do? What can my designers do and what do I do? And that's always kind of through the lens of. Where do I add unique value now?

What are the decisions that you do take part of? Do you have cycles of retrospectives and plannings and forecasting or like OK ours, stuff like that? OK, ours, yes, that's a big one. That is a big one.

Yeah, yeah, obviously we do make OK ours I think every half year or something just to know as an organization or as like a a part of the product, you know, what are the metrics that I want to move? And then obviously I have a like within Word copilot, I'm responsible for making people understand documents better. Like there are different jobs to be done whenever a user is opening a Word document, right?

Like it is understanding a document or writing a document, one of these things, usually I'm responsible for that understanding part of it. So, so like I'm responsible for the road map within that part. So part of my job is just thinking about, OK, you know, what should AI help users with with this regards like upcoming year? So I've kind of a a backlog or things that I want to jobs to be to be done that I want to improve. And what were you asking again? What?

What kind of cycles do you take part in with the team interaction? Yeah. So I make OK Rs for that. And I have like a road map in my head. And usually within Microsoft, we had planning cycles every three months or something. So every three months we would decide, oh, this is what we're going to work on. After months, that doesn't work anymore. And I really like literally like it's too long. It is way too long. It's way too long like. Three months already is too

long. Yes, yeah, three months already luck, you know, like I mean you can make all the plans in the world that you want and then next week there's a new model being released or and and then you're done. I mean, I love this because always when people are like, where do you want to be in five years? I have no clue. Yeah, this is no clue. Insane. It is insane.

Like, yeah, like big companies, you know, they're just like created out of the ground and then they're like next week they're acquired, next week there's a new model. It's like insane to keep track of, like what is happening in the industry. So you can't plan for three months. So we're trying to plan with my future crew every month. Yeah. So every month I'm kind of looking through whatever priorities in my opinion to work on. Then we're just like prioritizing them.

And, and again, it might happen that during that month we're switching gears just because something else became more important. Yeah. I was wondering how you deal with bias because if I'm responsible for kind of understanding a document, I do that on a day-to-day. I open Word, this document, I have to understand it. I've done this so many times,

so, so many biases I feel like. And then also when you have 40 other people that contribute towards making this a better product or creating products and like improvements as part of that, how do you deal with bias and kind of remove that from people's way of working thinking? Well, I think we should do this. And that really anchors people.

Yeah, because that's such an interesting question because what you don't want as a culture within a company is that you always have like these visionaries and innovators, etcetera, and they can talk really well and they've really great ideas, right. So it's really easy for them to influence the product just based on their gut feeling or like their visionary thinking. But that's not how we should

build a product. Obviously in every product there's a bit of a bit of gut feeling, but that's not the way that we need to make decisions. So coming back to that controversial product that I was talking about, a ton of people didn't like that I was building it. Like absolutely not all the way to our leadership team. A ton of people didn't believe in it. The only way I could prove that it might have been a good idea

was to experiment. So like purely data-driven, looking at the metrics that we care about and only communicating from that angle. So, so that's the best way to remove bias in my opinion. Like everyone can have opinions and that's great. I also have opinions and biases and those have been disproven so many times through running experiments where you feel like, oh, this is for sure going to move the metric this or this way. And it doesn't at all. It does the opposite thing,

right? And that is super exciting. So we're doing that and we're actively communicating within our organization as well, whatever our findings are, how we're experimenting, etcetera. I like that a lot. I thought you were going to say because people that are stubborn, they like their own ideas, right? And they fall in love with their own ideas. So then when metrics and data comes and disproves it, it's like sad and disheartening. And the next one would be even

harder. But I like you're saying it's super. Exciting, yeah. That means we can do many more things. Exactly. Go ahead. Yeah. Like I have so many like dailies with my engineers. We're looking at scorecards and we're like, oh, wow, this is like completely opposite to what we were thinking. And then we're discussing like, how come that we had that gut feeling, right? And what, what might these numbers actually say? So what do we have to rewire in our brain?

So it's a really cool culture to build, kind of this experiment called experimentation culture. Yeah, Yeah. I can also see that it's very challenging because quantitative data is one thing. Yeah. And I think it's great to rely on that. And there's also then the qualitative part of it, like if you're doing things with AI, you're super trailblazing.

So even if the data might say things yes, like talking to people might say otherwise, it's very hard to balance both feedback points. You're yeah, you're totally right. Also because like users don't like users usually don't really know what they want, right, Obviously. And especially in AI world, it's super hard. Like what do I have to build for you next week? That week? I wouldn't even imagine. I would have to see something

and be like OK, just. Click exactly and you're like, ah, this works or this doesn't work, but it's so hard to rationalize. So the way that we now interact with like I'm finding those user problems, finding, finding the things that we need to work on really changed as well. Because like pre AI, again, I gave that example of multilingual proofing, right? Those were established.

Like Outlook, for example, was an established product and we had like we had a backlog of users, like of of things users wanted us to work on, but that was all documented really nicely. You know, customers would reach out to us through different platforms. You know, we knew which features they wanted. It was just a matter of prioritization basically. It's not even that long ago, and it feels like so it.

Feels so peaceful. Yes, yes, it feels like prehistoric time, you know, but now indeed, like you can't anymore. So so that also really changes how we're engaging with users. So what we now have to do is actually literally sit next to those users and asking, OK, just tell me about your workflows. Like what are you doing with words? And then just observing really well and then understanding like, oh, in these kind of moments in a workflow, we can inject AI in a very natural way.

And that's the only way that we can kind of have that discovery process now. But it's super hard, so difficult. Yeah, discovery teams, I think those are going to be more and more at the forefront. I think so actually. Kind of pairing with users observing, users interviewing, investigating. And. Everything is still an assumption and then indeed testing. And then we're just experimenting. Yeah, exactly. And that's also, I think, a

really big part of being APM. Like you can't get away with not

deeply understanding a user anymore, right? You really have to deeply understand your customers at this point. Yeah, yeah. I get that. When everyone, or at least when you're in an organization where there's a lot of PMS and experimentation as a culture is kind of coming to the forefront because we have to, we have to deliver and then validate and do that over and over again until we have kind of a proof point of value. Does data become an issue?

Do you have, for example, enough data or access to data to validate those assumptions? Or does it become a bottleneck because everyone needs it? Everyone needs it. Yeah, it is a bottleneck in a way that we just, I mean, we have so many users, but still it's not an infinite amount of users, right? So we can only run this very many experiments in parallel. And that's my main bottleneck right now. I've like 12 experiments that I want to run like at the same time, but my user base isn't

that big. Like I count AB test 12 different experiments, I still get statistically significant results right? Because we only care about when it's statistically really proven. So yeah, the bottleneck right now, which is a luxury position to have though like to want to run so many. Experiments I want to do 12 yes. Yeah, yeah, yeah.

But right now, at this point, I think we're even that far in our experimentation culture within my team that if I didn't have this as a bottleneck, I would be worried. I'm like, oh, Stephanie, are you really experimenting? You know enough? Yeah, yeah, yeah. At least this tells me, oh, there are so many hypotheses that we're trying to investigate so that that that's cool.

Yeah, that's so cool. Yeah. And you mentioned very strongly that you don't want to be the one person that comes up with all these experiments. So all these experiments are like a combination of all the disciplines coming together to try and solve this. Problem. Yeah, yeah, yeah, for sure. It's super cool. It's it's on every angle as well. It is designers wanting to try like a few things and see like how slightly different elements and design changes engagement, right?

These engineers who are, for example, we're now very deep into chaos testing. So engineers who are trying to, I don't know, mess with our house booth and see how that influences engagement. This is me trying to experiment with different like features or or whatsoever and science when they can do things. So yeah, it's a really

combination of things. It's still I think as ABM responsibility to kind of create order in that chaos of like of excitement and and think through, OK, this is a priority. This isn't priority. But yeah, it is sourced from everyone for sure. Have you ever had an experiment where there's a really good outcome, like things make sense, but from your product vision you're like, well, it's just not a good fit? Like it's not what we're trying

to achieve with this. I've had that and therefore it is really important as well before you run an experiment like, OK, what am I going to do with this with the outputs? Because it's also very natural to be like, let's let's experiment of everything, you know?

But if you're not going to take an actionable result based on that experiment just because indeed, like even if the outcome is this is great, it just goes against your product feeling that you just like, why are you running the experiments in general? It might be to gather information, and that could be valid, yes. But then it's something that we're discussing upfront with the engineers, like, hey, we can run this, but we're not going to make any decisions based on this.

It's just to gather data and information. Yeah. So that happens. Gotcha. Yeah, I was wondering because this is mainly on the metrics level. I was responsible for this Gen. AI part and then it was like,

OK, what are we aiming for with regards to accuracy with regards to like time reduced, I can have kind of a guesstimate metric, the dot on the horizon. So that's what I did. We're aiming for these things and then continuously we're evaluating if those metrics still make sense. How do you put your metrics kind of on the table? Because you mentioned with together with data science, we do have a few metrics that we're like striving for, but how do

they originate? How do they originate So, so it kind of boils back maybe to like OK Rs. Like, I mean, I don't love OK Rs in terms of like writing them simply because it's hard, right? But I think it's also like a tendency to like, it's very easy to say I'm writing, OK, ours and let's just let's just increase engagement and that's it, right? But you really have like what you have to do is get back to, oh, like, what am I really trying to do for a user, right?

And like a user's not necessarily helped with them clicking on something more, right? They're trying to achieve something else. So kind of writing that down kind of like and you said time to what, what were you trying to optimize time to? Making people more productive, so we're losing time spent. We're losing time spent. Yeah, exactly.

But then so that is a really great, I think overarching goals which you shoot right down and then kind of distilling it back to, oh, that actually means that is obviously the hard part. I found that it really depends on the maturity as well of your team on how, how good you can or, or which metrics you can define. So for example, at the beginning of Copilot, we're really deep, like diving deep on those

metrics for the first time. To be honest, in Copilot in the AI world, just because it was starting, we were only looking at, yeah, excavation and engagement, right. But now like 2 1/2 years in or two years in or something like that, we're starting to see more like, Oh yeah, OK. We're we can't go for like optimizing productivity to engagement. What is like, what is in between and how are we, how can we define that?

And it's just a brainstorming exercise, to be honest, between ton of people, usually between PMS and managers, like engineering managers, but it's hard. It's such a hard thing to to create. Yeah, I feel like. Listening to your story and kind of the the scale of the operation you're in, you're at least engaging with a lot of people.

How do you build kind of those relationships to have trust in the other people that they have kind of the same intentions or build the same level baseline of way of working and also

communication cultures might be different completely. I feel like you need relationships to thrive. How do you build? Those. With so many people. It's, it's a really interesting question also because like you were chatting before the podcast, like all my engineers are in Serbia, right, Which is they're physically distant from me. It's like an entirely different culture.

I've personally, and, and then like I'm also working with a ton of folks from the US, like, like none of my colleagues are in the Netherlands, right. So there's always. Like the only one. I'm literally the only one. Yeah, literally the only product organization in the Netherlands sits here. All of it, all of it. Like I personally always come from deeply understanding the

person up so well. So like eventually your colleagues and like the the engineers you're working with are also kind of your customers in a way, right? You can see them like that. And just like you're trying to deeply understand your customers, you have to deeply understand your engineers as well. So I'm always trying to come from kind of the human and go things like, oh, why are you like, why are you here? Why are you working in Microsoft, right?

Like what are you trying to achieve as an engineer? And how I also see responsibility for me as a PM, like how can I help you as an engineer reach your goals, right? And that means, so for example, I've had engineers that I knew had a passion in something that I couldn't execute it well. That's great. And it's like on me to make sure that if a project like that comes along, I can actually put them on that, right?

Or Hyatt engineers reaching out and saying, hey, Stephanie, I really want to improve my product thinking. And I sometimes have ideas, but I feel afraid of speaking up in a large group. So can you help me with that? And then we have weekly calls where I'm just listening and they're pitching product ideas to me. And I'm saying, oh, yeah, you know, you did this work as well. This doesn't work well. Oh, maybe you can do this.

So bottom line is really deeply trying to understand people helps building trust so much, I feel, because then you kind of mutually understand what they're coming from, where they're coming from. And especially also in organization of Microsoft. Like I, I work in words Copilot and then there are like 100 different products that Microsoft is building and I have to communicate with and, and align with those kind of product managers as well from different

products. And sometimes obviously you see clashes, but those clashes always are coming from the fact that's what that person wants to achieve. It's just simply different from what you're trying to achieve. However, do I really understand what you're trying to achieve the other person and why that is?

And if I understand that, then I can maybe find an angle or a way, you know, that's like that, that that's going to help both of us. But if I don't ask you, if I don't deeply understand you, I have no idea. And that's where you're getting these, like, surface level crashes, right? Yeah. I really recognize what you're saying and it's it's mainly what I see, what I'm just observing when I see two people trying to push their opinion on the other person and they're just going

past. Each other. Yeah, and they're not noticing. No, no, exactly. And they're not listening either. No one's asking a question. Yeah, exactly. And there's probably like a base of understanding like they're shoot, there's somewhere a base where they can understand each other, right. But someone communicates it like this and it doesn't resonate with the other. So indeed, there's push back, right?

Even though those people would step back a little bit, we would like find some common ground from where we can then, like collaborate together. Yeah. I feel like the people that are curious kind of innately also not just about products, but also about people where they're coming from, behaviour. You even mentioned kind of ambitions and goals. I think that's fantastic, especially as a person that's responsible for product and kind

of work that gets done. You are equipped to put people in positions to thrive and if that, with their ambition and that's perfect. Yeah, yeah, exactly. And like eventually that creates a better product as well. If I have happy engineers working on the thing that they're so passionate about, that's only good for the

product, right? So it is in my best interest as a product, as a person responsible for the product as well, that my engineers are as happy as possible and know what they're working on, they're excited what they're working on. It might be the most kind of undervalued statistics.

Happy engineer. Yeah, Yeah, genuinely. Yeah, maybe it should be an OK, honestly, I think generally like people's skills in that way and deeply understanding people's a very unvalued skill. And like we always think about, oh, you know, like is it product manager or even an engineer or you're, you're as good as like the lines of code that you write, etcetera, but you're not. I was a when I got hired as a software engineer. I had absolutely no software engineering experience in real life.

In in real life roles obviously like coding by myself. Otherwise I wouldn't. Get a job, otherwise I wouldn't get a job. But not in any real life systems. And, and so I got a software engineering job at Microsoft, right? And it wasn't because of like the brilliant code that I wrote, but it was because I, I was a customer facing software

engineer. It was because I was good at understanding a customer and really trying like trying to see where they were coming from and what we needed to build and kind of, you know, build a bridge between the customer and our engineering team and not because I was like a the 10X coder, you know? Yeah, yeah. Yeah, I feel like that is such a life skill. Like you had that as a person, you got a job as a software engineer because of that. You're completely leveraging

that and the role that you now. Have. If there's ever another role, it will still be valuable. Yeah, yeah, yeah, for sure. Yeah. It's like those transferable skills, right? Yeah, absolutely. As APN because you're also the only person in the Netherlands who challenges you like from a craft perspective. That's an excellent question. All my engineers, honestly. Yeah, yeah.

I love it when I'm like kind of building a first version of a spec and I'm only building fee 1 specs at this point just because like I know that whenever you do, you get that feeling sometimes that you're building a spec and you've put so much love and effort into it and you're like, this is perfect and you're sharing a doubt and you get so much feedback that yeah, so many holes that it almost feels personal, right? I cannot do. That. Yeah, really bad at that. Exactly.

So I'm like, I'm just like going to make like OK, fee 0.1 specs. I'm just going to send it out to all my engineers and my, and, and what I mentioned I'm very lucky to work with engineers who are so product and customer focused. They constantly challenge me. And then also kind of my my direct people that I work with are the engineering managers and the science managers. I have like often more than once a week I have meetings with them. We're just discussing products and ideas, etcetera.

Yeah. So those are the people. Yeah, I heavily rely on them. Yeah, yeah. Some organizations that I've seen, I go in and out have like this, OK, you are part and responsible from your team, but then you have alignments with people like from the same craft. So I thought you would have a structure like that as well. But I really like that the challenge like on a day-to-day, I feel like it should come from a team. I think so, to raise the bar together.

Yeah, exactly, exactly, exactly. And it doesn't become so siloed, right? You make like everyone accountable for what you're doing. And obviously I have other PMS, but I'm not, no, I'm not necessarily, they're not necessarily challenging me on a daily basis. It's not the same. Yeah, they also, they're also not so deep into like our aspect of the product that we're trying to build. And my engineers are right, yeah. That's super exciting. Yeah.

You said you were lucky that you have people like that. But I feel like, I mean, if you were my PM, kind of the way you think about product and also the importance that you put on kind of feeling that other people need to think alike and think about the product. It's kind of also the culture that you've cultivated. Thank you. Yeah, yeah. At least that's my assumption.

Yeah, that is it is. It is like I'm not trying to like my own horn, but I think it is something that I get back from my engineers and because I'm not like I've been product managing for not long, right? Like a few years, maybe 4 or 4 1/2 years, something like that. So on paper, I'm for sure not the best PM that's out there because like I am, I work with people who've been PM ING for 15 years or longer, right? So they're probably better in kind of the, the technicalities

of PM ING. At the same times, my engineers tell me that they really enjoy working with me indeed because of that reason, because I set a, I think someone even said at a certain point like, oh, you really put back the kind of the enjoyment in my work, right? And I think that's indeed what can really make you stand out as APM, just creating that culture. That's a beautiful compliment. It was such a good compliment. My job is done, you know, I can retire.

Yeah. Are there because you went from software engineering, now you're PM and now you've dealing with kind of a lot of disciplines. Is PM like the thing for you, the thing that makes you most happy? Thank you so much. I love Yeming. Yeah, it's so good. It's a bit of everything. It is a bit of everything, yeah. And even before, like my road to here has been super weird, right? I studied hospitality management. So I come from a hospitality management background.

I worked in hotels before moving into tech. I moved from hospitality management to tech by doing sales. So I was first doing sales and software engineering and PM ING. So I've seen like a whole bunch of weird stuff. And I think this is indeed like the perfect combination of like what I love about hospitality management is the whole customer focus, right? And I can totally use it here. What I love about engineering is that we're actually building products.

I can totally do that here as AVM. I write marketing and stuff like that. So kind of the sales element isn't there as well. Like it is such a perfect blend of everything. And then it makes me so happy to work with people who are so good at what they're doing, right? There's engineer scientists and designers. And as a PM, just like, wow, this is amazing, you know, and we can build cool stuff together. So yeah, this is like, I'm staying here. Nobody's kicking me out of PMA.

Yeah. I can truly relate to that experience. Like as PM, I feel like still you're kind of in a leadership position. And I truly believe that the way you behave, the way that every leader behaves, like it trickles down into whomever you engage with kind of in expectations behaviours. Do you own up to the problems that you've caused or like the the things that you've done, or do you shy away from them? Or do you like law or do you like, are you integral?

All of that matters. In the people that you engage with, Yeah, yeah, you're right. Like you're responsible for the culture. Like not only for the product, but for the culture as well. And the culture trickles back into the product to which is really cool. Yeah. And like, the great thing about PM ING right now is like, it's so exciting to be responsible to think of. Oh, you know, what should AI be like in five years? Indeed, as you were saying, like, it's trailblazing. Nobody.

Nobody knows the answer. And it's so cool to be responsible for thinking through that. Cool. Yeah, I have one last question. Yes, and that's more from a tooling perspective. Are there any tools that you've used AI related that make you more productive that kind of yes help you in your product?

Yes, absolutely. Which ones do you love? Which ones do I love? I love Cursor, to be honest. Yeah, I don't do a ton of five coding just to like make designs and prototypes really quickly because I think that's such an easier way to communicate your idea. Like back in the days as well, when I had a feature idea, right, and I needed to convince people I would probably need to engage with designers. Designers wouldn't need to make time to kind of get my thinking on paper.

Yeah, back. And forth is not right. Back and forth exactly. Maybe you would want to engage in engineers, make like a quick prototype or something like that. Well, how long would that take? Like maybe 4 weeks, six weeks or something like that. Now as APMI can just sit, sit behind my screen, five goes something in a day, shared up with people and people are like, oh, this is a good idea or not a good idea. Like the velocity is so much faster, which is really, really

cool. So for short cursor, for five coding, I've tried obviously like lovable V-0, like all these kind of things. SOPM it's also your job to try out like ton of things play around. Yeah, obviously I'm loose using ChatGPT a lot as well. Also for consolidating user interviews, doing research, you know, all of those things. New entropic opus model is really cool. I'm not sure if you tried it. I I read about. It it's so good, yeah, it's so

good as APM. You're also like, you have to be so model forward thinking, right? So you have to understand what each model can do and like how to grow prompts and how to get the most out of your models. So whatever works there, like can be, it can be claws, whatever. You're also like super pragmatic with regards to getting stuff. Done but. All these responsibilities and now also data science kind of coming on your plate. Is it too much like from a

product sense? Do you think it's getting too wide with regards to things you need to know or kind of need to at least have an idea of? One interesting question I

think, I think I had to, I had to shift a ton of my responsibilities to engineering and designers like making like small decisions, etcetera. I don't want to be responsible for them anymore. I'm just kind of observing and if it goes completely wrong direction, I'm kind of steering them. But like small decisions totally up to them. That makes that I have much more space for kind of experimenting and like vibe coding and understanding models etcetera. So it's indeed about being super

deliberate with your time. And like what I mentioned before, thinking about, oh, where do I add unique value, what is super relevant of my role and that is understanding the industry, the mark, the models, et cetera. And then all of those product decisions, I just trust other people to do that because I've created that trust with the team. Yeah, that's amazing. Yeah, I've really enjoyed this conversation.

I think you have super cool energy, super pragmatic, passionate about what you do. Thank. You so much, I really enjoyed it as well. Thank you for inviting me. Appreciate it. Of course, we're going to round it off here. Let us know in the comments section what you thought. Like the episode? If you did, it's free, so click the like button and we'll see you next time.