Working backward from winning & preparing for future tech innovation w/ Evan Welbourne #181

Here's the truth. Hiring remote engineers doesn't have to be expensive and time consuming. Our friends at Revello can help you find and hire top engineers based in Latin America in a matter of days. Just tell them what skills and experience you're looking for. They'll come through their network of over 400,000 pre-vetted engineers and give you a short list of curated candidates that meet your specific hiring requirements. You interview

and hire on your terms and only pay for as long as you keep the engineer. To learn more, visit revello.com forward slash ELC today and save $2,500 off your first hire. And of course with AI, there's something new every week. You could stay in those stages forever. You can just keep iterating and trying new things. But at some point, you have to kind of cut it off. You've got a time box it and just go with something that you know

will work. You're constantly also calibrating between the quality of what you're delivering and the time it takes you to deliver it. And so a lot of that problem kind of backs into this early stage of the process. We do want to do a good job of understanding opportunity. But you know, there's analysis paralysis. We don't want to just get stuck there. Hello and welcome to the engineering leadership podcast brought to you by ELC,

the engineering leadership community. I'm Jerry Lee, founder of ELC and I'm Patrick Gallagher and we're your host. Our show shares the most critical perspectives habits and examples of great software engineering leaders to help evolve leadership in the tech industry. In this episode, we discuss how to navigate the delicate balance between meeting current customer needs while also preparing for future tech trends and opportunities with Evan

Wellburn, head of AI and data at Sumsara. This is a delicate balance, right? In a rapidly evolving AI ML landscape, we not only have to build for now, but we also have to anticipate where the future is going and then simultaneously prepare to build for all of the possibilities of the future tech advancements. It's a dilemma. So in this conversation, we cover merging your future forward looking tech conversations with your working backward from customer success,

product development conversations. We talk about stage gates in AI ML product development as a mechanism to help you reduce risk and ensure your products deliver their intended impact. We also get into how Sumsara leverages smaller scale projects to inform their future direction. AI ML team composition, Evan shares tactics and strategies on introducing new technology and scientific research into your teams, you know, disrupting some of the

historical ways of doing things. And we also get into introducing AI into mission critical internal tools. Let me introduce you to Evan. Evan Wellburn is the head of AI and data at Sumsara, leading the organization's machine learning, computer vision, data science, and data analytics teams, as well as data engineering and data platform for the company.

He's a long standing career in both machine learning and IOT. Before Sumsara, Evan held various roles at Amazon, including the head of machine learning for Alexa Smart Home and manager of the computer vision research group. He also led research teams at Samsung and Nokia. Enjoy our conversation with Evan Wellburn. Evan, thank you so much for joining us on the show. How are you doing today? Happy Thursday.

So, you know, the closer we get to the end of the week, I feel like the more fun these conversations are because I feel like it gets kind of people into a reflective space. To set some context for people listening in, you and I were talking a few weeks back about this tension that exists in the role, this challenge where, you know, companies have a demand where they need to build for customers now and solve customer problems

in the moment. And also have to simultaneously think strategically about the future and two to three years from now, where do we want to go and how do we get there and how do we also set ourselves up for that part? And this tension can be really challenging, especially given the rapid pace of change in our industry and all of these things that can surprise us and evolve and emerge. And this is something that you spent a lot of time thinking about

and working at Sumsara. So, really excited to get your perspective here. I was wondering, maybe to kick off our conversation, if you could, from your point of view, introduce us to this dilemma and what you've observed in having to balance these challenges, like, how did this come up for you and what have you seen in this space so far? Yeah, for sure. So, just for some quick background context on Sumsara, I work at Sumsara.

Our focus is on physical operations. You can think of this as all of the operations that power the global economy, things like transportation, flow of goods and services, buildings, infrastructure, utilities and so on. And our mission is to improve the safety,

efficiency and sustainability of operations for our customers. And so at Sumsara, I lead AI and data and that includes applying machine learning, computer vision, data science, analytics to all kind of help solve customer problems as well as managing the data platform that takes all the data. This problem is particularly acute in this space because, of course, with AI, things are changing so fast, especially in the last couple of years.

Not only do we have to kind of work backwards from customer problems, but we have to keep an eye on all of the emerging solutions and keep evaluating, assessing what we can do, what we'll be able to do in two years and how we can apply it to solve these problems. So there's just a lot of uncertainty and ambiguity in that setting.

In general, when we're working on AI problems, a good example is our safety products, which is where we're using AI to apply data processing to inertial sensors like accelerometers and cameras to report on important safety events that are happening in the cab of a truck or in some other kind of vehicle. And these are things like drivers are distracted while they're driving and maybe looking at their phone instead of looking at the road or they're

breaking too harshly or turning too harshly. That's something that the driver may need to be coached on later. When we're working on those cases and other kinds of new features using AI at some SARA, we have this intense customer focus. We're always focused on customer success. And then as I mentioned, we talk a lot about working backwards from winning with the solutions for customers. One of the foundational problems when you're working

with AI is that a lot of the best product ideas turn out to be just intractable. They just don't work either because maybe you don't have the right training data for the models or for some other technical reason like the machine learning techniques you want to use just start working or maybe they're, you know, you do have something that would work, but it's

computationally infeasible right now for some reason. We've been iterating a lot over the last few years on our process grappling with this dichotomy between staying customer focused and also kind of working forward as it were from the technologies and the capability.

When you're talking about working backwards from customer problems or assessing some of the emerging and like evolving technology and how that might apply at SEMSAR in the future, like how do you separate those conversations or what does it look like to address like those different sort of like product building processes? Like how do you, how do you separate or merge those two different types of conversations? There's kind of a set of mechanisms that we use in both directions. So one direction

we start with is the working backwards direction. When we're working backwards, the most important thing is to start with the customer. We start with the customer problem and then the time honored technique at SEMSAR is to run a feedback loop, right? So we're a SaaS business to business company. We've got the advantage of having a pretty close relationship with a lot of customers. So for example, we've got regular calls with customers where we hear a lot of

feedback directly. We also take feedback from dedicated support staff through things like customer tickets, so on. And then when we're building a feature, we say we get it into beta, then we have, usually we have feedback through the product itself. There's some UX where customers can, you know, for example, give a thumbs up or thumbs down on things like

AI outputs that they like or that didn't quite match their expectation. But really for AI, the most important part of this running of feedback loop and working back from the customer is really working with data from the customer devices. That's what we used to drive the model testing. And then alongside that data, we also have a group of human data lablers that will actually manually review data and model predictions and decide whether

the AI was right or wrong. And so all those feedback channels combined, we kind of mechanistically speaking, we have a few sort of recurring meetings and sort of checkpoints that we track throughout a project. And so it's like every week, you know, if we did have any specific feedback from a customer that something was working great, that's great. If it wasn't working, then we zoom in on that specific example and we see what could be fixed, what

could we tweak here? We also look for the trends in the UX data feedback to see if there's anything indicating, you know, an opportunity. And then on a daily basis, we actually use that continuous daily feedback and manual review process to do things like compute accuracy metrics and then we'll portraying and tune models to measure accuracy throughout the

process. We're sort of tracking the metrics and tracking how we're doing and we're moving through these larger stage gates, things like, are we now to point where the AI product is good enough to test on live data in the background and won't break anything. And then the next stage is maybe like a small beta with a few early adopters and then it different

stages or sort of different stages of the roll out to the more customers. So overall, when the working backwards part, it's mainly about setting up the feedback loop that kind of keeps us aligned with the customer value. Yeah. And so when you say like work backwards from from winning, because then you mentioned that a couple different times, like, how do you define what winning looks like for your customers, like to start to anchor the problems or the products that you're going to build?

On the one hand, sometimes our customers have a very specific idea of what they want, or at least the problem that they need to solve. One product that we've launched in the last year is crash detection using the machine learning. And this is something that's really important to customers because when a crash happens, they've got to respond. Of course,

they've got to get help out to their driver. And then also because our devices that live in the vehicle like our, we have an AI dash cam as well as this vehicle gateway that's capturing sensor data, they capture a record of the crash so they can later review what went wrong. And then in the case of any legal dispute, sometimes that data is what actually exonerates the driver from fall. It's really important that they detect these crashes and

then they can kind of analyze them. Customers know that they need to sort of respond to crashes, detect them, and so on. But a key challenge when developing this kind of AI product is that it's not entirely clear what constitutes a crash. Say, from the point of view of an accelerometer, like what really, you know, articulates that definition, because there's all kinds of things to it's to a migration sensor that look like a crash could be an actual crash or

driving over a pothole, make the driver accidentally kicked it. You know, there's all kinds of things that happen in the field. And so, you know, when we're kind of trying to calibrate that crash detection products to the customer need, one part for sure is like starting with this very clear need for the customer to respond and do something in that set of example

cases where it's like, all right, we've got to catch cases like this. The general strategy for us is to, you know, identify the challenges that assess what are the techniques we're going to use to solve the problem, choose one of them and kind of build a prototype. And then it's, you know, within that feedback, iteratively testing the model with the customers and looking at their data to improve over time. And we're evaluating the model performance and we're

moving it through these stage gates closer. And so we're not only just sort of, you know, marching up the precision or the accuracy chart, we're also calibrating to like what the customer actually needs. And some of that, you know, early on, just looking at examples and later on, it's customers actually looking at the feedback and we're kind of shifting

left or right or narrowing depending on what customers actually want. So it's, it's even that kind of product fit is happening at the same time as we're iterating on the model. As a US company, hiring remote engineers can be time consuming, expensive and frustrating. You could hire freelancers, but you don't know how much they'll focus on your business. Out sourcing to dev shops, risks, control over who works on your projects and expensive long-term

contracts. Or you could look overseas, but working across lots of time zones can really slow things down. Enter Revello. Your gateway to a talent network of over 400,000 pre-vetted engineers in Latin America who are proficient in English, work in your time zone and are dedicated exclusively to your business. Revello simplifies the hiring process, enables quick payroll in local currencies and provides expert staffing support. There are no big up front contracts. You only pay for each month

that you decide to keep your developers employed. With Revello, you're in complete control. You get to decide who to hire, what to offer and you get to decide how long to keep them on your team. To learn more, visit revello.com forward slash ELC today and save $2,500 off your first hire.

I was wondering if you talk a little bit more about the stage gates because in a couple of conversations we've had with folks in our community, as folks have been experimenting and building this and integrating with the different products or features that they're building out within the company, there's almost always this concern of how do we know if it's working or how do we verify accuracy in the maturity of this product and having false positive results from the different

AI models and stuff that we use. When you're talking about stage gates, this seems like a very mature verification process to ensure that the product that you're building is trustworthy but also successfully producing the types of results that you're looking for at the outset. Tell us a little more about the stage gates. Absolutely. We've established a pretty well-structured life cycle for AI products. We actually call that a machine learning maturity framework and that's something that tracks

AI product development through this series of stages. I'll talk about the maturity framework and there's a second series of stage gates that I'll talk about that are important for very specific tactical projects and how we interface with other engineering teams. In terms of the ML maturity framework and this is the way that we really communicate where we're at with the machine learning project with the science part of it. There's these stages so that the first stage

is really the idea stage where we've got an idea for how to solve a customer problem. In the case of this craft detection feature last year, it was self-supervised learning. It's a new trendy machine learning technique that seemed to be well-setted to this problem and then after the idea stage,

there's feasibility stage which is where we're kind of vetting the idea. We're actually usually vetting like many different possible ideas and trying to select one that works but at this feasibility stage, it's like, okay, we confirmed that it's a feasible approach and we believe that we could take it further. And then after the feasibility stage, we're just going to the basic stage. This is

the scrappy prototype. We're just proving that it's worth going further. It's a little bit more of the proof of concept and that's where we're really moving into the more intensive feedback. That's where we're really trying to start up with looking at customer data and as much as we can, get that customer feedback in because that's what's going to help us find that that's fit the customer experience. Then after the basic stage, that's actually kind of a long stage and then

the next stage is the operational stage. It's like, we've built the product out enough so that it's ready for customer testing. We may start beta tests with a group of our close customers, our early adopters. And then after that operational stage, we get the measured stage and that's where we've validated it. We've continued rolling it out incrementally. We've got it validated on a large population. We've got this critical mass of data telling us, all right, this product is actually

meeting the customer need. And then the last stage of sort of indicating where we're at in the machine learning process is the mature stage. And that's okay, we've GAED, but we're continuing to run the feedback loop all the time. The background just kind of maintained the model. There's almost always a way to improve it incrementally or adapt over time because sometimes customer needs

are changing over time. And so this maturity cycle is a great tool for communicating within the team about where we're at, where we want to get to, as well as outward about what to expect. If we're in the feasibility stage, we don't expect we're going to be in beta next week. We've built a lot more work to do before we even confident. The other thing across the stages, I mentioned before, but the biggest challenge is really moving through that basic stage where you've got the

prototype and it works well enough to provide some value to the customer. It's hard to do that because we don't have that much feedback yet. We got to kind of get the flywheel spinning. And we've done a lot to get the feedback loop started or started earlier, a lot of extra tooling and things like that. But just briefly, the other thing I would say is like, so this is kind of the

the maturity framework to just kind of convey the rough maturity of the product. We also, of course, have more typical product management around things like target dates or sign off on beta testing. And then the next stage of beta, then eventually GA, there's dates that we put on the calendar. And those are things that we will typically introduce those around the feasibility or the basic

stage where we start at the ballpark based on our speed of iteration. Maybe four months out, we'll be the time when we're ready to go to beta and then we'll add two months for the next stage of beta or however we estimate. And then those are dates that will pull in or push out depending on how the model is progressing. And sometimes we pull it in. If it's going well or we

might have to push it out if we need more time. But generally, we actually try to hit those dates even though there's a lot of uncertainty, we might work extra hard close to the next beta date just to try to snap to the timeline because of course, we're not operating in a vacuum. There's many other engineering teams in the organization that we have to collaborate with. Going from stage to stage, like what does the approval moment look like? Like is there a specific

conversation? Like how do you go from idea to feasibility or determine that feasibility is done removing the basic? Like what does that decision making process look like? There's a couple things about the maturity frameworks. On the one hand, the rigorous move between stages or steps is really that external project timeline where we have like a beta okay to go

go to beta sign off or like okay to GA. That's where we get everybody in the room including all of the product managers and partner engineering leaders and we sort of look at all the data and make the case that it's time to go to beta and then give me a have a debate and decide one way or another. That's the most rigorous steps in the process. The ML maturity framework for sure, we have guidance on how to move between stages but it's a little bit less rigorous because the

be honest like machine learning is kind of a squishy thing. So for example, when we're at the the basic stage okay we've got the prototype where it's getting better or gaining confidence and we're starting to get that feedback loop moving to get the operational stage. It kind of mirrors that that project timeline in the sense that we're kind of it's kind of like we're moving to beta but there's a lot of judgment involved. Is this really good enough for customers? Sometimes we have

enough customers. Sometimes a customer will even tell us you know I would use this if we've got it in their hands. But there's a lot of judgment around where we kind of set the bar for okay we're going to move from this basic stage to operational because we think of the crashes we detect by about a seven are correct and we believe they would represent what a customer wants. That seems good enough

to me. It seems good enough to start with these customers that we know well. We're often relying more on cut heuristics and judgment to move between the stages. Yeah. And so when you look at these different stages like where's the biggest like friction areas or like the parts that are like the opportunity for risk to be introduced where like things don't go well. Like where within like this this framework like does it have like kind of the greatest risk or opportunity for things to kind of

go wrong. Yeah there's a couple places I think that are worth calling out in the process and one is really there's the idea stage and then the feasibility stages next and those early stages are pretty high risk because it's so open and of course with AI there's something new every week you could stay in those stages forever. You can just keep iterating and trying new things and you probably will find better and better techniques. But at some point you have to kind of cut it off

you've got a time box it and just go with with something that you know will work. You're constantly also calibrating between you know the quality of what you're delivering and the time it takes you to deliver it. You know customers have a problem we have to solve right now and so a lot of that problem kind of backs into this early stage of the process. We do want to do a good job of

understanding opportunity but you know there's analysis paralysis. We don't want to just get stuck there and so there's like I say we both time box that stage and just try to go with what we have at the end of the time and I guess there's other approaches we take to just kind of staying up to date with different techniques and kind of vetting a collection. There's sort of always at hand

the tools that are ready for us and that what we trust and that helps as well. The other part of the the maturity cycle that's difficult is moving from the operational stage or the early beta to the measured stage which is you know more we're more confident in how the product is performing and we're rolling it out to a much larger population. Are we really confident that this is going to generalize well to all of the customers? And that's difficult at least at some SARA that's

difficult because there's so much diversity in our customer base. We have customers from every industry you know around the world and so what works for a customer in Canada may not work for the customer in Mexico you know or the the oil field customer is very different from taxi caps in New York City you know like there's a huge diversity even in this safety and transportation space.

There's also that risk of analysis paralysis. You can do this analysis on large data sets and kind of gains statistical confidence but we don't always have time to get the perfect analysis.

So there's there's judgment involved in that as well. Going back to the go-no-go within ideation and moving forward and how that can be sort of this space where like you're constantly researching and assessing on the emerging technology like what are like some of the signals that help give you confidence that moving forward with like a concept or an idea into the next stage like in like the feasibility testing part like what's some of that signal that helps give confidence to move forward

with a certain approach. So one part of it is just looking at standard machine learning metrics you know things like accuracy or model precision and recall and we look at those against what we expect would be required to deliver a good customer experience. There's a lot of these sort of standard practices in machine learning that we applied to build that confidence to shift stages.

Another part is you know as much as we can just sampling the customer experience where you know we even if we don't have the entire product implemented yet we can kind of emulate you know if we had the model make this prediction what would that look like to a customer and then just kind of review that from a UX perspective is that a satisfying experience is that really meet our bar.

And that's again that's the case where there's more judgment involved you can be more rigorous if you if you want that of course you can you know measure customer satisfaction scores you can you can administer surveys the internal like alpha testers within some sorry that's something that we do all the time we have people testing internally all of our products and that helps us kind of you know navigate that that early stage. Absolutely. So I know we launch right into working backwards

from a customer problem and like working backwards from winning. I wanted to like spend a little bit of time talking about some of the mechanisms when you talk about like anticipating and looking forward. And I think like we we've talked a little bit about like some of those things because we're talking about a lot of the different AI products and stuff that you've been building which I think inherently like in your world like you are spending a lot of time anticipating two to three years in

the future. So I guess to rewind a little bit like when it comes to taking the opposite approach so instead of walking backwards from customers winning now a certain to look forward what's possible with some of the emerging technology like what are some of those mechanisms that help you anticipate or prepare for future possibilities and and what is what are some of those things look like with you at Somsara. Sure so there's always something new in AI and we're always scanning the research

and the latest techniques for new opportunities. Every week we have these AI team meetups where we review some of the recent research or maybe like reading a paper and discussing together or reviewing techniques someone may give a presentation on an experiment they've done that's our mechanism we use to stay current I mean there's other kinds of mechanisms and people attend conferences to get the latest technology so there's there's one one part that is just staying up to date with

the technology which is kind of a big effort these days in AI. In that first phase we're sort of trying to match what we've learned from all these techniques all all of this paper review to a

particular problem and that's sort of a key part of that process. The other kind of of mechanisms we use are around long term planning and we're talking about that sort of looking out further ahead sort of a two to three year vision that's a lot harder and it's hard to see the path on implementation because everything in AI is changing you know we can't predict what it's going to

look like even two to three years ahead. It's fortunate that we have these smaller projects like the like the crash detection project for example because that's where we can test out these new techniques and then we can verify that they work in our domain and gain experience. So one of the things we do out it's sort of the macro scale is how to do a retro on each of these smaller project and maybe the project takes like five six months or nine months but we'll look back at that cadence and see

like what worked and what didn't work how would we adapt that going forward and we're kind of developing this toolkit or this this set of techniques that are coming out but that we we've developed validated that they they work in our in our space and we're looking forward to using

them on the next project for example. Yeah so I was going to ask like about like the future planning so like when you're looking at some of these small projects like do you sort of have like hypothesis for how they may inform future direction of larger scale products and things that you're

building. Absolutely yeah we we have a kind of a bias towards identifying techniques that will potentially be useful to kind of fulfill that big vision that we have for the long term when we're selecting algorithms that we might try at those early stages of the mm maturity cycle. We'll we'll buy us towards algorithms that look like a good match and we want to test them out and so building a smaller product or a smaller feature using one of those techniques is just a great

way to to try them out and verify. When it comes to planning especially for that like two to three hour year out vision like what does what does that planning then look like knowing that like the future may be a little bit more fuzzy like what does that planning conversation look like to also build in some of that flexibility for the future. It's a lot about establishing a vision and we can kind of see a trajectory about where AI is going to go and how that kind of maps our domain based

on our experience and the data we have. There's a lot of establishing a vision that we kind of we think we can get to. There's a big opportunity based on you know trends in AI research and an industry and and what we see internally with our customer demand and then you know the planning is pretty coarse in terms of that three years out we can break that down into certainly specific

products that we know we we're going to deliver for customers in the next year or two and then kind of a plan around the types of techniques that we want to apply in those projects that will help us learn and get there and then there is we do maintain a little bit of a bandwidth for these more

forward looking kind of research projects that are really the long stretch that will kind of tie things together and get us to the the big vision in a few years and so there's a balance and decomposing the future a few years to to meet a vision but it's a mix of practical projects and

a little bit of research with everything we've talked about so far both working backwards from what does winning look like and also then this process of starting to anticipate some of like the future capabilities and building that process like both kind of involve like practical aspects of

bringing a product to life and one of these elements that you mentioned sort of early on is like how some of the frameworks that you're you're applying here how some of this process like helps inform different stakeholders throughout this process and so I think like a specific audience I want to focus on like when it comes to stakeholder communication here because I think this has been an area a lot of folks in our community have shared is like a big friction point especially the more

in-depth the technical capabilities are the more challenging it can be to communicate to stakeholders like why the timeline is the way that it is so when you're thinking about some of these different elements like both sides of this spectrum for stakeholders maybe don't have technical depth like

what strategies or approaches have you found effective in conveying either the value or the progress of some of these like AI ML products yeah this is really a challenge and I actually find it's it's a big part of my job this kind of communication keeping stakeholders informed and in a

line so one of the approaches is just continuous education I actually do a lot of writing and presentations partly on new AI techniques and why we should care about them but also on you know whatever technique we're applying for a particular product and how it's going yeah on a monthly

actually on a weekly basis we provide updates to all of our stakeholders you know we actually have a meeting where we're reporting status and on a sort of monthly and certainly a quarterly basis we do some deeper dives and you know we invite people to come and read and discuss and and understand

where we're at really not just the metrics but kind of what's going on and what seems promising and where are the risks for a particular AI model or a product another thing that I think is important in sort of stakeholder management is surrounding metrics so if we're reporting a metric

that's really useful as long as people understand what the metric means and what it doesn't necessarily mean because it's the fastest way to report status and if we can say you know precision is X and recall is Y looks good signed off we're great but there's a lot more underneath the hood you know

sometimes even when the metrics are not looking so good we actually feel like we're on a good track we're actually doing fine so we've got to kind of find ways even in it in that executive status to unpack the real meeting and the real status behind the metrics and a lot of that comes to examples

of the customer experience so we can present certain metrics but then also give examples of like okay here's here's where it's actually working all of these important cases it's already working it's these corner cases where where maybe we have a problem at this early stage or here's

certain failure cases that we're trying to fix just kind of tying it back to the customer experience is is really important especially for stakeholders that may not have the the technical background of the time to really get into the technical details yeah so for for some more of these technically

complex challenges like are there any areas where there's like maybe common misinterpretation or common miscommunication like I guess higher risk areas of some of the things that you're reporting and then like what's a what's a more effective way to convey that and I guess like if there's any

specific storage or examples that you have around that that'd be awesome one challenge is that with certain kinds of machine learning techniques we don't really have a results there's a long lead time until we get to the results that people understand and so that there there tends to be this

challenge of keeping people warm and understanding that it's coming it's coming and that the project isn't installed we just don't have the results yet that was something that happened even with this crash detection project where we're exploring this new technique of self supervised learning there's

just a lot that's a technique where you're you're using a lot of data and you're sending a model off and saying basically go just go learn everything and then you come back and you try to extract the specific details that you want from the model kind of a cutting edge technique but it takes

a long time to get right and it takes a lot of time to get to the results that you can report to to stakeholders then they can say aha you know okay this this did work so part of that is I mean again going back to the education part of it was just educating people on like what this thing

was that we were doing this self supervised learning approach is very different from supervised learning or the traditional approaches we've used before and just explaining what we're doing and why we think it will work but then also kind of giving not not the metrics but giving some

insight into the process about like how much data we're using and how we're able to separate the data and meaningful ways using this process just to kind of show a little bit under the hood what's what's actually materializing that's something that was really helpful in the crash modeling project

and in general I find having that additional next layer of kind of simplified detail is really useful for keeping stakeholders in the loop the the dilemma you're talking about was like how to keep people warm and walk away like it's coming like especially for a long lead time like I can hear

the impatience like there's like an analog conversation in my head where I'm like I can just hear that impatient tone from somebody who's less familiar with the building part I want to pivot to a different element of this because I think what's so fun about this conversation is like some

sorry sits at this unique place where like the people using your product are so different and the like sources of data are so varied and so what you're doing in terms of like building all these different products like it covers such a wide area and and I think it's super unique in that

perspective but I think the other part is like because you've experimented with so many different applications of some of these different technologies like the teams that you've worked with I think are interesting so I wanted to kind of shift to this team composition conversation that you and I

talked about specifically like more and more like the products that people are focusing on are more of this like integrated AI products within their teams and so now all of a sudden teams are going to be working not just with just engineers product and design but there's going to be folks within

like the AI ML world PhDs scientists things like that and it's a little bit it's a different team composition and so I'd love to get a sense of like you know how do you work with AI teams when it's not just engineers or engineers kind of working across the stack of a product but it's also expert

scientists and you know people that maybe have been an academia that now are heavily involved like within software companies like tell us about like that experience and in what's worked well I mean it's hard but it's fun and I think that's one of the most rewarding aspects of my job like

bringing these diverse teams together to build something my primary approach across the board is really to build high ownership teams scientists and engineers and designers etc. you're all working together everyone has a role where they own something and we all buy into the same vision

we own it together there's a lot of bottoms up thinking and navigating that goes along with that you know let the team do the work and just and decide where to go in addition to kind of building these these high ownership teams I think it's also with AI in particular it's also a lot about

supporting curiosity right we do have the PhD scientists but we also have junior engineers pretty much everybody is interested in AI and wants to learn so we're creating a space where they can actually learn and experiment some of that is you know these weekly meetings where we have you

know reviews of papers everyone is invited to that is a welcoming atmosphere it's not too intense in terms of debates and so it's it creates kind of a safe space for for people to learn and also a safe space for people to fail because that's just a big part of innovation especially in the early

phase we try lots of things and I think don't work you know just creating that curious atmosphere is also something that's really supportive and kind of building this group that's going out into the dark all on the same journey this is another place where continuous sort of not just curiosity but

but learning and education show up because right not everyone has the same technical expertise when when maybe needed and so whether it's the weekly team meetings or in some sort of mechanistic say monthly sink we're all also taking the time to educate all the team members on like we

know what's going on with with the AI model and why one one particular element that you mentioned me that I thought was super interesting is sometimes both like populations of like maybe like the scientist or academic heavy academia elements and like maybe your full stack engineer elements like

can sometimes optimize for different things and you have to like align or motivate you know them to focus on like a different area and so as one of you share maybe the story about like the alignment and motivation maybe not dilemmas but like the alignment and motivation conversations maybe

that come into play and then how do you help shift or a lot realign or motivate folks to focus on areas maybe that they have less experience in or less focus on historically what usually happens is that the scientist wants to keep iterating on the model and making it better and the engineers

are ready to just let's get it out deliver it to the customer like get it in the next phase of beta so on and that's a really helpful tension I mean within the team to have that tension that's that represents what we need to do so I think in a nutshell my approach is to kind of bring them

closer together right and sometimes this means like engineers and scientists they're on the same team they're like literally they're working together to make this happen so that there's probably a few ways to do that besides right putting them literally on the same team it's kind of you

aligning them to the customer need and to what what we're trying to deliver for the customer and making sure everyone's kind of has the same goal working backwards from the same objective so that we can at any point in the process and say in the ML maturity cycle we can align on a judgment

about you know is the next step to actually roll it out to beta or do we actually need to improve the model from the perspective of the customer what is the right thing to do and what's kind of a series of tenants for a particular product we can often come to a rough agreement on like okay

yeah we actually should roll it out to the customer now we don't need to iterate again or there's other ways to sort of balance like you know with tooling that allows us to continually iterate even after we roll out that's another way to kind of keep the teams aligned like okay you're gonna keep

iterating on the model scientists but you're gonna do it after we launch there there's more to do there so some of it is kind of tooling allows allows for parallelization but then you know the most important part is kind of bringing the team closer together around the value we're delivering the

customer the other sort of dilemma that you and I talked about was the one where you know engineering teams who established like a certain way of doing things and then get introduced to maybe new technology or new ways of doing things like some things can kind of get stuck there so it's

already if you can introduce us to to that dilemma how do you help open up to different ways of doing or introducing new technologies or scientific functions so I keep part of the approach I've taken is to reserve time for understanding and in research in any project and right if we're mapping back

to this ML maturity cycle it corresponds with early stages of like idea and feasibility it leave if you need but only a few weeks that's kind of sacred space because it allows the team to to actually you know without too much external pressure explore opportunities and new technologies

things that you know we haven't tried before and one of the results of that phase is some feasibility results that will help convince various stakeholders whether it's with you know a other engineers within the team or product managers or other people it's a way to kind of show the promise of

a some new technique that's not been tried before in a way that's like that's understandable it's not just someone's passionate and interested about something that sounds cool it actually is shown to work so I would again say you know it's really important to provide that safe space for

trying new things and failing possibly because that's really what comes out with more trust worthy proof of a particular new technique the last question I want to ask before we jump into rapid fire questions you know because here we are at this point we're talking about some of the

ways to build for now and also prepare for future both internally for the organization and aligning on the types of products we want to build for customers in the future and I think there's this interesting element too of like incorporating different tools like into internal operations as well

so kind of like as a as a final question like having built a lot of tools for folks that are now incorporating these like into like really critical operations for their their companies like do you have any perspectives or insights around incorporating new AI ML tools into your operations

and balancing that with the reality of where the industry is heading like what are your thoughts there and looking towards the the future for integrating these types of tools yeah so there's a few practices I think are useful when you're introducing AI technology into mission critical kinds

of tools and it depends on a particular use case and you know the particular mission that you're supporting but one approach it's really about how you tune your AI models again depending on the use case you might want to start with a model that's tuned for high precision like when it makes any

prediction it's almost always going to be correct you don't want it to be wrong in that mission critical situation and then you kind of iterate from there so you you start with the safest place for your product and then over time you can learn with customer feedback and more data

how to tune the model to make it even better in some cases and maybe the other way it may be that you tune for high recall for example you never want to it in the case this was a case similar to the crash detection we actually don't want to miss crashes it's okay sometimes if we get them wrong

but we want really high recall there's a crash that happens we want to catch it and so you you may alternatively start with high recall and then iterate with customer feedback on making it you know more precise over time so that that's probably the first thing is just having some judgment

on how to tune the model before you take that first step of integrating it into your mission critical applications the other is just around clean operations for AI you know ML ops that there's a lot of work to be done to be able to continuously evaluate how your model is performing and then like

other software systems you'll need especially for mission critical applications to set up alarms and dashboards and metrics that show you what's going on in any particular time and of course they get tricky when you're talking about the latest AI technology things like large language

models that are kind of hard to measure over time and so there's all this new research and state of the art maybe open source tools that you might adopt to help you look at the metrics and build those dashboards to help your team you know have judgment of what to do next thank you for

helping us look towards the the future and and how to integrate some of this are you ready to jump into some rapid fire questions yeah all right so first question what are you reading or listening to right now it's a few years old but I'm I'm revisiting Stuart Russell's book it's called human

compatible Stuart Russell's computer science professor Berkeley author of textbooks among other things in this book he's he's framing AI as a technology with you know incredible benefits and also extreme risks and kind of like a key part of the thesis is that we should build AI systems to

be inherently uncertain about human preferences but also committed to pursue our objectives you know rather than going off on their own tangent and so this is a way to address the big AI alignment problem with like a never ending feedback it's rebuilt right into the system and so thinking

about that a lot it's it's great not only for building products but you know keeping AI safe in the long term love it what tool or methodology has had a big impact on you kind of an interesting one's more of a leadership methodology I actually went through a program a few years ago it's

called pathwise leadership it's actually a leadership practice that's based on graduate level psychology as well as some neuroscience even some psychoanalysis it's just really transformative for senior leaders you can allow you to get a much deeper understanding of yourself and some of

the unconscious perceptions behaviors that kind of tend to guide your decisions or limit relationship and also of course helps you in understanding other people and like kind of the nuanced psychology and in dynamics of workplace so that was a really valuable program to me just

kind of like understanding myself at a deeper level and how I'm making decisions based on you know my own preferences and habits wow that's incredible thinking about like the level of self-awareness there like from that part just the way you're describing that like sounds incredible especially

in your ability to understand the impact and relationships on others everything for senior leadership is so relationship based like so I imagine the most been super impactful what's a trend you're seeing or following that's been interesting or hasn't hit the mainstream yet I think an

interesting one is using large language models like cloud or chat GPT for aggregating and summarizing software project status that's something I've seen some people doing what we're starting to experiment with that here and you know the idea is like agile software development leaves paper

trailers here are tickets there's wikis you know other other co-reviews documents things like that and then video calls are often are you can transcribe them so you know we're experimenting with using say chat GPT to automatically suck in all these materials and then on a weekly basis you

know it can auto generate status report or even provide like a little Q&A interface about your own projects so that I know I actually have to be at every meeting I can just ask my chat bot you know what I need the impact on cross-functional stakeholders like so like you know maybe like partner

executives that like you're interfacing so like you're your VP of sales who is like when is this feature like is this feature on track for delivery and so then maybe they have like a little more self-service autonomy for sure that's my dream because you know a large large language model

can not only summarize all the status and give them the tactical details but they can also probably just they can ask questions about like what is self-supervised learning why should I care I probably answer those questions also last question Evan is there a quote or a mantra that you live

by or a quote that's been resonating with you right now well I'm a former Amazonian so I think a lot of Jeff Bezos quotes and one that resonates lately is to be stubborn on vision and flexible on details and so it's like we need to experiment we have to be willing to fail try new paths you know

it's really helpful to keep an open mind when we're working in a complex and ambiguous problem like a big diverse growing team an incredibly poetic way to conclude our conversation Evan thank you so much for guiding us into this tension between building for now and navigating the the future

and and what's going on there thank you so much for for joining us today excellent thank you so much for having me if you enjoyed the episode make sure that you click subscribe if you're listening on apple podcasts or follow if you're listening on Spotify and if you love the show we also have

a ton of other ways to stay involved with the engineering leadership community to stay up to date and learn more about all of our upcoming events our peer groups and other programs that are going on head to sfelc.com that's sfelc.com see you next time on the engineering leadership podcast