#203 - Building Effective and Thriving Machine Learning Teams - David Tan & Dave Colls

ML projects tend to fail in this common failure modes. 1 is what I call POC health. Another failure mode is that it's easy to get maybe the first thing out of the door, but then it's kind of stuck together with duct tape and sticky gum and it's caught to test or change. In my experience of working with data intensive ML initiatives or other technology specializations, I'd found that the ability to build the thing wasn't always the major success factor.

Often it was understanding what the right thing to build was, making sure that there was alignment between technical teams and and business teams and product teams on the right thing to build. You can't ML OPS your problems away just like how you can't dev OPS your problems away. ML OPS helps in a few ways, but ML OPS not going to run your tests for you. They're not going to talk to users and make sure you're writing the same right features or implementing the right features.

It's useful to think about the essential characteristics of of ML solutions that we have to do things at speed and scale that no human could. But they'll make mistakes. We wouldn't consider an ML solution if we knew the right answer every time. We'd write some rules instead. One of the quotes opening quotes in our book was from Edward Stemming saying a bad system would be a good person every

time. The whole thesis of Outlook is how do we create those systems to help teams build the right thing, build the thing right, and in a way that's right for people. Hello everyone, welcome back to another new episode of the Technician podcast. Today I have with me two David's very excited. One is David Tan, he is actually my ex colleague in thought books and the other one is David Coles. So they are the co-authors together with another one Ada

who couldn't join us today. They wrote a book titled Effective Machine Learning Teams, even though it's a machine learning as part of the book. But I think we are not going to cover solely just machine learning because I'm also not an ML expert, But we'll discuss things like how we can build effective machine learning teams, what kind of practices that we should be aware of and things like that. So welcome to the show.

I'll maybe mention Dave as David calls right, and David for David Tan. So welcome to the show guys. Yeah. Thanks for having us, Henry. Happy to be here. Thanks Henry, nice to be here.

Right. I always love to ask my guests maybe first to share a little bit of themselves by sharing your career turning points that you think we all can learn from that so any one of you can start first. Cool so I'm currently engineering manager in the Air Products group in zero and my career into tech hasn't always been straight. Like I was actually non-technical when I graduated from university.

So I was working in government for maybe 2-3 years and I got by accident handful and mouth disease and I had to like be quarantined for eight days. And from there I started like, you know, watching some YouTube videos about data analytics or coding and then, you know, try some tutorials and, you know, it's really cool. So then decided to, you know, join the programming boot camp, quit my job, did that.

And then giant ThoughtWorks. And yeah, learnt a lot about agile, about test driven development, refactoring, CICD, got into Mr. Engineering and so, you know, finally got into this exciting space of Mr. Engineering and OPS building Mr. products. So yeah, I have to thank this eight day quarantine back in the day for my career. Turning point? Yeah, maybe.

How about you? To to pick on an external theme like that, I'd probably choose deciding to play ultimate Frisbee early in my career where I met a number of people involved in the software scene, but in particular a thought worker who encouraged me to interview at ThoughtWorks. And when I think about turning points, I guess that kind of leads me into the fact that I started as a developer in

technical data intensive roles. I then, you know, looked for ways to make more impact in the leadership space. And when I joined ThoughtWorks, although I'd actually passed a coding interview, I joined as a project manager. Then I've worked for a number of years in agile transformations and organizational transformation before then

picking up the data practice. So I guess my career has turned from deeply technical to leadership positions and organizational design a number of times over that period. Thank you so much for sharing your story. Very interesting, right? So the kind of serendipity that could happen through our career, right? And what led us to where we are at the moment.

So today we're going to discuss topics from your book, Effective Machine Learning Teams. Maybe in the beginning, let's share what made you wrote this book. Yeah, this book started as a blog post actually. So back in 2019 we had wrapped up a project where it involved a lot of refactoring of data science code bases. So we had data science piece for wrote code in the Jupiter notebooks. No tests, a lot of lines.

You know, it's good like it was solving real business problem and then we had to kind of productionize that. And so we started writing about how we did that. How do we encapsulate code into functions? How do we have automated tests? So we are, you know, clear about that, about the quality. How do we do CICD for those

changes? And then we wrote a blog post and it kind of like exploded in back then Twitter, someone from the our community shared it and a lot of likes and it's like, OK, I think we're on to something here. There's this desire, I think in the data sense community or ML community to have more kind of engineering and robust practices. So, yeah, well then we started writing more and more about from the subsequent projects or what we did to kind of large scale ML training pipelines, large scale

high volume ML products. And you know, the practices that helped us build that reliably build ML products that are safe, you get fast feedback on your changes. And some of those things worked really well. So we wanted to share it more broadly with our readers. And that's just the engineering space. And I think Dave also has some other kind of yeah, aspects that

we know. We started with this thread only engineering practices for ML and then we discovered this whole kind of world of other practices that also help ML teams. Yeah. So yeah, in my experience of working with, I guess, data intensive ML initiatives or other technology specializations, I'd found that the ability to build the thing wasn't always the major success

factor. Often it was understanding what the right thing to build was, making sure that there was alignment between technical teams and and business teams and product teams on the right thing to build, which could be complex when people don't speak the same language. So, you know, what sort of processes and techniques can you use to build that alignment, especially when there's a lot of uncertainty both about the problem and the solution.

And then also because of the specialized nature of this work, it's often hard to build a team that can execute end to end. So then the factors around either building in a single team, how do you bring all of those different perspectives together? Or when you have to rely on multiple teams to get a piece of work out the door, what's the best way to achieve fast flow and high quality in a way that's sustainable for the people working in that environment?

So for me, those were the perspectives I wanted to bring to machine learning product development, which is an essentially multidisciplinary activity. You know, how do we do that better as teams? Right, when I write your book, I must admit that I was thinking that I was going to learn a lot of the technical things about specific ML, you know, projects and ML products and things like that. But actually you cover kind of

like a holistic approach. It's not just the engineering, but also like product delivery and things like that. And actually there are so many things that any software engineering team I believe could learn just by reading the book as well. I mean, putting aside some parts of the ML, but let's start maybe in the 1st place because I'm

sure many listeners here not coming from the ML background, what are the differences between, you know, ML engineering and the typical software engineering these days where people build websites, APIs and things like that? Yeah. But that's a great question.

And I think ML systems fundamentally it's orders of magnitude, I think harder and more complex than software engineering in some ways that of course software engineering is harder than we've got micro services, you've got distributed computing, you've got a lot of things like in itself it's an art and a whole discipline. ML itself, the difficulties manifest themselves in some common ways, like, you know, it's not just one service or one component.

Usually it's like a data pipeline dependencies upstream that you know is complex in itself. You've got your skill or compute requirements for your own ML workload, which is also can be done in many ways. You know, you can have a really large instance with really large memory, do everything there, or you know, at some point that breaks down. So how do you architect or make it simple to scale up large workloads? And then there's also the monitoring.

Like when you think about software products, typically your monitoring is kind of your 4 golden signals, right? Success rate, latency, things like that. You can do that for ML and you can be all green and good. But that says nothing about like the quality of the predictions that models are producing and then did the correctness of those. So it adds another layer of challenges that I think the ML OPS community have come to solve

in recent years. And then further, even further down, like, yes, we've built this product is giving accurate predictions to an extent. What is the kind of business impact or what levers on our Dios is it moving? So that's like kind of essentially the fundamental difference between the ML product and software engineering products. There's a small business diagram from Google where like ML is just like a little box and there's like infra, there's monitoring, there's data

quality, data skew. So a lot of moving parts and that's why we wanted to bring a lot of practices that kind of help you obtain the complexity in these different subcomponents. And when you look at how to manage the work as well, then I guess one of the characteristics of building an ML feature is there's a lot less certainty about how it's going to perform upfront.

And so often you have to integrate this exploratory data and analysis or building prototypes or architecture search as it might be described into the product delivery process as well. And then as you converge on a potential solution, then you're also dealing with another vector of change, which is the data and the world changing as well. As David said around monitoring, often that changes slowly, but sometimes we can see step changes.

COVID was a classic example with recommender systems moving from lowest price to highest to best availability and you know, ML models needing to catch up with that. So yeah, you both have less certainty and you have another vector of change to deal with when it comes to managing that work. Yeah. So thanks for sharing some of these complexities, right.

So definitely for people who may not be able to relate, I think typically from what I can see, right, there are so many data pipelines when you build ML, you probably need more than just a small data, right? Something like a medium data of big data and a kind of a compute that you need. Sometimes it's more like a distributed computing with large

scale kind of a pipelines. And I think typically what I can see as well, the feedback loop might be long in an ML project simply because you have this training kind of thing in the

1st place. Maybe let's clarify. When you say ML, I think we are not just associating it with LLM, right? Like these days people are crazy about LLM and maybe they associate ML now as LLM. So maybe a little bit of insight here, like what do you mean by

ML projects? Yeah, I guess we, yeah, Yeah, great observation, Henry. You know, was thinking, you know, as you talked about feedback cycles and big data and, and all of those elements, yes, those are the things we traditionally associate with supervised machine learning, which I guess is our sort of primary focus in the book, which might be anything from a binary classification problem. Like is this transaction a fraudulent transaction?

And, you know, to solve that with the supervised paradigm, we get a lot of historical data about transactions, their amount, the the age of the account, you know, maybe some information about the network around that account. And then we label those transactions whether they were fraudulent or not. That's actually a nice example of where the labels might be easy to get because we might get them from customer reports of fraud. But in general, that effort of labelling the data set can be

really intensive as well. So that's the sort of main paradigm we're looking at. But we think that, you know, looking through this lens of working with uncertainty, working with data, working with specializations, but there's a whole lot of other paradigms of MLAI that that might fit into that. So obviously working with a large language model in inference mode, that's still a

challenging environment. The supervised paradigm might apply to fine tuning or even training from scratch your own large learning model as well. But then beyond that sort of paradigm, we're also looking at initiatives that might be might use reinforcement learning, for instance, might. It's another approach to recommender systems. And this is where you might not need big data to start with.

You might not have historical data to start with, but you might actually build or tune your data set as you go. We might also look further afield to techniques like simulation or operations research for optimization. They share a lot of the same characteristics. Simulation is actually very similar to machine learning when it comes to productionizing it. It's just that instead of a learned model of the world, you're working with an explicit

model of the world. But you still need to go through feature engineering ability to monitor and assess business impact and align to stakeholder expectations too. So while we're focused on supervised machine learning as a paradigm, yes, there's a big world out there. Anything that's decision making driven off data or models of the world, or requires specialization might fit into this view that we've put forward in the book as well.

Thanks for the clarification. So maybe let's start going deeper into the book, right? I think one thing that picked my interest when I read the book in the first few chapters of the book, right? I think you cover some statistics here. The first one is actually many ML projects doesn't make it to production, right? So and the second one, even though they reach production, they actually don't solve the real business problem or they

don't bring any value, right? So maybe let's share a little bit why this is the case. What do you see out there as ML practitioners, right? So why is it so hard to actually bringing ML projects to production? Yeah, thanks, Henry. Like that's the crux of the book in that ML projects tend to fail in this, the common failure modes that we describe, and I can go through them and it's kind of preventable types of failure.

Like if we can learn from experience, from history, and then we would bring in the right tactics to make sure that, as you mentioned, let's say a project that, you know, let's say we invest half a year, nine months into it and ship it. And we found that actually it's not solving the right problem or users didn't really care about it. It's like then what techniques can we bring in there with prototype testing or customer research?

So that's kind of the whole kind of the main thesis of the book is like how have we failed and what have we learnt and how can we do better. So 1 common failure mode in ML projects is what I call POC health. It's like a combination of factors. Maybe business don't have enough of the appetite to release

something to users. So we might take the first easy step to build a prototype to test it, maybe internal demo of it, but the resources or risk to productionized it and put it in the hands of users is kind of too great. So what the lack of commitment there means that teams on the ground just kind of build Pocs of the Pocs of the Pocs. So that's detrimental to, you know, the business in some ways that you're missing out on opportunities, so detrimental to

the morale of the team. You do a few and then, you know, after a while you say, what's the point? So, you know, you move on and then you've got churn and you've got onboarding and you're losing talent, right? Another common, I think a failure mode is that it's easy to get maybe the first thing out of the door. You hustle, you get the data, you get the compute, you train

the model, you deploy the thing. But then it's kind of stuck together with duct tape and sticky gum and it's caught to test or change after Evolve based on, you know, new customer feedback, things like that. So then that comes to kind of continuous delivery CICD. How confident can we be to test and deploy a change? And so this set of problems is solved by ML OPS, by continuous delivery for machine learning. There's probably a couple more. I don't want to hop the mic.

Dave, was there any other failure modes that came to mind, all these challenges? Yeah, those are two big ones. It's, you know, is there any value in doing this at all? And that can be really challenging when their responsibility is divided across different teams who might even be in different parts of the organization. There might be AI guess the release of ChatGPT sparked a lot of curiosity in how do we best use LLMS in products. So, but we might have responsibility diffused across

the organization. We might have an the ML team that's interested in looking at the use of large language models and we might have a product team that has their own road map where they don't have these features anywhere on that road map. And then we might rely on data, especially in the case of Gen. AI, that's unstructured data that's not well governed. It's hard to make available in a responsible way to these systems.

And so while each group can do what's within their control to move a little bit towards a future state, it's not aligned or stitched up in a in a cohesive way that allows us to establish whether there is some value quickly with cheap and and

low risk tests. And then, as David said, once you've established this some value, you know, if you have productionize something that's not supported by a lot of these good practices around ML OPS and and continuous delivery, which is often often the case to, to, you know, understand if there is value there, then you start to understand how much value you're

leaving on the table. And then this can be an opportunity to invest in those practices to be able to maximize the lifetime value of an ML product. But then that requires a certain, again, a certain degree of maturity in and a certain ability to work across the existing organizational boundaries or reshape them.

Yeah. And that's also kind of the flip side of the failure modes is the success, right? So we've also worked with and seen teams, ML teams successfully deliver, you know, really great demo products and what tactics do they use to succeed like we could see. Things like customer testing and user testing. So before they build out a prototype or MVP, which is a really expensive way to test something, even just talking to users, understanding what is the pinpoint, what are their jobs to

be done, where could ML help? And then once they bought in, they say, OK, this is the right ML product that will solve those problems. Then combating that failure mode of what they've mentioned, like multiple teams kind of throwing across each other. If you remember the DevOps comic where you have devs on one side, you have OPS.

So instead of throwing code between scientists and you know, ML OPS engineers, you know, we do that inverse Conway manoeuvre where we have a cross functional team shipping end to end for this initiative. And when we talk to the team members who worked on that project, they enjoyed that end to end ownership, the satisfaction of putting something releasing to customers. You get to touch different parts of the stack, learn about data science, you learn about OPS, you know, so they help them.

You know, you can get fast feedback. You have the same standard. You ship things iteratively. You showcase it every, you know, 2 springs, 3 springs. So that feeling of flow and speed was like really amazing. And then, you know, when something is released, you know, you've got your continuous delivery, production monitoring when things are not going well. Yeah, and you'll have safety for

changes. If I need to make a refactoring upgrade library, If your CI checks are all green, your model monitoring dashboard is saying performance is good, then you know, you merge that PR and you don't feel nervous or stressful about any releases. So yeah, there there. There are also bright spots in in how teams deliver ML products. Right. So I think when I heard you mentioned, you know in the beginning about POC, hell, you know, so many duct tapes.

So when you bring it to production, I could relate to some other ML projects that I knew of before. So I think many, many ML teams probably in this mode, right? They keep building POC because maybe the expectation from the stakeholders or from the business also kind of like with a lot of hype, right? Because these days people think

AI can do a lot of magic, right? And especially with all the success of GGPT and all that, they even think that it is easy to do. But I think the reality may not be the case, right? And the other thing is about the practices, right? So many ML teams first, either they don't have many good software engineers, so they are like data scientists or you know, ML engineers who just have a lot of expertise in building the models.

But actually they don't have all these other skills like refactoring, continuous delivery that you mentioned, right, testing as well. Or the other flip side, which is a lot of software engineers being turned into ML engineers. So they are not necessarily an ML engineer, but they just come from, you know, like web application development and turn into ML engineer. So I think some of these things

definitely I can relate. And just to add to that, there's also the flip side of engineers or ML engineers being excellent engineers, but not treating it as a data science problem Where there is uncertainties is the need for experimentation to get fast feedback. So yeah, I think the kind of emphasizes the need I think for cost functional collaboration from both like data science from I know OPS from SRE, things like that. So maybe let's go there, right?

So the composition of the team, you mentioned a few times about cross functional teams and you know, these silos between, for example, data science or maybe the ML OPS or whoever operates the system in the end, maybe there are also other functions like for example, other teams, because ML typically also realized a lot of dependencies like data or maybe it's some kind of model or things like that. So maybe what are the best composition here in your experience? Maybe you can advise us?

I guess here there's another question that is and it's what we tackle towards the end of the book. We look at sort of multiple levels of team effectiveness and one of the levels we look at is an individual in a team, the types of practices you can use, the expertise you can build as an individual. The next level we look at is within teams. Again, how those practices around technology delivering product augment teams but also broader team dynamics. But then the third level we look

at is between teams. So how can teams be effective between teams? And so the so the best make up of a team depends a bit on the shape of the team and its

interactions with other teams. And so this is where we use the team topologies model to identify the different types of ML teams that you might sit in. And so to run through it quickly and we can come back and dive into details, you know, you, you might have a streamline team and this we'd say this is the basic unit to think of first ML project, a streamline team that can deliver end to end. You're not going to be conducting ground breaking ML research.

You're going to be using tried and true techniques in this case. Or at the other end, where you have an established ML ecosystem internally, you can add another streamline team that draws on those existing services internally quite easily. Then as you scale beyond one team, there might be two routes that you take. And so you might go down the route of where you have a sort of low level common concerns across different initiatives.

That might be the opportunity for a technology platform to support ML at at at a lower level like compute and, and, and data and and feature engineering. And so that would be a platform team in the the team topologies team shape and that would aim to provide as a service and that would be composed differently from a stream aligned team. The other route you might take to scale is you might identify

business clusters of ML needs. So there might be something around audience engagement or there might be something around asset valuation or there might be something around content moderation. And so those are sort of specific set of business needs that also come with an associated set of ML paradigms as well and that don't necessarily have a common support in a technology platform at that right business level of abstraction. So then you might have a what's called a complicated subsystem

team. And again, the makeup of that might be a little bit different. And then as you, you know, as you have a bigger ecosystem, then there'll be a range of problems that come up of a similar nature, but like have a unique presentation each time, which might be around, say privacy or ethical use of data or optimizing particular techniques. And this is where you might have an enabling ML team as well that sort of acts as a consultancy to other ML teams to make themselves redundant.

And so that was a long way of saying it depends on the ideal make up of a team, but you might assume it's a stream aligned team and talk about the roles that sit in that. Maybe I'll throw to David. I think Team Topologies is a really useful set of constructs, the ones that they've went through because it helps teams scale through the team's API.

So as Dave mentioned, if we treat everything, let's say as a stream align team, like in software engineering, we're very familiar with cross functional teams. But then the failure mode there is that teams like Conway's Law, right? We've got three teams. We're going to build three sets of, you know, basically some architecture, rebuild certain tools that we need. Then that's where like maybe platform team comes in to

abstract all of that. So then with that, what we've seen worked really well in one particular case was this complicated subsystem team. Like as you mentioned, ML is hard, a lot of moving parts they took on the effort to build this ML product. Let's imagine it's, I can't say this specific example, but let's say it's a car valuations, right? That's kind of ML product data

product. The API to this team or this product is I will give you the planning of cars and now that is self serviceable by other teams, they could embed it on the mobile. Mobile team could integrate with this API and expose that ML capability. A web team can do the same or e-mail marketing team can do the same and then send personalized emails to say you know about car

evaluations. So then that was how that team scaled rather than having to integrate with each particular thing, encapsulating that complicated subsystem as kind of formal set of APIs either through batch or through real time that help the team like achieve more and get more mileage out of the ML product they built. Yes, yes, that's a great example. The complicated subsystem team is probably going to have a

bunch of specialists. It's going to look maybe most like what people imagine an ML team looks like. But as David said, to scale their impact in the organization, they do need that business domain expertise. They do need to deliver as a service instead of constantly collaborating with other teams. You know, maybe some product thinking helps with that service definition as well. So, you know, that might be one team composition.

Yeah, that's right. And if you replace that example there from correlations, which is a bit like I think not everybody can be with that, let's say it's like travel recommendations or product recommendations, then that one team that spent all the effort in building product recommendations now can impact multiple parts of the business by having the right team topology to have that fracture

playing around. OK, my team is doing product recommendations and yeah, applying the kind of data product disciplines, exposing this as an API, encapsulating the details. Very exciting to hear about team topologies mentioned for ML product ML teams, right. So I think it's kind of like back then, right? It was a revolutionary approach to how we kind of like create

different teams. And I, I'm glad that you brought it up because still I believe in many companies, they think, OK, we want to build ML product and ML project. They just hired a few data scientists or, you know, these ML experts and they just asked them to kind of like build the model first without actually involving the other aspects of software engineers. Could be the streamline team up from the product side, or it could be, you know, the data or

it could be anything, right? But I think that tends to kind of like have its challenges. So I think bringing the concepts such as Steam Topologies to actually think holistically how we're going to the different ML projects is something right, really, really important. Yeah, yeah. So that's a really. Great point and I think one of the quotes opening quotes in our book was from Edwards Emming saying a bad system would be a good person every time.

So you can hire the smartest data scientists, put them in an environment where it's not the right system, then, you know, we get what you described there. So the whole thesis of Outlook is how do we create those systems to help teams build the right thing, you know, solve the right problem, build the thing right, You know, engineering and my engineering data science.

And then in a way that's right for people, I think what they've puts in a really good way, like it's not just shipping and shipping, but in a way that has the right team shape, right collaboration, more right trust, psychological safety and also the right processes to deliver ship early and often. So, yeah, I really was intrigued when you mentioned, you know, you can't just put a group of data scientists together or

engineers together. You really got to create that system where they can, you know, ship to the right and solve the right problems. Yeah, thanks for adding that To

come back to the theme of, you know this product discipline, right. So I think we know that a lot of ML projects doesn't make into production, right or they solve the wrong problem or they don't bring value, right. So I think they're, like they've mentioned in the beginning, there are a lot of uncertainties when you actually build ML model ML product, right? Because I mean, the way it works also, it's kind of like prediction. It's kind of like there's some kind of ambiguities inside LLM,

there's a hallucination, right? So how can you actually come up with an approach such that, you know, when you first building the ML product, you can actually build something that is kind of like bringing business value either to the users or to the organization. So I think this is probably 1 hard aspect as well. So typically how would you run this? It is hard and it's it goes beyond the technical, although the technical informs it.

I, I find it's useful to think about the essential characteristics of of ML solutions. We're exploring them because we think they're going to be superhuman in some aspects. They're probably not going to beat the best experts in a field. That's maybe one myth that we should tackle straight up, but you know, they'll be able to do things at speed and scale that no human could. But then we need to also consider that they'll make mistakes by the very nature.

Again, we wouldn't consider an ML solution if we knew the right answer every time. We'd write some rules instead. So there'll be some percentage of mistakes, however small, in, in any ML solution by design, as well as the unforeseen mistakes.

And then when it comes to those mistakes by design, we really need to understand the the cost sensitivity of what's, you know, what value do we get out of a bunch of right answers and what is the impact of maybe a very small number of wrong answers, but they could have a very huge impact across all sorts of dimensions to financial as well as security bias and fair treatment of all our stakeholders. So yeah, we'd need to consider that fallible nature of them as

well. And so, you know, we need to start with products that are designed to handle that failure. You know, they have some upside from when ML gets it right, but they're robust to the times when ML gets it wrong because it will. So starting from that perspective, you know, we can then identify some experiments about how well does this need to work. We can start with very simple baselines.

Sometimes, you know, even just predicting the majority class or or random guessing and you know, seeing how that works as as a product, but you know, ideally yet to resolve this uncertainty, we're getting into some real data and understanding the predictive potential of the real data. And so this is again where it's like it's a challenging multidisciplinary exercise where we're trying to proceed on

multiple fronts. So we building the right thing can out, yet can our solution support or a proposed solution support the performance that we expect and so on. Yeah. Well, just to add to that, I think that emphasizes the importance of that cross functional, the nature of the work as well.

Like if we frame this as a data science or ML problem, then we can try our level best to go from, let's say 55% accuracy to 99. You will never get to 100 and we will spend many, many weeks and months trying to get there. So it's not just the ML problem where we try to improve the model's accuracy or recall precision, but also how can we design for these failure modes of the ML model?

So like for example, you know, hacks, displaying, designing a product in a way, right, to show users that, OK, this is not a competent prediction or this is a prediction, but would you correct that? And also maybe even giving users options like these are the top three, like the models, top one prediction may be, you know, not where we want it to be, but the top three, OK, is much higher. So designing the product in a way that mitigates these failure modes of the ML model.

And yeah, I it's easy for teams if they don't have the right capabilities or right skill sets to try to solve it. Like if you are a hammer, everything is a nail and it's very costly to try to level up the accuracy. We may never get there. But yeah, having that cross functional approach, like how do we design it in a different way? How do we talk to users? Like the users find this OK, that's a kind of more holistic way to solve the problem.

Yeah, that, that hammer and nail is, is really important as you start to shift to OK, yeah, how do we make this viable or how do we, yeah, make it viable from the perspective of solving the problem effectively, but also from being economically sustainable to to maintain it. And I guess there's another essential characteristic around it, the fact that ML solutions are narrow. The very training process is to optimize a loss function, which is a narrow definition of

success, but they're composable. And you know, this is, I think where Gen. AI can be really interesting. And I'm not the only one to take this perspective, but I've described like Gen. AI is a stone soup for innovation. So the story of Stone Soup if if you haven't heard it, is that a weary traveller arrives at a village late at night and all they have in their knapsack is a stone. So they go to the first house in the village and they ask the villager there, could I have an

onion to make stone soup? You know, I've got the stone, all I need from you is an onion. That villager says, oh great, yeah, I'll provide an onion. They go to the next house and repeat and ask for carrots and, you know, proceed around the village. By the end of that process, they cook up an amazing soup that feeds the whole village and it

was all cooked from a stone. Considering that, you know, you might have a spark of an idea, you might be able to prototype it easily, but it might actually be made-up of many different components composed together to produce something that looks like that behaves intelligently. It needs to be factored into

that process as well. And you know, one of those big components will be the differentiated data that you bring and often like the step from prototyping something in a experimental environment to actually plumbing those data pipelines in a way that's sustainable for production use. You know, that can be a major step as well. That needs to be considered upfront.

And so, you know, when we've talked about, when we've explored AI and ML initiatives, you know, we put a heavy weighting on that factor of where will the data come from and yeah, how will you deliver it to the product or application. You know, that's again, you know, it's one of those laws where it always takes longer than you expect, even when you plan for it to take longer than you expected. Like any software engineering

project out there, right? So I think the most important thing is ML product ML project, right? So you need to still have the product thinking concept in the very beginning, right? So you, you've mentioned a little bit about, you know, is user interviews experiment, you know, building prototypes, making sure the data that you feed into the OR the ML training and all that is also appropriate, right? And I, I like the mentioning

about failure modes, right? Because unlike other products out there, we kind of like know the input and output that we want the features to be, right? So ML product typically could fail in AI don't know unpredictable way, right? So we have so many things mentioned in the news, like for example, Google, you know, image classification, you know, the ChatGPT and you know, Gemini or bot giving wrong hallucinating answers. So like, how do you tackle that?

Plus, I think the whole aspect of, you know, data security bias, right? And also others associated aspects of Fair data that you use in the training, right? I think it's also another thing that you should put your product thinking concept holistically so that you can come up with a very useful and valuable product. So let's go to the other

discipline, which is the engineering side, right? So I think what I could see in the past as well, like a lot of ML code is kind of like highly unstructured Jose, right? So it's like procedural. There's no proper modeling. It's just function calls over function calls, very complex to trace. So maybe in your view, and you mentioned in the beginning as well, you took a project to

refactor ML project, right? So what are the disciplines that typically are lacking in the engineering aspect of ML product and how we could do better? Yeah, I personally can resonate and relate with that experience. I have to get glasses recently, maybe because of oh, but I think mainly because looking at too much code all the time and sometimes in the late nights because of strengths of you know, which is.

Other things which we as a healthy team, you know, if we do those right practices with automated testing, with continuous delivery, automated deployment, then nobody should need to work late night. So I think a couple of things that you mentioned there. Number one, I think test automation is a big part. Like every ML engineer, data scientist we've worked with, they enjoy the automated tests

that we introduced and added. And so it's, I think here at this point is a problem of information asymmetry. Like we've got pockets of teams of people who know how to do automated testing for ML systems. And every team that I've joined and worked with, they say, oh, I didn't know this, you could do that. So I think that's the desire and demand for, you know, more automated testing ML systems. 1 encapsulating story was we had a project, it has kind of close to 0 test coverage.

It was an LLM system. So over time we added the test pyramid like the simple things like unit tests, integration tests to touch our whole LLM application, check that it's OK, those are still point based tests, right?

And we also have that kind of more like deeper like model eval tests to suite off however many examples we run it 5 minutes later, we know OK model accuracy is 75% or whatever number that is. So then we had this automated dependency manager like on upgrade called SNC or renovate or dependent board. So SNC open up APR saying you need to upgrade this it automatically. The PR have all these screen takes tests for passing. We automatically trigger model eval.

We know, OK, performance is just as good. So in 15 minutes we could merge the PR, no stress, no effort. So that's how we grow capacity of a team just by having this automated testing model eval. You touched a little bit on software design or code design as well. And so yeah, that's, I think any code base, not just ML is

susceptible to that. And so I think taking that next level of discipline to say, can I attract function for this, Can I have a readable variable name, not just DF or X, all of those software hedging practices which we can link some in the show notes, single responsibility principle, my favorite one is open close principle like can you design something that is open to extension, but you don't have to modify it every time?

Probably I'm going to too much detail there, but I think the design of it or lack of design sometimes stems from the lack of tests. As you know, if there's no test, then nobody can refactor. Refactoring so scary, so risky, like nobody does that. We pick the path of the least resistance. So yeah, I think the teams that we've worked with, the moment we added that safety harness, when you have that test on the path to production, then a lot of

things can happen. One time we did massive refactoring of like this variable that was LinkedIn all different places. You know, it was, you use ID shortcut, replaced it in like, I don't know, 50 places. And then, you know, test pass commit done, right. It was like either effort reduced from maybe days of testing to again minutes and then in 20 minutes it was running in production. So yeah, it's like a lot of these practices that have been emerging.

I think it's just about spreading it more and it's why we wrote the book so that our teams don't do late nights writing code like I did in one project. And yeah, just enjoy the flow and work life balance and yeah, zero stress and production deployments because, you know, tester passing, you've got production monitoring. So a lot of these engineering practices will, you know, really help team feel the joy and flow of with ML products.

Yeah, I think the call out around testing is really crucial and actually allocating your effort effectively. So in a regular software product we might use the test pyramid to direct effort so that we have you know large number of cheap high level tests. You know, we have a medium number of integration level tests and then, you know, we won't have a small number of end to end tests. We can actually in when we're looking at ML applications and other data intensive applications, we can add a

second dimension to that. So it's agreed instead of a instead of a period. And that dimension is the data dimension. So you know, we might have, you know, a lot of small cheap tests around individual data points. So Canaries, I guess you might, you might also describe those as we might also then have samples tests at a sample level. So these are going to give us a little bit more insight than a point data test. They're going to have some

variability. So there's going to be some tuning there, but they offer us much faster feedback than the final level, which might be a kind of global, let's test on all the data that we have. And often people don't think hard enough about balancing the tests across that spectrum of data. So, you know, you might be able to do a very quick training run on a very small subset of data.

If anything's misconfigured in the training run and the training doesn't work properly, for instance, you know, that will break and you'll get that feedback really quickly rather than waiting hours for it. But you might also, you know, the training might pass, you might have a, a lower benchmark or threshold for acceptable performance on that low run. But at least you've tested end to end and got some feedback with a, with a sample data set that it's, it's likely to work

at the large scale. So it's all about, you know, bringing that feedback back. And I think when when we come to that uncertainty in the front end as well, also being thoughtful about how you test under those conditions of uncertainty, when you don't even know what it is you're looking for in exploratory data analysis. And so there it's kind of moving from the unknown unknowns to known unknowns to known knowns through testing.

You know, visualisation is really key to be able to look at the data and understand what it's telling you or use automated tools to find relationships in the data. And then when you sort of understand what the data's telling you so that visualization doesn't look right, does it look like? I expect that can actually be a form of testing called a visualization driven development

at times as well. But then once you understand qualitatively what you're looking for, then you know, there's a whole range of data science techniques that you can use to turn that into a binary expectation that can pass or fail. That, you know, might be useful in an exploratory environment, but then might also be something that you promote into a production integration pipeline

as well as David was describing. So really, you know, thinking hard about how you use testing to get fast feedback all the way through the life cycle is pretty crucial. And just to add to that, LLMS, as you mentioned is all the rage for the past two years. And so one of the teams we worked with, we had to innovate and think about how to shift that. So as Dave mentioned, full evaluation can be costly, it takes time, it can cost money, especially for LRMS. So what we ended up doing was to

shift again that left. So before we kick start with big eval or any further deployments, we had a integration test and one of the challenges that the team said was like how can you test something that's non deterministic? You know, the answer is different every time. So we had to, you know, we wrote a assertion function that asserts on intent rather than vocabulary. So we can still evaluate that this response given these conditions. Yes, it's the intent of what we

we expect what we had expected. We were using pie ham Crest for that kind of amateur style extensions or you could use, you know, other tools as well. But yeah, I think sometimes at the forefront of this new capability, we have to be a bit creative on how can we shoot that left to get the feedback that they've mentioned. Right, thanks for mentioning some of these techniques. I'm always intrigued like you

mentioned, right? So I'm always intrigued, how can you test something that is non deterministic and you know, so many variables that could come in into play, right? So I think thanks for bringing also the importance of automatic testing. I think I like what you mentioned when you explained that, right? There's a little bit of information asymmetry.

Maybe there are some MLA engineers who are never exposed to some of these techniques and when they know it, actually they could actually follow the discipline and make sure that the products are getting better and better. And I like also the approach of, you know, slicing the data for

different stages of tests. I think that's also key in making sure that the ML projects also kind of like still behaves as what we expect because like, for example, you can tweak the model a little bit, you know, the output can change so much, right? So we don't want that happen in the production. The other aspect of ML that

people always talk about lately is about ML OPS, you know, building platforms, you know how to actually deploy a model and operate it. So maybe a little bit here. What do you think about ML OPS? Is it something that we all need for building ML product and what problem does it solve? Yeah, I think it's yet another tool in our toolkit, which I very much welcome. You know, back in the day we have to wrangle and think how to solve large scale distributed

processing. But now there are these ML OPS tools that let you abstract away that concern. So in one case, one team we have built at ML platform where now the data center is anyone who doesn't have know anything about infrastructure or AWS or Kubernetes, they won't just write plain Python And say, I want to have this, you know, large vertical scaling this way, I want to fan out and all of that starting from Pythons Compute is one part of the ML op

stack. You know, there's experiment tracking, which has really helped us as well. Every pull request runs an experiment that reports some results that we can kind of check over time if David creates a new PR that this is better than our champion model. So that ML OPS practice was really, really welcome as well. The challenge here is like too many tools and it's hard to navigate. Powerworks had this article called A Guide to Evaluating ML Platforms, which we could link

in the show notes. That really helped. Like thinking about the capability, like some platforms try to do everything, some are narrow. So how do you pick what's right? And how do you avoid like shotgun surgery, like vendor coupling, things like that. But yeah, end of the day, I think ML OPS is about abstracting away complexity so that you can focus on solving the right problem, not having to deal with undifferentiated labor

in your day-to-day work. And I think, yeah, coming back to the testing perspective as well, I think one of the things we highlight in the book to get the most out of ML OPS automation and abstraction, you also need to ensure that you're doing the right testing to give you confidence that when you're moving fast, you're doing so safely.

Yep, and to add to that as well, like that's a key point of making the book in that ML OPS you can't ML OPS your problems away just like how you can't dev OPS your problems away. Last week you had on the show on last episode DX with Laura Tyco and talking about Dev X. So I really like this diagram of the DFX triangle. How do you get faster feedback loops? How do you manage cognitive load? How do you get in the flow

state? ML OPS helps in a few ways, but ML is not going to write your tests for you. They're not going to make talk to users and make sure you're writing the same right features or implementing the right features. They're not going to make sure your code is nicely factored and readable so that you can stay in

the flow. So, yeah, I think it's another tool button is to be coupled with these other disciplines that Dave and I mentioned in this podcast and in the bulk, Yeah. Yeah, thanks for the plug for the developer experience as well, right. So don't forget any kind of ML product essentially it's also like a software engineering problem, right? So it's a social technical. So don't forget also the aspect of this feedback loops, ecological safety you also mentioned at the beginning, right.

So all this, like I mentioned at the very beginning, right, it's not just ML technicals that you need to understand, but it's actually at the end, it's a software engineering thing that you have to handle really, really well.

So we have talked a lot about the other things as we move towards the end. Is there anything that we haven't covered that you think should be mentioned as well? I think one key take away is how do we make good easy as engineering leaders, as ML practitioners, We've talked a lot about a lot of different practices.

If we can make good easy, then teams can kind of just by following the team practices, following exemplar repos, then you get that for free in your CICD setup, your test strategy, even maybe hygiene checks of talking to users. Have you put a business case together before you start asking people to work on this for six months? So yeah, it can get out of hand easily with so many moving

parts. So I think as an engineering leader, how do we make good, easy, make teams on the ground when they get the mission to do a certain piece of work? Like it's kind of built into the way of working. Yeah. So I like it make good easy, right. So sometimes we all get excited about the technology, so many moving parts, so many technologies that we can play with, right. But we forgot to expect to make it easy for people to adopt, make it easy to get the buy in as well.

So I think thanks for mentioning that.

So it's been an exciting conversation. I learned a lot about what it takes to actually build an ML projects, which I find it really complicated. But as we reach the end of our conversation, I have one last question that I'd like to ask you. I call this the three technical leadership question. Just treat a bit like an advice that you want to give to us as a listeners. So what will be the three technical leadership wisdom that you can share with us?

Shall I go first? I think again, in line with some of the philosophy of the book, being able to take different perspectives on technical problems is really key for your leadership growth. And so looking for opportunities to play different roles in projects, even for a short time, it gives you that understanding of what other stakeholders require and how to make them successful as you aim to be successful yourself. Yep, I thought of 2. So 1 is I call it focus, function and fire.

So this was an idea I got from Todd Henry in his book I think Taming Tigers. So we are building things every day. Teams can get distracted with many things. So how can we to set our team up

for success? How can we give them that focus, that clear mission, the milestone, the why we're doing it for the customers, the benefit for the business, the business case, Like how will this benefit, you know, the metrics that we care about function, you know, the way of working instead of throwing work or communications over two teams. If have we got the right way of working and fire, you know, meaning the kind of implicit motivation like why are we doing this?

How is this helping people? How is this helping business? So that that for me is one principle I take to my teams. The second one was really interesting. We covered this in the book as well about trust and psychological safety. So when it's absent from the room, then a simple conversation becomes a process of bureaucracy. OK, I got to write up this documentation. We've got to pre read it and let's have a meeting in two weeks to discuss.

And when it's also not there, then team members are afraid to voice concerns of how things might fail. So then we continue to track down the wrong direction. So, yeah, as kind of tech leaders, how do we make sure we have, you know, embody and encourage and ensure we have the psychological safety in the team so that we can all do our best? Work on David's final point. This is what I would say that idea of trust is, is also really important in a multidisciplinary

team and innovative initiatives. Under certain under conditions of uncertainty, we want anyone to be able to speak out with good ideas or concerns that things might be broken. And so to be able to create those serendipitous moments as well as the well understood moments that trust facilitates is a pretty key focus for leaders. Yeah, I like the last aspect that you mentioned about

psychological safety. So you also mentioned that if it's not there, right, things can tend to become like a bureaucratic kind of thing, right? So I think that's really a good plot. So if people want to, you know, learn more about these exciting things that you mentioned in the book, or if they want to discuss with you, maybe is there a place where they can find any of you

or both of you online? Yeah, you can find me on LinkedIn, David Coles. Yep, and I'm David Feidt and I can share a link in the show notes or we can share a link in the show notes. And our book Effective Machine Learning Teams is also the 1st chapter. And the preface, actually, sorry, the preface is available for free on the link that we can share in the show notes. So that gives you an overview of everything we've talked about in this podcast in seven pages. And also the book itself.

We can look in the show note on where you can yeah, read or listen to it. Thank you so much. So it's been a pleasure to have both of you David's in the show. So I hope people learn a lot about the aspects of machine learning model and software engineering good practices anyway at the end. Yeah, Thanks so much, Henry. This is a great chat. Thanks for having us, Henry. That was great.