Accelerating & upskilling your AI learning journey w/ Maher Hanafi #211

We're doing a special in-episode feature on the future of AI-powered incident management with our friends and sponsor X-Map. People as a primary integration layer is really fragile. With multiple people and all of that coordination, you become slower to find the root cause. The slower you find the root cause, you then don't know what action you need to take to resolve it. Getting to that fast is the goal.

Later in the episode, Mike Bennett, who leads the engineering team at X Matters, shares why human-driven coordination creates outage risk and how AI-powered orchestration can dramatically accelerate your path from event to resolution.

We had very clear vision from the leadership team that we need to pursue AI as a key factor of, you know, higher output and value. We went and built things that we will think we will get a good value out of, but also will give us the training, the maturity that we need to be able to build the next set of AI features. Early on when we start working on AI, knowing that it's undeterministic, bringing this into a software that was

Most of the time always deterministic. It's very hard to just automate everything and give, you know, an unpredictable system to own this and make these decisions. So it was very important to build up our trust in the system, cover for all these use cases, meet our compliance, and then learn from these so we can go to the next phase of maturity in the system.

Hello and welcome to the Engineering Leadership Podcast brought to you by ELC, the engineering leadership community. I'm Jerry Lee, founder of ELC. And I'm Patrick Gallagher, and we're your hosts. Our show shares the most critical perspectives and the most important. Habits and examples of great software engineering leaders to help evolve leadership in the tech industry. In this episode, we explore how to explore.

Accelerate your AI learning journey with Maher Hanafi, SVP of Engineering at BetterWorks. This was a tactical conversation that distilled the learning journey that many of us are on right now in a really impactful way. It's not just about AI or AI feature integration. This is about how do you learn faster and apply your learning to real use cases as

as fast and effectively as possible. Meher shares with us his progression from using LLMs as a proof of concept to tackling real world use cases and moving on to more complex challenges. Things like data governance, data flow, AI features for enterprise, and AI agents. It's the whole learning

flow. Mahare also dissects the importance of team experimentation with real use cases. We talk about when to revisit AI decisions with your team, going horizontal versus vertical with your learning journey, overcoming POC to production challenges, Evaluating the ROI of your AI decisions and more. Let me introduce you to Mihare. Mahare is SVP of engineering at BetterWorks.

He leads the AI vision and applications for their AI-powered performance management software, overseeing the integration of AI tools that enhance HR functions like performance reviews, goal setting, and employee development. He's a board advisor for the Strategic Artificial Intelligence Program at University of San Francisco and was former CTO at Thrive Global.

I also want to give a special thank you to Corey Kodo for co-hosting with me. Corey's one of our leaders of ELC Seattle. He was former SVP of engineering at PluralSight. is actively building in the AI space right now and is documenting a lot of his AI learning journey as well. So I was really excited to include his perspective in the conversation. Enjoy our conversation with Maher Hanafi.

Well, welcome you both. It's exciting to have everybody here for the conversation. And for everybody listening in, we got uh a co host today joining us. We've got Cory Koto. Corey's one of the leaders of ELC Seattle. So welcome, Corey. How are you doing? I'm doing great. Hey Patrick. And then we got Maher here. Meher, how are you doing? I'm doing excellent. Thank you for having me on your podcast.

I've been really excited for our conversation because I think we're gonna be able to to examine this from a couple different perspectives and really do the topic justice in terms of accelerating learning with AI and helping people become more effective with championing AI initiatives in their company.

Everybody listening in here, like Mehair, you know this. Like AI is moving fast. And then the other hand is for everybody listening in, they are under constant pressure to keep up. And this is the operating environment that we're that we're in right now. This is really the topic of our conversation is how to accelerate our learning and keep up with the advancements of AI and how that impacts our engineering.

conversations and we've got some incredible stories to get into with you. So a story that I'd love to hear from you first, Meher, is what was a moment in your AI journey where you really realized you had to rethink your entire approach? Where you're like, wow, I need to upskill and learn fast in order to get on top of this new advancement or this new initiative.

Yeah, that's a good question. I mean, I don't come from an extensive machine learning and AI background. I don't have a PhD in data science or AI myself. So my background was more on software engineering, distributed systems, SaaS products, and all of that. But a few years back I was Yeah, found myself in a position as a VP of engineering in an AI startup, leading AI teams and AI projects.

And when I looked at that from, you know, that perspective, I was bringing my, you know, ways of leading software engineering teams and all of that, but AI was completely different. The way you plan, the way you kind of manage.

process and pipeline for AI was different. So for me it was like a moment where I was like, well I need to step up my game here and I need to really be able to first understand these complex concepts and be able to lead the team from a vision, from an execution perspective. So I realized that I need to be able to speak the language. I need to be able to lead the team and hopefully could be a good leader for the group. What I did is I started learning.

And by spending a lot of time on my own reading and learning on my own. But one of the ways I was not thinking about at the time, which I was surprised. to appreciate the value of it was to learn from my team. Like really learn from the people who I manage and they report to me. But I ended up, you know, going to them, asking a lot of questions. They were very generous with their answers and knowledge.

And I learned from them first before I learned from any other source. And I think that learning was also curated for the topics and the context we had at hand. And then over time, I think when uh later when genetic AI became a thing, right? Like I think that was a moment where everything was Booming, genitive AI, LLMs. Uh I couldn't just sit and watch this going, you know, moving that fast. you know, increasingly kind of impacting software engineering development, so I had to go even deeper.

And I think the way I did it was by trying to find the right balance between, you know, how far I go into understanding, you know, some basic concepts and how deep I need to go into some areas. because I need to understand how these will bring value to our software and to what we do. I think at the end of the day I ended up, you know, uh reading a lot.

uh following experts. And then I thought, you know, that experimentation could be a good way to learn too. So I was really focused on getting small prototypes, small examples running. These early days, you know, it was everything was through an API, but then over time became easier Just run things locally and deploy to your own, you know, environment with the open source also development.

So at the end of the day, uh looking back at my journey, I think the most important aspect of my hopefully growth in this area was like how I was able to manage agile learning, really learn. But also, you know, self reflect and take some time and see, okay, is it good enough? Uh, should I need to go deeper into some areas? Should I completely switch and go and learn something else?

The other thing, hands-on experimentation was key for me to grasp these concepts earlier and faster and then help my team uh move forward. And then collaboration again, learning from my team and then learn from others in the community and then bringing this as Key uh value that I bring to my team in terms of like being the AI champion, building a vision, and executing on our roadmap.

You know, at the end of the day, as an engineer leader, your goal is to execute business goals and outcomes, and leveraging AI was key, and understanding and learning about it was just instrumental. So that's my journey again. And I think that moment where I felt like going from an engineered leader who was leading like a very specific set of technologies to

get AI into my org and then I had to learn it and speak the language and with the genetic AI boom. Everything happened a little bit organically, but you know, it turned into a passion at the end of the day that really fueled my hunger to learn more. The way I learned is a little bit different and I put myself in positions where I committed to explain AI to other stakeholders.

That pushed me into learning and I didn't have any other way around it than just going deep into understanding this because I think the best way to teach something was to really deeply understand it.

So you're talking about basic concepts and experimentation. Are there certain milestones you want to mark along this journey? So early experiments that you you tested or what what was like the concept progression that you you moved through?

Yeah, I think at the end of the day, you know, when when Genetive AI at least started, the whole idea was that you have this amazing LLM that is a black box. You don't necessarily know how it works, but you had an API to access it and you would just throwing text at it and you get some very impressive human like language back.

So experimenting with that was the first thing I did through an API Chat GPT at the time. And then as you start going deeper into trying to bring this to our own, you know, systems and frameworks and products. Uh, you get to understand, you know, some of the different nuances of using AI and the complexity of the concepts behind it and how it works, and also the risk.

And all of that. So I think in terms of like iterations or milestones, for me it was very important to just understand how you know these things work from a basic concepts for of tokenization, you know, context, windows, and all of that. And then, you know, generating the next.

token was very key to understand and go deeper into that concept. And then I start experimenting with real use cases of scenarios that might be the next product features we have. Like really how would you rephrase a feedback? or how would you uh use AI to help you set up goals for yourself? Again, we're talking about I currently work at BetterWorks where we do performance enterprise software. So it was very important to apply real world use cases.

to my learning of AI. And then again went into deeper concepts like RAG and fine-tuning and some other concepts where it became a little bit clearer and more tangible use cases of really how much AI can empower these features and how can they create Advanced experiences for the end user just by leveraging AI. And I think that was a key moment where AI grew to a point where, especially open source.

grew to a point where it was easier to just, you know, as an aging leader, I don't spend much time coding and building myself. But it was easy for me to just set up few frameworks and tools. I helped me really experiment with AI locally. Think about OLAMA and other techniques where you can run any models, you know, open source models locally, and experiment with that and build these system prompts and then use rag and use you know semantic search and all of that.

So it was again, there were many iterations and many steps throughout the journey, uh, but each one of them got my level of understanding higher over time, and I was able to help my team develop better systems again to to get to the final outcome, which is building more advanced features.

You know, a big Part of of learning, and Maher touched on this, is like applying the learning, experimenting with generative AI in a domain that you're already working in can really help accelerate the learning that takes place. And I like how you're applying it. You applied it to performance management since that's the domain that you're working in to try new things. I had a similar experience when I was getting excited about generative AI.

A few years ago as that was ramping up is applied to the domain that I'm working in. Software engineering intelligence, developer experience, developer productivity, because you can go and test. the frontiers of the generative AI systems, because if you're already a subject matter expert in those domains, you can see where the limitations are, and you might find novel output.

from the generative AI systems that cause you to go deeper, which can you know create new innovations that you can bring back to your business. Applying the learnings, experimenting with it is a is a key part to to go and benefit them.

We're taking a quick break for a special feature on the future of AI powered incident management with our friends and sponsor XMap. Mike Bennett, who leads the engineering team at X Matters, shares why human-driven coordination creates outage risk and how AI-powered orchestration can dramatically accelerate your path from event to resolution.

We're the ones that are correlating the alerts across the platforms. We're the ones that have to remember that a similar issue happened six months ago and this is what we did about it. We're the ones that have to figure out this is a symptom in service A But it has a dependency in service B that we need to know what that dependency is and how that could impact this thing. We decide on who is going to be page based on some informal knowledge.

It's it's not scalable. I mean that all of that works in a in a very small scale environment. But as as systems grow, as teams grow, people as a primary integration layer is really fragile. So the outage risk is with with multiple people and all of that coordination, uh you become slower to find the root cause. The slower you find the root cause, you then don't know what action you need to take.

N not knowing immediately what the problem is, so you don't know what the route for that mitigation is. goal and is the key problem when you've when you're relying on people to do it. When a signal comes into X Matters, the first thing that you can do is based off of that signal, you can then make a call out to the right people.

for our incident process internally, a signal comes in that says we've got a system problem, the first thing it does it pages out an incident commander and an ops person because those are the two people that are most likely to be on the you know required in any call. From there, the incident commander can then use automations that are set up in the incident because it it automatically creates an incident for us.

It's linked to the ticket that generated the incident. And from there we can determine, okay, well I've seen I've seen this before because my incident suggestions is saying this looks similar to this incident you had last week. We've got built-in automations that can do stuff. So within an instant you might have an automation that says automatically restart pods or automatically rollback services. Like I mentioned before, we can also do that as part of a response.

to the signal that comes out to say, okay, this has happened, do a rollback and I can just Touch my phone and go back to bed without even getting out of bed. All of the automation, the flexibility of the tool and all the the things that you can build in along with the data that you've got with the service catalogue, with your on call, with your who's on duties and get you to get the right people

at the right time on the call if you need to get to a point where you're in a conference. X Matters automates the entire incident lifecycle, taking you from initial event to final resolution. To see how their purpose-built AI slashes your resolution times and gives your team the context to stop disruptions before they start, head to xmatters.com. That's xm-a-t-t-e-r-s. Dot com.

One thing you mentioned, Maher, was about like bringing the team along. in your, you know, you did a lot of work on your own. How did you set aside time with your team to go and experiment, play, learn about generative AI and how they can apply it in their work? Yeah, that's a good question. And the reality is, again, at BetterWorks, at the time where we were kind of exploring generative AI, we didn't have a dedicated AI team. We didn't have AI experts.

So the whole story started where we had very clear vision from the leadership team that we need to pursue AI as a key factor off, you know, higher output and value proposition, I would say, and also stay and be competitive in the space. And we are an HR technology. And this has a lot of different nuances to compliance and Sensitive data and all of that. So we couldn't just go and experiment as other domains and other software can do, because we're really in this highly compliant space.

So the way again we were going through this journey of learning, we had other uh thought leadership and other partners in the company that were trying to push, you know, their understanding as well. And we created this sort of framework that helped us really navigate The people, the process and the technology.

It was mainly a flywheel framework where we really needed to go and understand what we are trying to do and find out these good use cases that are highly impactful, but also maybe on the low hanging food area from an AI perspective where you just need to leverage one of the easiest feature like summarization or rephrasing from AI instead of going deep into like and leveraging the more sophisticated ones.

So we need to find these good use cases first and then understand them and decide that we're gonna pursue these.

And then next was to be able to first build a proof concept and then deploy it to production. Hopefully we'll have some time to go deeper into that because that was one of the biggest challenges you face as an engineering team or as a business, because proof concepts are easy, are very exciting, but to get that to a level of production standards and meeting all the compliance and challenges you have was critical and went through many other steps.

And then once you have this kind of build running, you put it, you know, available to your end users, and then you go again into your flywheel and go and optimize. Because your first approach and with the pace AI is going from from an innovation perspective, that first approach you start doing is maybe irrelevant.

Or not optimized as it should be. And also maybe for, you know, meeting your AI goals and deployments and implementation versus, you know, your other systems and distributed systems implementation, maybe the way you did it was. Or just the easiest way to get out there. And now you need to go and optimize and and make this system, you know, building AI on top of building distributed systems work hand in hand and more in tandem.

So I think these are the things that we're going like this flywheel helped us really go into building smaller features, focus on value, impactful features, but then go and over time get to more and more sophisticated AI techniques. versus you know going ahead and pursuing just AI agents from the first initial approach.

So that's how we we have been able to keep up with AI pace of innovation and building more features over time, but all the features we'll be able to go back and try to optimize and get them better over time.

What you're talking about with working on use cases and building up that skill and capability reminds me of the story or the parable that you hear about. You know, the the people in photography class who got the best grades were the ones who took the most photos, not the ones who pursued the best.

singular photo. And so it sound it almost sounds like intentionally building your experience and and building different use cases in that process amplifies your learning and then allows you to then get to more sophisticated and better products or features or integration. Is that is that sort of a fair a fair summary? Yeah, absolutely. Yeah. We went with that exact mindset. We went and built things that we will

think we will get a good value out of, but also that we thought will give us the training, the maturity that we need to be able to build the next set of, you know, AI features. And again, I I I talked earlier about process. people in technology. We also leverage in different AI technology frameworks. This one is the AI maturity framework. The idea here is to make sure that you have different levels of complexity and support.

Sophistication and you go and leverage them one after the other. You start from the bottom going up. And as you build more features, you get to learn and you build up more. Confidence in the entrust into the system because it we do we should not forget that early on when we started working on AI, you know, the topic of the day was hallucination, was bias and all these kind of issues and challenges working with AI, knowing that it's undeterministic. Bringing this into a software that was

most of the time always deterministic. It's a human-centric product when we talk about people performance and performance management. It's very hard to just automate everything and give, you know, an unpredictable system to own this and make these decisions. So it was very important to build up our trust in the system, you know, cover for all these use cases, meet our compliance, and then learn from these so we can go to the next.

Phase of maturity in the system. And that was the again, that was the vertical axis, the horizontal axis of this AI maturity framework. Is how much responsible AI you're building. And by exposing more data to these AI systems, by sharing more, the risk gets higher.

So as you do that, you need to also build more understanding of what are the risks you're introducing, how are you mitigating them, how much safeguards and guard rates you're adding to control that and control AI from becoming a determined to the you know the features and the set of features you have in your system and become a problem. So again, these these are different frameworks and techniques that we use that helped us build more trust, confidence, and maturity in our development of AI.

I like also Mahair how You as a teen revisited what you built. after you launch that initial value, because the foundation on which we're building AI capabilities in our businesses is evolving so rapidly. Month over month, things are changing, or even every three months, it's drastically different.

So the fact that you as a team are intentional about revisiting those decisions, implementing the new ways of working with those foundations, really, I see how it can really accelerate the warning that the team has. So that over time they just get stronger and stronger. Yeah, absolutely. One good example of this is when we started the journey, you know, early on and this time, we we wanted to leverage off the shelf AI solutions like API driven, you know, AI.

So you just have to connect with one of these vendors, you share with them your data, you get the output back, and you leverage this. But again, looking backward into this, trying to optimize it even more, understanding our specific domain constraints and compliance. It was very hard for us to kind of sell this to our customers, enterprise customers, where data sensitivity is high, data leakage risk is is big, and you need to make sure like things stay in your system as much as possible.

So when we went back to these same AI systems we deployed, we learned more about, you know, self-hosting LLMs and deploying these to our cloud solution, cloud systems. Leverage open source models or any other models, honestly, that would can serve this, but keep the boundaries of your system or keep the data in the boundaries of your system.

So that was one of the key optimization we did by just taking everything back into our system to make sure our customers, you know, understand the the where where the data is going. And again, we we we will be talking later about this, hopefully, you know, all the standard classic

software development is still there. You know, we have the cloud distributed systems, we have microservices, we have APIs, we have data warehouses and data pipelines. We needed to make sure all the controls and systems in place still work with AI. So to do this we had to iterate, we had to go back and revisit these and say, okay, how can we optimize this?

is there a lazy loading approach we can do to save time? Because again, AI is data heavy and uses a lot of data. So if you just do wanna do everything in runtime, it's gonna take forever for these AI responses to come back. So we start optimizing for these to be a little bit like happening on background threads and, you know, develop as much as possible techniques to optimize for these end user. So these are incredible examples.

that that you're talking about in terms and also like unexpected ways to accelerate learning, like the one you're talking about here, but the revisiting of decisions and how that opened up opportunities to drive a better build that meets the needs of more enterprise customers. Like I that's such an incredible example.

I want to get into like how did you determine how deep to go into each area of learning? You laid out a couple of important milestones here, and now we're talking about revisiting decisions and applying learning there. Like walk us through your framework for how you were approaching how deep you should go into certain concepts or new frameworks and things.

Yeah, I think this is a this is a very this is a traditional question, especially for engineering leaders. Like how much you go horizontal versus vertical, how much you stack your skills in learning versus how much you spread them out. And I think for AI in particular, because of the pace of the innovation when it comes to AI, like all these things are happening really fast.

So for me at least or for engineering leaders in general, I think we need to keep an eye on what is going on out there. So we need to be aware of the innovation. You should not go and read and go deep into every aspect, everything you see on a weekly basis, like there's a new thing, you go and read everything about it, you become an expert in it. That's not gonna happen. That's not even possible at this page.

So what you need to do is really again go back to the basics and understand, you know, what AI is capable of and its application to your domain. When you get to that level of understanding, you can start seeing some mappings, some affinities.

that become more and more ways for you or triggers for you to go in deeper into these areas. And again, this is one of the soft skills you need to develop as an engineering leader is to know when to go horizontal, when to go vertical. And the key is to have that right balance. And to always do some self reflection from time to time, leveraging, you know, where are you now? What's the big picture out there?

How many things you don't know? You know, knowing what you don't know is important at this stage and how many of these things are out there and what out of the things you don't know, what things you might be game changer for you and your team and your business and you spend some more time on it.

For me it was very important again to go and understand the basics, but then at some point we needed to be able to serve enterprise customers, multi-tenant, leverage, you know, data governance and and really understand how the data will flow. So sometimes I we had to go deeper into these areas and I had to go and understand this a little bit more. And sometimes by doing that,

I was able to help my team also go and make the right decisions. Again, this our job as engine leaders is to guide the team. Even if you might have some people who are more deeply knowledgeable about some aspects. for you to have that kind of right triangle between horizontal and vertical is very key to just guide

And trust your judgment and get the team to think about what is next for you and what is the next thing you need to learn or experiment with. And going back to what I said earlier, sometimes there is a good opportunity, especially with AI now, with everything that is out there. To be able to experiment hands on before even you get into saying this is the way to go or this is what I think is the next thing for us.

So experimentation is also important. Um the summary of everything I said honestly is developing a skill to be able to have this sort of agile learning. There's not just, you know, we usually use continuous learning. I don't think for AI applies in the same way of just keep learning all the time. I think it's more about the agility and when to learn, when to take a break, when to go deep, when to go horizontal. I think that has been over time my my journey learning that.

It's different. The way I need to learn about AI is different, and I need to really trust my judgment on when to go deep or when to go horizontal.

Maher, you teased this earlier, the challenge of going from proof of concept to production. So bring us into a story or example of your journey there, and let's dive into some of the lessons learned. Yeah, definitely again. I think every I hope every AI project today has been, you know, driven by everything we see in the world of AI today, which I call mainly like exciting, impressive proof concepts.

And definitely Gen AI that had this that wow moment where you see, you know, with Chat GPT first and some other follow-up, you know, features with Doll E. Like it was like very impressive.

And if you have to adopt that and build the same and do it yourself in your own business, in your own organization, the leaders and teams in general found a very challenging journey to get So for us it was exactly the same and definitely we we needed to be a little bit more creative and smart and proactive about how we do it. So that's where I want I want to go back to that framework where the focus was on the smallest

Easy, low-hanging fruit from a technology perspective. Like what is the thing we can do leveraging the lowest amount of techniques from Genetive AI? Like just plugging in an API for an LLM off the shelf. Again, it's just an API. We're very familiar with this. We have capacity. We have the software engineers to do this. There was no infrastructure involved. It's just you know kind of pure API concept. And that's where we start building some of these proof concepts.

But then over time we went into again understanding our domain and our, you know, sensitivity to, you know, data and PIIs and you know all of that. We need to bring this more into our system. So that's where we were involving a little bit more the infrastructure aspect, ML ops, all these things were new to us. So we really wanted to go and

Spend some time, and this is going back to the previous question. This is where we needed to go deeper into this and understanding what are our options? Are we going to self-deploy all of all of these into native like resources like uh virtual machines or will we be leveraging some of the managed services by the cloud providers that will just again give you AI as a service. But deployed to your own environment. So we went into that space, we understand a little bit more the cost, the risk.

The infrastructure costs. In the ROI return. I think at the end of the day, when we start building one and two and then the third feature, all these features were, I would say, on their respective verticals. Like every feature were related to a specific domain, think about summarizing feedback. What's for feedback?

Helping you certain goals using AI was for goals and some other features were just independent. And as we move into production, it become more interesting to explore cross-feature AI techniques. So to be able to achieve that now we get into the complexity of The distributed system, the microservices, the data, you know, governance.

Who owns the data that I need? How can I get that data? What kind of role-based access controls I have on these data and how can I keep them applied to the AI interactions? versus, you know, losing control of who needs to see what and breaking all of these guardrails and kind of go into data leak. So I think the journey was again full of learning m milestones and kind of pivots at some point. But the most important part for us was that we knew exactly where we are, what kind of

level of skills we had and resources we had and time we had and then we started small and we start focused first of all on the how much of these will be impactful to the end user. We really wanted to decrease the game changer features, but the ones that were easy to do. And again, going back to these use cases, helping people rephrase feedback as an example was just very impactful. Helping people summarize feedback instead of going and reading hundreds of feedback.

If you get the summary of that, was an impressive time-saving feature, helping people set goals using AI, seeing templates and drafts based on the history of goals they created for the past years, looking at their manager goals, looking at their company top goals. You know, this used to take weeks.

for some people and some organization to come with these goals. And now with AI, it's out there. You have eighty percent of this done as our focus is on keeping the human in the driver's seat. You just need to do the last twenty percent and that's a huge time.

Yeah, Maher, I'm I'm curious if you could go in a little deeper about How you and your team thought about evaluating the ROI for applying AI in the product? Because just because we can apply AI or build some feature doesn't mean we should. What mental models did you uh use? to teach the team, you know, how how to go and measure the ROI. I think it's still tricky to really measure this. I I I honestly call it measuring the unmeasurable.

As of today, I think it's very hard for us and for many other businesses to just, you know, map this to a dollar amount or like to really exact value. But what was important for us is to really understand our end users and build AI features around their needs, not around our own perception of AI and our own needs to label our product.

Powered by AI or leveraging generative AI and not just powered by our need to be competitive in this space and to be one of the early adopter of generative AI in an HR tech space. The focus was really to have that kind of impactful user experiences. And we think that's the ROI again, it could be measured in in the NPS score, E NPS score.

It could be like how much our customers are satisfied with with our AI enhanced features compared to our legacy features the way they were before the AI edition. I mean when I think about this now, I think the focus won't just on impact. Impact. And I think impact is what we would love to be able to measure at some point where we come back and say, okay, well, this feature, this AI addition was impactful. We over time going into that kind of flywheel and optimization in kind of

trying to understand how it's doing, we were adding more and more ways for our end users and customers to tell us exactly how they feel about our AI features. You know, the easiest thing was a thumbs up, thumbs down approach to any generated AI content.

If we want to rephrase something, you can tell us as an end user if you like it or not. That will give us again something to be able to measure, some data to be able to look at, if we need to go back and improve our AI, to fine-tune, to do anything that will make that, you know, your experience next time better. So again, AROI is still like an open area of discussion and improvements, but definitely for us the most value or the most metric we're trying to get to is use.

Yeah, I I think that's the right way to look at it. Because it is very hard to quantify ROI, but thinking about the customer impact and the customer value, I think is the right perspective. At the same time, you know, we're running a business as engineering leaders. And if we're self-hosting models or we're

setting up GPU clusters that's real capital that we're allocating that's gonna impact our profit margins. How do you think about like that with regards to like your overall like operating expense? How much of the our workloads should we be, you know, throwing towards these generative AI use cases?

Well I think the most important aspect of this when you look at it, you know, part of phase two of our flywheel was to be able to build. And to be able to build we needed to go through an evaluation process. What are we going to build using why?

You know, looking into the different models that were out there, you know, the licensing cost, if we want to use this uh off-the-shelf solution or self-host or open source, you know, look into these properties of these models, the capability, their, you know, the accuracy, their performance on the task. And then looking at the cost from an infrastructural perspective, again, you might be using an off-the-shelf, so that's like kind of cost per operation or per token or a cost of unit.

If you self host all of this, you will pay for resources, compute and GPU and all of that. So we looked into this data and honestly again, we wanted to really start small and keep scaling up our infrastructure as if needed. We did some estimates and we found out that the cost is not exorbitant, it's not too high for us to stop at this time and say, like, okay, this is. Too much, too expensive. We cannot pursue this, and we have guaranteed that revenue will be higher to cover for these costs.

I think our based on our estimates and the way we wanted to go really small at the beginning and iterate and increase over time was a good guidance for us to really adopt AI in that manner. And looking at the business in general, AI has been definitely a competitive leverage for us. Like when we talk to some customers and they understand that we have AI feature. Obviously, over the last few years, I think everyone was just going deeper into understanding AI, the risk.

And how it's using AI and how responsible it is and all of that. I think we have been able to turn a lot of customers into buying customers and move forward with our approach using AI. And I think AI was a key into that decision. Again, very hard to measure, you know, one-to-one, I would say airwise aspects, but definitely understanding the cost from an engineering perspective. It's part of any engineering leader job to keep an eye on cost.

So based on these estimates the cost was not too high and the cost can keep increasing over time as we get more customers and the system. So I think the cost was part of the equation, but the estimates, the early estimates, you know, were were not in any way a showstopper for us to pursue this.

As we go today, we have more customers, we have more revenue coming. I think our cost is growing, but it's significantly lower than initially. And part of this kind of estimates was also if we used a system like an off-the-shelf I think the cost will keep increasing as we have more kind of transactions happening with AI. But when we went and self deployed this into our system, you know, we will have these resources compute running and the increase in costs will happen at bigger levels of scale.

And it's not incremental. It's not like when you as you have more customers, you have higher costs. Sometimes even when more customers come, you know, your system can't handle that much traffic and that much operation. And I think that kind of study and understanding the cost will help us, you know, going and pursuing the open source self-hosted approach from a cost perspective and licensing perspective.

So you're talking about like ROI might be unclear, and you introduce this to a lot of other ways to sort of introduce that and help people get a good signal. But from the other side of like confidence in the outset, How do you help develop confidence in your AI strategy and vision? And then the other side of that is like the people challenges that arise in in any of these things. So how did you manage stakeholders or some of the people challenges that came?

Yeah, I mean becoming an AI champion again I'm not sure a hundred percent if that's like a word that would say, but definitely it's more organic than planned for. It's something that happens based on and there's something I always use in engineering leadership.

passion growth development, I think. If this thing is becoming more and more something you're interested in, if and you pursue as a passion, you're you're very kind of you have a lot of energy to learn about, to bring it to your day job, to really have an impact on your business. You over time built up that

Organic image of someone who is really pushing the boundaries of what can be done using AI in your business. You're guiding your team, you're helping everyone, different stakeholders, different departments. But I think the most important aspect of my job at this level was to navigate the risk.

risks of AI more than the advantages of AI. I think again, because of that image and brand AI has today, with all the exciting, impressive you know things we see online all every day, you know, there is a lot of interest from different stakeholders and the people aspect. Here, like they want to leverage AI, they want to add AI, and they think AI is capable of doing everything, which is you know might be true to a certain degree, but as an engine leader, trying to understand the risk.

Where that kind of limit of whatever is impressive

should stop to keep everything under control and running within the compliance you are in, I think was very important. So learning about the risks of AI, you know, hallucinating, the bias and the then the training data, you know, how would you leverage and how would you build features that would not expose this despised and that will not just become hurting your business or your product over time because you were just excited about building it first at the first, you know, time.

Um, so managing all of that, building the right uh understanding of AI, explaining that again, teaching this and explaining this to other stakeholders was key for us to really build again, prioritize the features we had in mind. And sometimes you know that something is possible, you know it's out there, but your confidence

in yourself, your team, your business getting there and building it the right way is still not too high. So you that's why you need to deprioritize it a little bit as you build up more learning and more confidence in the system as you go. I think when it comes to people dynamics, obviously we had a lot of people excited about AI. We wanted all to pursue AI again internally as a tool to improve performance and productivity.

Especially software engineers today, I think they're using a lot of AI tools, co pilots and all of that. So that's all great. But again, there is this picture of like AI becoming kind of replacing people and when it comes to the product itself, AI taking over some of the decisions. And it was very important for us because it's again

It's a very human centric product. It's all about managing people's performance, feedback, conversations and goals. And we couldn't delegate all of this to AI and move on. So we really needed to build the features where We always keep the human in the in the center of the loop. And we help use AI to really automate and improve and save time and kind of give a better user experience. But at the end of the day, as the end user, you still have control of all of that.

So navigating all of these different, you know, perspectives of AI, excitement and hype around AI, but you know, grounding this in real truth, putting together clear guide rails and safeguards to really leverage AI and boost your product without introducing risk. was where we was most challenging part of my job. But over time, as you get into these conversations, as you said, you become some sort of like a reference.

an AI champion, someone who people can reach out to to really make sure their understanding is correct or their vision of a product feature is correct and accurate and can be done within the boundaries of the system. Do you have any strategies for non-AI experts?

for them to build awareness, gain confidence and drive AI integrations and in whatever parts of the business that they operate. I mean, to be very honest, you know, AI experts are hard to find in most of the organizations we're talking about today. I think Genetive AI in particular took the word by storm very short time. Everyone became like an AI expert. So I really label everyone a non-AI expert, including myself.

I start in this journey like a few years ago, but I don't have the strong hands-on IC in research AI expertise. But I would from what I take from your question is like the non AI experts, I would say the non technical people, other stakeholders who are out there either asked to include AI or integrate AI or build AI or ask to just

Assess AI addition to certain products and features or processes, you know, just adopting AI to enhance some of the productivity. From that perspective, I think it's very important that. As someone who has been eager into learning about AI and understanding this, I think you need to be able to build some language. That is a little bit more non technical. This happens a lot when you have conversations with customers. And I'm again as an engine leader, I get into the sales.

pitches and conversations with customer clarifying some concepts about AI. So practicing that a little bit more, going into pursuing again learning about these concepts and how can you communicate around them to S someone who's not an expert or not purely technical, I think was key for me to help myself and the business in general to get more.

trust and confidence that our solution is gonna be better over time as we adopt more AI and as we build more AI and as the end user experience is not being impacted by all the things that might go wrong with AI. So I think over time again it takes time to get to that level of

capacity of speaking uh high level language that is not purely technical. We're not gonna focus on what AI is is just purely made out of it's not about the how necessarily and it's more about the why we are using AI and how we are introducing it and what kind of impact it will have at the end of the day. It's a topic that has been ongoing for me as I was focused on the technical aspect of this, but then over time I need to take a step back and be able

to speak to non-technical people and explain to them and help them build their trust and confidence in AI, including internal stakeholders, but also external customers. Uh what about your AI roadmap at BetterWorks has you really excited for twenty twenty five? Yeah, I think what I'm very excited about is this kind of across domain AI. When we started and part of our maturity framework that we're using, we we will keep adding more sophistication to the system.

And building AI features for these verticals is fun and exciting and definitely impressive at some point. But I think that when the time comes when we have really deeply cross domain features using AI, I think that will be a game changer. And that's something I'm very excited about. In reality, what will happen here is that

You can use AI to just leverage all of these domains, all these data points, and create a whole program for performance management. I can give you a very specific example. Like think about a manager who has a team of like, Ten people. And, you know, based on all the skills they have, all the feedback they receive. All the goals they have been trying to do, the ones that they were successful, the ones they failed, your own goals as a manager of that team, your top company goals.

Think about one day AI capable of putting a program for the whole group, including yourself. Like what things you need to go and learn. And again, we have integrations with uh learning management systems like Udemy and Coursera. you know what what courses you need to take from there, you and your team members, what skills you have, uh, you know, you have to improve, what kind of conversations you need to have, how many meetings you need to be able to set up.

Think about AI helping you put together all this program for you to review, to tweak a little bit and then to execute on for the next quarter or the next half of the year. Again, that's what I'm very excited about. That's where our roadmap is aiming to, and that's where I think AI will be completely game changer across different features, not just one, and empower these managers, empower these employees to just get to the next level of performance.

with ease and with the least amount of preparation and time spent into putting together these programs. I'm very excited about that and we're definitely on a journey to get there and so far we are on track.

That sounds very exciting for I can see a lot of managers being excited for that. We've got some rapid fire questions if you're ready to jump in. Yeah, let's go. All right. What are you reading or listening to right now? I'm on a journey to uh taking courses on the deep learning dotai platform from NDUNNEG. It's been a good journey. The free material out there was just great and I'm going into a pay tier to go deeper into these. So not necessarily a book I'm reading, I think with the

f the pace AI innovation is going, you know, books might get irrelevant very quickly. There are some key books out there that are also on my list to read. And I think the next thing I wanna be able to do to deepen up my knowledge and understanding is to build an LLM from the ground up and there is a good book for that, um presume. I am not surprised that you focused on the concrete applications for for learning here. So I think that's a that's a great answer to the question. Awesome.

Second question, what is a tool or methodology that's had a big impact on you? Daniel Pink uh drive book was was a game changer for me. That's the light and the darkness where I can, you know, every time I wanna focus on how can I get things better done for me and my team, I follow that framework. And the framework is all about building autonomous teams.

and building teams that can master the art of building software and it's leveraging a few things. And there is a twist to that framework. I mean, I kept reading about even more people taking that framework and twisting it a little bit more. And the one I really appreciate is, you know, focus on three pillars.

a common shared understanding. As an engineering leader, you really need to have the same understanding of what needs to be done, you and the rest of your team. Competence, you need to be able to help yourself and your team develop skills. And then trust. I think trust is very important to create this good environment of flexibility, autonomy, and then the center of all of this is autonomous master teams.

Uh that framework has been, again, a game changer for me in my engineering leadership journey. And hopefully everyone I work with and everyone who goes into my teams kind of hopefully reflect on this and and really appreciate the framework uh success. Wonderful. What's a trend you're seeing or following that's interesting or hasn't hit the mainstream yet? Yeah, I think uh I think AI agents becoming easier to use.

Uh and also easier to control. I think with AI agents, there is a little bit off risk of because of its autonomous decision-making capability and the hype around it. It could be I don't want people to go and use AI agents as the first proof of concept and built around it first, you know, versus exploring LLMs. So I think AI agencies today are very good, very interesting and exciting. But this brings me back to the journeys with LLM where they were also exciting.

So understanding a little bit more AI gen agents and I think they will become more mature over time for it to become the next table stakes for integrations with AI and its capability of making decisions could be a little bit more controlled for environments and domains where you cannot go and be, you know, make any decisions you want. You have a very restrictive system, you have boundaries, you have safeguards and guardrails.

For this AI agent to work with. So I'm very excited about that. And I think that will empower what I was talking about earlier: cross domain, cross data point. features that will just go through different things and bring a bigger program, bigger approaches to what you can do leveraging AI. Last question, Mahair, is there a quote or mantra you live by or a quote that's resonating with you right now? Yes, there's there's one I love. This came from early on in internship I had, I don't know.

Like many, many years ago. One of my manager at the time, I think asked me to do one thing. I said, Yeah, sure, I will try to do it. And he said n he interrupted me. He said, No, you're not gonna try to do it. You will do it. So that reminded me of like the way Yoda from Star Wars speaks, is like do or do not, there is no try. And that kind of quote is still

Something I use today. I use it even with my kids when I ask for something, she says, you know, my daughter says, I will try. I was like, No, you will do it, especially when, you know, trying to motivate her. And the the the key to this and why this was important for me because I really ended up learning that first that manager trusted me more than I trusted myself.

I was like, wow, he he knew like I had the skills to do this. So why I'm saying like I will try? Definitely I will do it. And then saying trying implies you know there is a risk to face. Versus, you know, saying I will do it. There's a commitment, there's an engagement and kind of a motivation to do it. So that thing is like again, even if it happens many, many years ago, it's still like something I live by to today. A modern take on the enduring wisdom of Jedi Master Yoda. Exactly.

Corey, just want to say thank you for joining us as a co-host, jumping in and helping guide our conversation. Uh it's been a pleasure to jump in this with you. Maher, just also wanted to say, you know, a couple final reactions. One of the big takeaways I have is that you were talking about what's going on right now and that it's a level playing field and I takeaway that if you can learn fast, then there's a lot of opportunity. And you shared so many great practices with us about

how to learn fast and experiment and and break a lot of this down. So thank you for for spending your time with us and supporting our community. It was definitely my pleasure to be uh your your guest today. Thank you, Corey, for enriching the conversation. If you're listening to this and you're wondering, how can I connect with other engineering leaders in my city? Pull up your phone right now and go to elc.community, click our

chapters page. You can see that on the menu on the left. Find your local chapter and click join. We're hosting virtual and in-person events all the time. And this is the best way to help you get involved, expand your network in your city, and support your leadership and career growth.

So pull up your phone, head to elc.community, join your local chapter, and get involved. A huge thank you to all of our local leaders who make community happen and thank you for listening to the Engineering Leadership Podcast.