#226 - Ex-Google Duplex Eng Lead on Disrupting $2B Clinical Trials with AI - Patrick Leung

Clinical trial can take upwards of 10 years. It costs on average 1 to $2 billion per drug to get to market. These are huge numbers and huge timeframes. Writing clinical grade documentation where the stakes are really high, we're talking about human lives and human safety and things like that is incredibly challenging. And it just staggers me that how many people still do this in Microsoft Word, use Microsoft Word to essentially model much at the author of model these

trials. Patrick Lung is the CTO of Pharaoh Health and a former engineering lead at Google Duplex. So we published a paper with Merck that sort of. To analyze like what is the benefit of using the FIRE study designer to optimize clinical trial design? And we found that the savings over the course of the lifetime of the trial.

Can be north of $100 million. We have customers using this document generation system and they're super impressed with documentation that normally takes potentially months and it can be generated in a matter of, in some cases, minutes. We're also in charge on building the Google Duplex, right? Did you already use something like LLM or is it using a different technology altogether? It didn't use LLMS at all.

It predated that. People started freaking out when we launched this product because it sounded so lifelike. And there were even some people, prominent names of the press, who thought that we faked the whole thing. And yet it wasn't using any of this modern LLM architecture at all. It was using other methods that were much, much simpler. What do you think people should do in this kind of era where things moving so fast?

AI won't replace humans per SE, but people who are able to effectively wield AI will definitely replace people who aren't. Hello guys. Welcome back to another new episode of the Technology General Podcast. Today I have with me the CEO of Pharaoh Health, Patrick Leong. So we have been hearing a lot about AI development recently. Viral health itself is trying to apply AI on, you know, clinical trials and life sciences and

health and all that. I think it's going to be really interesting to understand how AI can be applied in this type of work as well. So Patrick, welcome to the show. Thanks, Henry. Great to be here.

Right, Patrick, I always love to invite my guests to share a little bit more about yourself, maybe by sharing any career turning points that you think we all can learn from you. Yeah. I mean, I guess the beginning of my career was very entrepreneurial. Like I joined a company that was in the very early stages and it went all the way to an IPO and then back down to, you know, near 0 again duringthe.com boom. And so that was a huge learning

experience. And then sometime after that I made the big decision to join a big company. I joined Google back in 2007. And so that was a huge career turning point because I never thought I'd find myself. At a big company. But against all odds, I did and I ended up there for over 10 years.

So I think it's really interesting as well. Back then when you were at Google, right, you work on some of the AI technologies like Duplex, right? So I think since then you have been working a lot on AI related stuff even before this Gen. AI era, right? So what do you see in the last, I don't know, maybe 10 years or maybe 8 years, right. So what are the some advancements that you can see working on AI things right in the past compared to now? So what do you see are the

differences? Yeah, I mean, it's easy to forget now because this whole large language model revolution has just completely taken over everything. And so when most people on this planet think about AI, they they really kind of thinking about large language models as opposed to AI in general, like data science in general. And So what I saw when I was back at Google was there was already an AI revolution that was well in full swing, which was mostly related to image recognition.

Like because processing power kind of passed a certain threshold, we became much more able to interpret images. I'm using AI. And so this resulted in all sorts of amazing advances that started showing up and things like Google Photos and other products like that. But now all that's forgotten because it got completely sort of superseded or overshadowed by this more recent large language model revolution. But there have been multiple waves here. And that's been really, really

interesting to see how. As processing power hasn't increased, there are these new forms of AI that come out and start really solving certain problems really, really well.

Yeah, it's interesting that we know LLM is like LLM or generative AI, right, The bigger categories like the all the things that people talk about regarding AI. So in the past I remember people are talking about machine learning, recommendation system, image recognition, just like what you said. It seems like all of those kind

of like toned down a little bit. Do you still see advancement in those area or is generative AI is like the go to AI strategy that we are going as a human species I guess. Yeah, I think it's pretty extraordinary. How? General purpose and powerful. These models have become and of course, you know, with a lot of the image generation, there are different methods involved

there. Like it's not necessarily LLMS, it's more like diffusion models, but just these multi model large language models are pretty incredible, like going beyond human language into things like genetic coding. And of course, you know, computer coding, like actually programming, has become a huge. Use case for. These language models, and they originated out of the transform architecture that Google and others sort of introduced.

A while ago, which originally was applied to the use case of translation from one human language to another. So from that initial use case, it's become much more general purpose. So it's pretty incredible.

But I do think, I continue to think that there's always going to be many, many use cases out there for which other forms of machine learning that have been sort of largely maybe overlooked during this whole LLM revolution are going to continue to be really useful for solving certain other types of problems involving particularly involving quantitative information like predicting numbers and stuff like that.

Right. So back then you were also in charge on building the Google Duplex, right. So when I remember back then when it was introduced in Google IO or something like that, that was really, you know, like incredible to me. Did you already use something like LLM or is it using a different technology altogether? Yeah, I mean, it's really interesting because it didn't, it didn't use LLMS at all. It predated that. And so in some ways, that was sort of like AI wouldn't say a

prototype. It was almost like sort of a preview of what was to come because people started freaking out when we launched this product because it sounded so lifelike. And some people like the press had this huge media frenzy around this, right? Because Google introduces some kind of AI that sounds like a human being. And it's like. Oh, it's unethical or oh, the game over, you know, the bots are going to take over the world.

And there were even some people, some, you know, prominent members of the press who thought that we faked the whole thing. So it was super interesting to be involved in sort of like a media frenzy situation like that. It was on the Good Morning Show, it was on, you know, Colbert. It was like really hit this very, very big nerve in popular culture. And yet it. Wasn't using any of this modern LLM architecture at all. It was using other methods that were much, much simpler.

And so since then it's just gotten way crazier in terms of what these systems are capable of. But it was really a privilege and just so much fun to be involved in a system that was very much the forefront of the time in doing this kind of thing. Yeah, I think Google itself has to rebrand it, you know, those kind of things like becoming like Google Assistant now Google Assistants, becoming like Gemini, right.

I think all those speakers that they introduced probably is like using some kind of things that you use in Google Duplex as well because the human tone are so natural when I chat with like Gemini using voice, right. So I think it's really cool

technologies, you know, that comes out of those innovations. And today specifically we are going to talk how to apply AI in health or life science and clinical development. So your company, Vero Health, if you can share a little bit more, how does Vero Health use AI in terms of, you know, like solving your problem? First of all, just in brief, Firo has developed the SAS platform for designing clinical

trials. So this is a really intensely complicated process that can take, you know, like a clinical trial can take. Upwards of 10 years. It costs on average 1 to $2 billion per drug to get to market. Like these are huge numbers and huge time frames. And so we developed a platform that allows essentially biotech companies and pharma companies to design better clinical trials. So we're applying. AI in a couple of different ways. First of all.

One of the big time and cost thinks in the whole development process is clinical writing. So writing really complex clinical document like the clinical protocol that essentially defines the drug trial like so it lays out the whole design and the schedule and all the things that need to happen, all the details. So this is a really complex document that can be north of 100 and 5200 pages and typically takes an entire team of people to write.

It takes months and. We. Starting last year, we started applying large language models to essentially automatically generate various sections in the clinical protocol document and. You might think, well, what's the big deal? You know, it's easy to ask GPT to go and generate a document, right? I would do a little time. It's a write better emails or they're, you know, do my homework or this particular. Use cases, right? But it turns out.

That writing clinical grade documentation that passes master with regulatory groups like the FDA and where the stakes are really high, we're talking about human lives and human safety and things like that is incredibly challenging. And so we went through many iterations when we started designing this system to try to figure out the best way to actually produce really high quality clinical output that meets all the needs and requirements of a clinical

protocol document. Yeah, this resulted in a lot of new architecture and new processes and all sorts of interesting things in order to make that happen.

Right. It's very interesting that you mentioned applying like generative AI or LLM to solve something that is, you know, highly regulated. And it is also kind of like mission critical, right? So to speak, Right, Because it involves people's lives. So people, you know, these days know about the danger of hallucination and all that. How do you actually tackle these kind of things, applying this to kind of like this highly regulated and mission critical kind of a problem?

Yeah. I mean, at first we sort of naively assumed that we could just adjust our prompts. You know, every time we saw a hallucination, we're like, oh, we need to go back and change the prompt. But it turned out that what we really needed was to evaluate comprehensively the output systematically, right. And so when you generate. Say a PK sampling section or ECG

section. Like some highly typical part of the protocol dock, you need to actually evaluate the output according to a whole bunch of criteria. And do this. Per subsection, because essentially each subsection is so different and has different sort of technical details that are fed in through our fire study designer tool. So we have this whole staff platform that's modelling the trial in great detail and we pass that data into the LLM. To generate the protocol

document sections. And so we found that we had to really rigorously sort of query the results and say, well, did you include this? Did you have this? Tone, did you sort of like phrase this in this way? And we have this very specific checklist for every single subsection that we run to make sure that the quality is high. And we've also designed this in such a way that customers can

actually make their own checks. So if they have additional requirements that are particular to them, they can augment or change the ones that we ship our product with. So we've made it very highly configurable to match the customer's needs. So this is what we found was

needed. And over time, that initial evaluation architecture has evolved into more of an agentic architecture where there's all sorts of different agents that are examining the content of the generated clinical protocol section and looking at it for stylistic consistency with the rest of the document and tone and deep level of detail and formatting and stuff like that. In addition to the factual content to prevent hallucinations and also. In addition to. Hallucinations.

We also found that sometimes the LLM just misses things, so it doesn't make things up, it just misses things. And so we have to sort of check for that as well. So there's a whole bunch of examination that occurs once you generate the output, and then you feed that feedback from the evaluation models back into another iteration of the

generation cycle. O we developed this whole architecture that's way more complex than just sort of sending queries to chat GT in order to get the system actually working. Yeah, you mentioned about

missing, you know, information and also again, like the danger of hallucinating, right. So how do you actually do these kind of checks? Is it like still human involved, like at the end someone who is, you know, maybe capable enough or expert enough, like reviewing the output? Because I would imagine, right, these kind of documents is highly technical, right? Sometimes you probably need a little bit more time to research whether it's factual or accurate

or not. Or do you completely rely on AI agents to do that for you? And if so, like how do you to build this kind of like evaluation using AI? Is it like something like eval model that people are using to solve, you know, certain kind of a programming challenge and things like that? Yeah, I mean in this.

Respect. What we've built with clinical protocol writing is kind of similar to other AI systems I've worked on in the past, where in the early stages when you're first getting the AI model to kind of start producing interesting content, you need to be very, very hands on and have a human examine everything. With a fine tooth comb. And then after a while you can codify that feedback into these checks and into queries that you can then use to sort of automate

the process. And after a while, you build enough confidence against a range of different inputs, you know, looking at different types of therapeutic areas to make sure that the content is right. Because looking at a cancer trial like an oncology trial versus say an immunology trial or infectious. Disease trial, it's just kind of really different. And so we want to make sure that the models we build are applicable across a range of different types of clinical trials.

And so once we reach a level of confidence that, oh, it looks like, you know, in the 80%. Case the content is pretty good then we're confident enough to share this with our customers, our early adopted customers and sort of get them to evaluate themselves and they probably come up with some issues as well. And then we sort of gradually improve the quality of the model using that process.

But we have a quite an in depth process involving engineers, you know, data scientists, clinical writers, QA, all sorts of different people involved in this to sort of bootstrap the process and get everything running. And then after a while it becomes more automated as we refine the model. Thanks for sharing that.

So apart from like, content writing, right, is there anything else where you apply AI as part of the work? Oh, yeah. So we're also looking at actually developing what we're calling an AI research assistant. And so the idea is that, you know, right now actually designing the clinical trial is quite laborious, like choosing what kind of activities you want to include in the schedule, designing the population schema, all these kinds of things, objectives and endpoints.

These will require a lot of care and in many cases a lot of research like going out there on the Internet, you know, checking out the latest clinical research that being published, taking a look at identifying and taking a look at comparable, the trials that are out there, learning from them, reading through written hundreds of pages. This is super laborious and we can actually automate a lot of that.

So we are applying AI to essentially help to automatically identify trials that are similar to the one that you are working on. And then you know, aggregate metrics and stats do comparisons. Look at what happened with these trials. Did they require amendments? Like why were they able to enroll patients? Were they able to, you know, stay on track? Did they result in a successful

outcome or not? And so clinical developers who are or clinical scientists who are working on these trials can essentially really accelerate the process by which they come up with an optimal model. And of course, you know, we can provide insights, we can say, oh, here's your patient burden

of this proposed trial. Here is your, you know, the site complexity and cost lots of different metrics that help guide what you're doing and help you come to the right balance of patient experience, site feasibility, and overall cost

and time to market. So this is a really sort of multivariate kind of thing that people are trying to optimize when they put together these trials and it just staggers me that how many people, many groups still do this in Microsoft Word. It's like sort of my sort of catch cry is like not even Excel. You've used Microsoft Word to essentially model much author of model these trials and I think that you know, people deserve a

better tool than that. So that's what we're developing with the help of AI is like really modernizing and shifting the way that people develop these and design these trials in such a way that it makes it much, much quicker and with much better outcomes. Yeah.

So since this is like a very highly niche, you know, specific kind of a problem domain, right. So I'm just assuming like people know things like Chachi, PT and Gemini and all that, right. It's more like a general purpose AI models. Do you actually have to fine tune or even build your own model to work on this specific niche? Because like one of the most important things when using general purpose AI, it's like, what kind of data are they

trained on, right? For example, if you've never seen this kind of problem before, I'm sure AI will hallucinate even much worse than, you know, if you train it with specific model and data. So do you actually have to tweak or fun tune the model or even come up with your own model? Yeah, we haven't found the need to do that yet.

So the good news is that a lot of the major LLM AI vendors like Google and Anthropic are actually focusing quite a bit on the life sciences space and producing models that are kind of specific to that domain. And so we've designed our system in such a way that it's kind of agnostic to the model, like we're not fixed on GBT or Gemini or Anthropic or anything like

that. And so because we have this evaluation system, we can pretty easily evaluate new LLMS and figure out which ones are going to produce the best clinical output. I would say it's quite possible that we might in the future want to potentially have our own model with respect to maybe vectorization like which is a process that all the text kind of goes through before it gets

indexed. So we built this kind of retrieval augmented generation RAG system that allows customers to sort of upload documents that we can then parse and use to generate certain pieces of the protocol document that require really technical information. And so that's an area where we might want to develop our own vectorization scheme that's really oriented around clinical terminology. But so far we haven't. Seen the need to even fine tune

an existing model. Even fine tuning these days can be quite an expensive and laborious process. And so we haven't really seen the need to do that yet. What we've really seen is that this evaluation model can combined with the ability to actually sort of fine tune prompts is enough to be able to produce clinical grade documentation. It might change in the future as we get into different documents and and domains and so on, but so far that's been really good for us.

Yeah. So what I know when you do clinical trials, right, like you mentioned, it could be, yes, right. It could be a decade or even more. Not to mention, you know, like having to do the testing and wait, you know, for the effect in multi years. So using this kind of technologies, what do you see potentials, time savings or maybe even effort, right. You mentioned like $2 billion just to finish the whole clinical trial.

So maybe what are some of the things that you think can be reduced or optimized by having this kind of AI tools? Yeah, I mean, the interesting thing is that when you're talking about a project that takes, you know, upwards of 10 years in many cases or more to execute, it means that small changes in the design upfront can actually result in huge cost savings. So we actually published a paper with Merck. Maybe we can include a link to it. It's on my.

It's somewhere on my LinkedIn profile that. Sort of analyzed like what is the benefit of using the fire study designer to optimize clinical trial design? And we found that the savings over the course of the lifetime of the trial can be north of $100 million. So obviously this was a really big Merck is a huge, you know, top ten, top 20 pharma company and so on. And so that they do these really significant trials, but it just gives you an idea of what's possible, right?

And this was even before some of these AI features that we have coming up like clinical writing and so on. So the true savings actually could be even more than that when you factor in the time savings as well of accelerating the clinical writing process and making the output better. Because getting to market faster is hugely, hugely valuable to a

pharma company, right? Like every day is like hundreds and hundreds of thousands of dollars of additional revenue that you can realize sooner when you divorce the drug faster. So it's not only cost savings, it's also getting to market faster with the new treatment. So we're really excited about that because there's also the potential to really over time, as this technology becomes widespread to really sort of like for a smaller players to come in here and really try

things out. And it's going to kind of enable a wider range of different treatments and trials to become more feasible. And we're really excited by that because there's so much innovation going on in the very early stage drug discovery realm, but it gets sort of blocked or delayed when it comes to actually going through trials. And so we want to really widen

the pipeline there. And so hopefully there'll be a lot more treatments for more conditions and lives saved and suffering alleviated all the good things.

Yeah. And hopefully reduce the cost of the kind of the medicine that is produced out of the clinical trials as well, so that people can get much more options in terms of availabilities and also like the cost is not less expensive. So you mentioned about regulations. So I think in some parts of the world, they are still kind of like cautious about applying AI technologies again, especially if it affects a lot of

population. So what do you see some of the stance from regulations or do you even have to convince them hard to actually ensure that the process is robust enough? I think that by their very nature, regulatory bodies are very, very oriented around evaluating the quality of what's

presented to them. And so at least in my mind, like if the quality of the output is good, at least as good as human generated protocols and we strongly believe that it will be, then the regulatory bodies will be OK. I think there are various risks involved in using AI. There's many, many different types of risks. And so we can mitigate a lot of them. Like for instance we are running our AI models in a sort of a

private. Environment where data is not shared with any other groups and so on. So we don't have any of those risks of leakage and people being able to kind of discover what other companies are doing or submitting. So there's that risk that we can mitigate. I think there's a bunch of others involving potentially bad actors coming in and trying to sort of discover things they shouldn't be discovering by manipulating queries and stuff like that.

That's mitigated by the fact this is an enterprise tool. It's not a consumer tool. And so only people who have signed up as Faro, you know, customers will be able to use the. System so we just have to go case by case and look at all the different risks involved in AI and mitigate them and we have AAI governance policy that we've been working really hard on to sort of like really.

Alleviate our own customers concerns because however concerned the FDA is, you know, our customers are even more concerned about all these things because it's their crown jewels, it's their intellectual property that they're, you know essentially developing using our platform and so we take this

very seriously and. We want to actually at some point probably publish our AI governance policy so that other people can learn from it and we can help raise the bar across the whole board in our industry because we just think this is really important to. Everybody to get right.

Thanks for sharing that. So any kind of a cool success stories that you have done so far, you know, with Farrow or is there anything that is already applied in the, you know, real world or in the life science industry? Yeah, I mean, I think that again, the MOC study is sort of what we have as far as publicly available case studies of just how impactful this technology can be.

I would say just anecdotally, like we have customers using this document generation system and they're super impressive how quickly it's able to come up with documentation that normally takes potentially months and it can be generated in a matter of, in some cases minutes. That's a startling extraordinary kind of multiplier in terms of productivity. We're just really excited about that.

You know, like, I think that in some ways the industry is waiting for a really widespread and impactful application of AI because often times when you get past the surface level of like, oh, I play with GBT and, you know, it's really cool and interesting. Like once you really start to get serious about applying AI, then that can rapidly turn into disappointment or sort of like, oh, you know, it doesn't go deep

enough. And there's all these stories of people who start building with AI, but then they kind of get disillusioned because it turns out to be way harder than they think. And so that's where I guess, you know, companies like Faro come in, is that we'll take it to the next level. You know, we will figure out all the hurdle and the challenges and the risks involved in actually applying AI to do something really impactful, like want to make clinical, you know, protocol documentation or like

optimize a trial design. And so that our customers don't have to do that themselves because that requires a lot of data science expertise, software engineering expertise, clinical design expertise, and so on that they may not want to sort of invest in becoming a tech company themselves. And so we're really excited about that.

Yeah. So I think these days some people also kind of like envision the AGI thing, right? That could happen in some years. I don't know whether it's short term, medium term, long term. So for someone who has worked on this field, AI fields for quite some time now, how do you feel about this AGI? Is it going to happen? Are you bullish about it or do you have some kind of scepticism? So maybe share a little bit of

your thoughts here. Yeah, it seems like, you know, in the past, everybody thought AGI was really, really far away. Like the joke is that for the last 50 years of, you know, AGI has been 20 years away. So so, you know, sort of like always in the future. But now there's a lot of very prominent people in this field who believe that it's actually only a matter of like three to five years away, maybe even shorter. Who knows?

And I am not in that camp. Like, I just think that there's a test out there called Arc AGIARCAGI and it's sort of presents itself as being like, oh, this is a test, you know, for AGI. And you look at what this test

actually involved. And it's the kind of thing that many of the questions are sort of like a bright kid could answer these questions, you know, like moving blocks around on a grid, you know, in such a way that the block sort of fills in a space, stuff that smart kids should be able to do. And I sort of to me, and you know, there's systems that is, that is gradually getting better at this, but it's not yet human

level. And I just to me, that just means that we're so far away from having something that's genuinely creative and intelligent. Now, I will say that there have been really brilliant moments in the history of AI, like if you remember, you know, Alphago, which was this non LLM based system was using reinforcement learning, was using other methods of AI that are now completely sort of forgotten in the wake of the LLM revolution.

But Alpha Go there was at the moment when it was playing against the world human champion and it made this move that nobody in the 3000 year history of this game had ever seen before. And people were just all the experts were like gasping like, Oh my God, this is incredible. What a brilliant move. Like how did I come up with that? And so, you know, AI can, under the right conditions, be capable of behaving like a genius. But it's such a special that was such a specialized example.

And so I'm sure as time goes on, there'll be more examples of that where, oh, there's a flash of brilliance and AI came up with some mathematical proof that was beyond the, you know, like what any scientist has come up with. But I think in terms of just having an intelligent conversation and really being inspired and looking at a piece of art and being truly like this belongs amongst the classics, I think we're a ways away from that.

But I could be wrong. I mean, this definitely the progress even since these live language models are first introduced has been really nothing short of amazing in terms of the increasing capabilities and so on. But I think AGI in terms of like this is a peer to humanity. This is a, you know, like a system that we can have a really truly intelligent conversation about life and philosophy and things like that. I'm not so sure.

And I sort of, I guess I come down on the young Likun side of things where just knowing a little bit about how these systems work, it is sort of like a super glorified, super, super well read autocomplete. And I know that there are efforts out there to introduce more real world knowledge and factual information into these LLMS to give them some grounding

in reality. But I think that's going to be a long path to make sure that those that grounding is actually sort of super accurate and meaningful. But we'll see, Yeah. And I mean, those kind of the

LLM Gen. AI leaders, thought leaders, right, think that maybe the path to AGI is using LLM. I'm personally also not very convinced since simply because of the way it works right at so many probabilistic thing that could go wrong as well. So do you think LLM or generative AI kind of things is going to be the path to AGI? Or is it more like combining all different, you know, AI, machine learning, kind of like capabilities that we have?

And I haven't really heard about people, you know, trying to combine, combine different models from, you know, LLM or maybe other things, neural networks and all that being combined to kind of like solve a particular problem. So do you still see some rooms where we can apply, you know, more AI advancements to the part of AGI? Yeah. I mean, I find it hard to believe that an LLM based architecture alone would be

capable of producing AGI. However, as I said before, you know, there are groups that are looking to merge in knowledge systems that are aware of facts in the world and how they relate together. Kind of going back to some of the really much older architectures for AI where they were attempting to like CYC, where they were attempting to actually model reality, like create this big huge knowledge graph that links together all these different concepts that

collectively model reality. This was sort of a path that AI took a long, long time ago that was kind of abandoned when deep learning came along in favor of a more statistical probabilistic methods, including LLMS. Ultimately. So I think, you know, I can't help thinking and I think I'm not alone in this. I'm pretty sure well others might agree, like some kind of Harvard where we have this fantastic language generation capability of the LLMS combined with an actual. Sort of factual.

System that helps to guide the LLM to not really say stupid things sometimes that ignore reality. And we'll just have to see how that goes. You know, like it's one of those things where the more computing power and the more efficient these algorithms and architectures become, the more things we can try out.

And of course, there's a whole wild card of quantum comuting, like once quantum comuting reaches a level of maturity where it's able to be practically applied to this and all bets are off as far as what's possible.

Yeah. So at one time, I feel excited, but at the other time, I feel a little bit scared as well. You know, so many people are thinking about jobs being changed. You know, people's roles are going to be diminished and all that. And obviously, you also have dealt with this kind of worries as well. What do you think people should do in this kind of era where things moving so fast every

other day? I think you might have heard about cool things being applied using AI to improve productivity, remove redundancies and all that. So, yeah. What do you think people should do in this era? Yeah, I think first of all, maybe a little bit of an inspirational story. Like I love to give this example of radiologist.

So, you know, seven or eight years ago, Geoffrey Hinton, who's a brilliant guy, like one of the forefathers of deep learning, one of the people who really helped make this whole AI, the first AI revolution really happened. He made this prediction that, you know, there would be no more radiologists within a few years, that we should just, you know, forget about radiologists because they're going to be replaced by computer vision based systems.

And Fast forward to today and there's more radiologists than ever. So how could that be? And the answer is, well, you know, radiology is a high stakes kind of field in the sense that there are human lives at risk here. And so there is a need for a human in the loop. And the fact that we automated so much using machine learning means that essentially there's just more and more people who are out there looking to actually have radiology based procedures performed.

Like these days, if you get an injury, it's like, Oh yeah, just get an MRI. You know, like just get a diagnostic MRI to find out if you might have some sort of problem. Like it's. Become so commonplace that could not have happened, I don't think, without these computer vision sort of advances. And it's resulted in actually more radiology jobs because there's still needs to be a human in the loop.

And so it's kind of a volume thing where even though a lot of tasks would have made it, the volume increased. So that's point #1 and I think point #2 is there's this quote by, I think a former CEO of IBM saying, you know, AI won't replace humans per SE, but people who are able to effectively wield AI will definitely replace people who

aren't. And so if you're listening to this, if you're going to take anything away from this whole conversation, that would be really immerse yourself, like look into how you might be able to use AI to make your own job better, to make the way you do your job better. There was a study, actually a very recent study published by the Harvard Business Review called the Cybernetic Work Companion, something like that. If you Google have a business review and cybernetic, you'll find it.

And it's super interesting. Like they actually quantified this and said, oh, you know, somebody who's with the help of AI, suitably trained, can actually perform the work of two people. And I think that actually might be sort of higher than that, but it just goes. To show like wow, you could be at least twice as productive potentially using AI so. Figure out how to do that, because if you don't, someone else you know might. And so that's what I have to say about that.

Yeah. So definitely you have to give it a try, right? And know what are the possibilities? Because sometimes like even out of our imagination, our current imagination, right, you wouldn't think that these kind of things can be solved by AI or some kind of a tools that can actually shortcut, you know, the time and effort required, right? So knowing just that possibilities, I think it's also really important and especially if you have used the system right.

I mean, as you can imagine, we've been really looking into the use of AI internally, not just in our product, but internally the way we developed our technology and our software. And at first I was sort of a bit of AI would say a sceptic, you know, thinking like, hey, it's one thing to create a proof of concept or an early stage prototype using AI, but actual

production code, no way. You know, we're still going to need a whole bunch of software engineers doing things pretty much the way they've always done them to do that. But now I'm not so. Sure, like we've had engineers do extraordinary things. Using the latest version of these code generation tools. And I really am starting to fully believe that the Harvard Business Review articleized, I have mentioned before about how people can be twice as productive. It absolutely holds for software

engineering. And so, yeah, I mean, we're practicing what we preach. We're using AI to develop software a lot more, you know, productively and faster than before. And it's a pretty exciting time because the underlying tools also are evolving so fast. We're finding that the latest versions of the code. Generation tools are just way more capable than the ones even just from a few months ago, so that's also really exciting to be surfing that wave of advancement in AI in that way as

well. Yeah, personally, as someone who have been around in the industry for quite some time, right, I find myself during this transition also a little bit skeptic at once, but actually seeing maybe the junior people doing stuff that I couldn't imagine. Sometimes I feel like, wow, amazing. Like sometimes I'm having to, you know, switch my perspectives to be more optimistic rather than skeptic, right?

I think this is also maybe some challenge for those of you who have been around in industry because we used to think this is how we solve things. But now I think the possibilities are there, right? So as long as you can imagine, you give it a try and maybe I could help you in some parts of the job. And I think your productivity can be proof as well. So you mentioned about

engineers, right? So let's switch a little bit to that side. So is there any difference managing or leading AI engineers versus typical software engineers, You know, like things like back end, front end and those kind of things? Yeah. I mean, even before the LLM revolution came along, it's always been quite different managing data scientists or ML engineers as they sort of used to be called, right? Simply because a lot of what you're doing is exploration.

Like, I love what you're doing is you had this idea like, oh, you know, we can use data science to solve some problem, but it's not like traditional software engineering where you have this idea, you design it, you specify it, and then you sort of build it all and according to spec. And it's a very deterministic process. With data science in general, you know, you're just going to find out, like, is this feasible?

OK, I have some idea. I can predict, you know, let's just say I can look at the whole bunch of historical trials and I can start to predict things like amendments or outcomes of any kind. That's a thesis that you need to go out there and just test kind of like science, right? That's why they're called data science. And so that inherently is quite different because you have to

embrace more uncertainty. Now I think maybe people also want to sort of know what about all this AI vibe coding or AI enabled coding? Like how does that change? And I think that what it does is essentially it's a force multiplier for people. So maybe to build a prototype these days, you don't necessarily have to be a software engineer anymore. You can get to a certain point without any software engineering skills whatsoever.

So that's kind of different. What we're observing out there actually is that there's a lot of entrepreneurs and even sort of like groups that are out there selling who are basically building prototypes where before they would have been presenting marks or, you know, conceptual sort of descriptions of what they intend to build. And now it's like, no, we actually just went ahead and built it because these coding systems, these automated coding systems make things so much

easier. So there's this huge shift towards essentially what a lot of the people who are really behind the agile process we're already advocating, which is you just got to prototype stuff, try it out and test it with real customers. And I think in the past, a lot of people didn't do that because it was just too much work. It was like, well, you know, prototyping, it just takes so much work to try to make all these ideas real, like, and then

potentially throw them away. It's just like we don't like throwing things away. Whereas now the bar is so much lower as far as what the amount of effort it takes to produce a working prototype. That it's much more. Feasible to do what people like Marty Kagan, you know, out there say, which is just pretty like a whole bunch of stuff, you know, within two weeks or less. Before you fall in love with the idea, just test it out and throw it away if it doesn't work.

So that philosophy and that process will become way more feasible with AI, which is really exciting too.

Yeah, definitely makes sense, right? So the process, the iteration now can be much faster and even people are not skilled enough in development now can do software development in some sort of form, right? So as Acto yourself, you mentioned about vibe coding. What is your view about vibe coding? Do you see your engineers doing some kind of vibe coding? And if so, right, is there any danger that you foresee and how do you actually mitigate that? I think vibe coding is great.

Meaning just using these AIF systems to generate code that you don't even necessarily look at, you just prompted and prompted and prompted until something works. I think that's great for doing proof of concept initial 0 to one kind of prototyping. I don't think that's a viable path to actually building enterprise software simply because when things go wrong, as they inevitably will, you know,

with AI generated code. Like I started playing with some of these vibe coding tools myself and very quickly I dug a hole. Like once I passed a certain threshold of complexity, it just became really hard to add new features. They had this magical button called like try to fix it. And I was pressing that button a

lot. And so I don't know, like I, I, I think it's got a place and I think it's wonderful that people who don't have a computer science background or haven't or not, you know, experienced software engineers can now actually build technology like that's magical, that's incredible. But it only goes so far. And I think in the future, sure, vibe coding will become more powerful. It will become better.

But I do think at the end of the day, if you are producing enterprise grade software that's used by real paying customers, you're going to want to have people who actually understand the technology, even if they're using AI to help them code. To actually fix things when they go. Wrong and to. Ensure that things are designed in a good way. At least that's my view. Yeah. So you mentioned like building

enterprise system, right? So the things that are complex and maybe need to evolve over time, right? Because I still don't know whether you know, just simply doing vibe coding can actually evolve your system, you know, in a predictable manner, right? So rather than, you know, keep prompting AI until it gets fixed, right? I think the code that gets produced also can be quite terrible if human are looking at it, right? Because simply they don't maybe adhere to a certain design

pattern. There's no refactoring done in a concerted effort, I guess. So I think vibe coding is definitely is wonderful for things like POC or things that you're not looking at or something that is more siloed. I guess that you can just change the whole thing once you think the quality is not there. So I think also another thing

when we hire engineers, right? Do you see any kind of difference these days in terms of, I don't know, attitude, skill set or non-technical skills that you see when you hire AI engineers? Yeah, I mean, I think that we're still at the early stages of this. And what I do observe as a sort of a hiring manager is there's a lot of resumes out there with AI on them because everybody wants to be as perceived as someone who's versant in this because they know that there's a lot of demand.

But when we start interviewing, when we start testing people's actual coding capabilities, because this is still relatively recent, it's still kind of difficult to find people who really do have experience with this and who have demonstrable

ability in this area. And so I think it's going to shift over time, just like in the old days, you know, it was kind of hard to find like way back in the 90s was hard to find people who knew HTML and he knew like how to build full and then later to who really knew how to build full stack applications. And before too long, within a few years, everybody did.

It was the norm. And so I think this is just a natural technology adoption curve where we're sort of moving into a more of a mainstream situation where everybody's seen the value and. You know the majority of. Software engineers out there are going to be conversant in this technology and before too long it'll be ubiquitous. But we haven't quite seen that in the hiring pool yet. But I think, as I said, it's sort of inevitable.

Maybe from your existing engineers that you have in the companies, Do you see someone that stood out simply because you know they can combine all these possibilities and also with a certain attitude, do you see some kind of persona of engineers that stood out that you think you can share with us here? Like, what makes a better engineer now that AI technologies are there?

Yeah. I mean, I talk a lot about career development and about the qualities that I've observed over the many years that I've been working in this industry that lead people to become successful and sort of more, quote, senior in some way. And one of the big ones is kind of curiosity. There's this thing kind of inherently excited about learning new things and trying

things out. And that's a quality that I think really, really serves people well in this new world of AI, because a lot of it is kind of unknown. It's like just trying to apply AI in ways that. Have been done before. Like we're doing here at Faro. And so I do think that that's an example of a quality that is

really a strong one. And I think also people who can span multiple domain by people who are able to think in terms of product, like what makes a good product, what makes a well designed product like usability wise, along with the ability to sort of decompose that into commands that you can give to the AI that it's going to understand. So this is a combination of different skill sets. It's not just like you do a computer science degree and

you're good. It's more like, hey, there's a part of this which is customer facing, like knowing what customers really would respond well to this. Part of this, which is communication, being able to kind of express those requirements well, like a product manager does. And it's part of this which is like UX designer and of course part of it, which is straight up

programmer. So I think the people who can think cross functionally also will do really well in this new world because they now have the force multiplier, you know, in the form of these automated coding assistants. And so on to really be able to do the work of multiple people potentially. Yeah.

So definitely, those skills are still kind of like the most important things, right? Especially if you are engineers listening to this and worrying about your jobs getting disappeared simply because getting replaced by a IS. So how about leaders themselves? So I know some leaders maybe have decades of experience. Maybe they are up and coming leaders as well, like engineering managers and all that. Do you see a certain skill set or attitude that must change in this EI era as well?

Yeah, I do think that every time there's a technology disruption, it can be a little bit scary simply because perhaps the processes and tools and behaviours that have served you really well in your career to date, it might be sort of under threat or, you know, about to be transformed. And that can be exciting. And it can also be kind of scary and threatening like, Oh my God, what do I do?

I can't depend on the same set of skills and habits and behaviors that have got me to where I am in my career. And I think that the people who can sort of take that uncertainty on board and re step into the unknown are the ones who can thrive. And I think that applies at a company level as well. Every single company out there at this point is wondering like what do we do about AI? And I think it's the ones that are bold and are willing to

disrupt themselves. Like I, I just spoke to the company earlier on this week and said we're essentially disrupting ourselves. Like we started this company as a traditional SaaS provider. Now we're becoming an AI company and that is a process of disruption.

And I would look at Apple in the years of Steve Jobs, if you want to look at an example of a company that's has the courage and the sort of the fortitude to disrupt itself repeatedly like Apple did that you remember, like the iPod that used to be the best thing ever, like Apple disrupted their own product with

the iPhone. And I think that that is the kind of thinking that is needed here when this new technology wave comes along at the company level as well as the management individual leader level is giving your teams. First of all, have possessing that courage to step into the unknown yourself, but also inspiring your teams to do so and saying hey, this is you know, if we don't do this, someone else will and we will get disrupted.

And you can look at the history of this, like if you look at things like the Innovator's Dilemma, you know, Clayton Christensen's book that talks about this is like a natural law, like companies get disrupted because not all of them have the fortitude to actually jump on new technologies and disrupt themselves. Yeah. So I think thanks for pointing

that out, right? So definitely it's something that for some of us leaders, right, have to think about disrupting yourself, disrupting your team and disrupting your company, right? So I think those things can be healthy, especially if it can also revolutionize the way things are done. So how about the juniors here,

juniors who probably are thinking whether software development is going to be their career or someone who is studying software engineering. But I think that they now have to switch. Do you still think there's a room for them to actually enter the industry? Because I heard so many times that people say simply because you can be more productive now, maybe we need less junior engineers. So what is your view here?

I think that there's never a shortage for people who are able to create technology, and AI makes that easier. And so, yes, you could argue the field of programming is becoming more commoditized because of all this automation. But like I said before, AI can't necessarily take the place of having good product sense or having the combination of programming skills and design skills and so on.

There's different ways of framing this, but I would say thinking of it as being a force multiplier, like you're able to do more, whether it's as an entrepreneur or whether it's as an engineer who's part of a larger organization. If you're able to wield these tools well, that'll serve you very well. And I think we're still also in

the early stage of this. So if you are able to, for instance, start a site project, a personal project, and put that in GitHub or put that on your resume and show that you have the ability to do this. Even if it's some personal project you just happen to be really excited about, definitely do that. Because that shows that you can do it and that shows that you can adapt and that you have some level of mastery or facility in this area and that will serve

you well as you look for jobs. Because the employers that we're looking for people who can do this now and they're still kind of hard to come by. Maybe not for too many more months or years, but there's an opportunity to jump in there if you really are proactive and potentially stand out from. Other candidates because of that.

Yeah. So I think definitely don't get this heartened especially if you're really passionate about software engineering, right, creating application products and even like enterprise software still kind of like the go to things that you are passionate about. So I think that don't get this heartened and maybe a try out stuff just like Patrick mentioned.

So we have talked a lot about you know, AI in clinical trials, AI in general and also software engineering and engineering teams. Is there anything else that you think you want to share with us today that we haven't talked about? Maybe just coming back to the theme of career development, like some of the most successful and interesting careers I've seen of all the people I've worked with have been people who really focus on just trying things out. There.

Was this one maybe an inspirationalist degree? Like there was this one really junior product manager I worked with when I first joined Google and I encouraged them to sort of become a user of our application, which at that time was this kind of e-commerce product. And so I said, hey, you should probably sign up as a merchant, start your own little sort of fake business and kind of try out the product. And he ended up really getting

into e-commerce. Like he built this company that ended up becoming successful and he ended up sort of becoming AVP at Shopify. And I just kind of, I was giving my talk earlier on this year at the company on career. Development and I actually spoke to him and just said, hey, just how would you? Sum up like your success because it's like such an incredible. Career path that he followed. As an entrepreneur and now senior executive and. Like all these things. And he just said, you just got

to try things out. And he reflected back to me, like AI just enables so many more people to do this now. Like to do the kind of things that he did that at the time required software engineering expertise, machine learning expertise. Like now you can potentially vibe code some of the stuff that he did really easily.

And so the opportunity for people who have ideas that they observe problems in the world, the gap between observing that problem and trying to solve it is now much, much smaller than it was. And so whether you're trying to impress some big company that you want to get a job at or whether you actually want to start a new company. It's just much easier to do that.

So I would. Sort of, you know, learn from his experience and maybe be inspired by his story because, you know, he was really, really understoring the fact that it's just much easier to try things out these days. And trying things out is often what leads to great inventions and innovations in the world. Thanks for sharing such inspirational story. I think definitely worth to

think about, right? So especially you having cross functional skills back when you mentioned about cross functional thing, right? If you have cross functional skills, definitely the opportunities and the possibilities are there for us to give it a try, right? And vibe coding, yeah, it could be one way to actually build POCS and kinda like test the market early before you actually build the actual product. So Patrick, thank you so much

for sharing today. So I learnt a lot, especially from those inspirational stories that you mentioned. But before I let you go, I have one last question that I always ask my guests. I call this the three technical leadership wisdom. You can think of it just like an advice that you want to give to the listeners. What are the top three things that you want to share today? Definitely, I'll return to the curiosity.

Like that's super important. I would say definitely following your excitement and all things like just don't settle. For doing work that you don't find and personally inspiring. That you don't feel like, oh, I'm excited to get to work in the morning. Like if you don't feel that way, then think of other things that you might get more excited about. And it sort of sounds obvious, but I think that people get conditioned after a while to not

doing that and just settling. And I think that it's important to really be excited about your work for a number of different reasons. And I think the third thing would be definitely, I always advise people to find a mentor. And that also sounds kind of obvious, but I think the majority of people don't actually actively seek out a mentor. And it can be super, super rewarding. Like I keep coming back to some

of the people in my own. Career who have really helped me, even if it's just a sentence. Or a catch phrase or a certain just way of doing one thing. It can change your life over time really by just helping to reframe problems or deal with, you know, communication issues or whatever the case may be over time in your career. That can have a really compounding effect. So seek out at least one mentor, ideally multiple over the course of your career. And it's really.

Going to help you. Yeah, thanks for pointing that out. So you mentioned like even a simple sentence phrase and all that could actually change someone's lives, right, And can change your life as well. So I think for people who have been around in the industry as well. So be a mentor, I think I would say as well, because you never know how much impact that you can make to someone's lives. So, Patrick, if people love this conversation, they wanna reach out to you or maybe learn more

from you. Is there a place where they can find you online? Yeah, The best way is LinkedIn. You can find me on LinkedIn pretty easily. Right. OK. So thank you so much for sharing today, Patrick. I really learned a lot. And again, thank you so much. It was a real pleasure to be here with Henry. Thank you.