Hi everyone. My name is Patrick Akil and joining me today is Stacey Cashmore, AI Product Lead over at Omniplan and returning guests from about 3 1/2 years ago. We discussed what she's been building in AI features and how AI has made her more productive as software engineer. So enjoy. I feel like with AI, people are getting to the end result as fast as possible.
And I just had a conversation on, OK, this path that we're on with AI, some certain decisions have been made, certain people were involved, certain companies, certain hype. Is it now good for engineering or not what we have? And I was of the mindset of, OK, you can use it because the gap with regards to knowledge and figuring things out has gotten a lot smaller. So I feel like it, it can be
beneficial. But then the contrary opinion was, well, if I'm trying to generate something, I just get frustrated with the output and I just go trial and error basically, and I don't learn anything. Yeah. Oh. I I've been there with some stuff. I first really started using it because I needed to write something in Python because I couldn't get the C# libraries to work as I wanted them to. Fine AI supposed to be all about Python from everybody that you speak to.
So that's the. Best one, the Python route and I was using Claude and mixed with Claude and copilot. It never gave me an answer that was good enough, but it did always point me in a close enough direction that I could figure it out and I think that's the thing that scares me. Just saying you can use it as a junior developer. Junior developers learn. I wouldn't give it to a junior developer and expect them to be a senior.
I think it would actually go the other way because if you don't have the knowledge that you built up, then you're just going to take what it puts out and it's going to ruin something. So you really need to keep on top of it. I, I experienced this at the weekend. I was writing something, I couldn't find something online. I needed a URL for something, and it's like auth 0 is making it really hard to find. So I'm just going to ask Claude. Yeah. And Claude gave me the URL and
it looked correct. OK, Nothing worked. Oh, really? I was like, what the Hell's going on here? Yeah. So I spent probably an hour adding debugging all over my application trying to find out what went wrong. It gave me an under score instead of a dash part of the URL. I was like, of course that's a dash. But worse than that, it's like, fine, let's tell it that it's made a mistake. So in that same conversation, it's like you gave me the wrong URL. It should have been this.
I gave rid the correct URL. It's like, I don't understand. You've given me two URLs that are exactly the same, and I was like, no, they're definitely different. It's one of them has an under score, one of them has a dash. The URLs are the same. The one that I gave you is absolutely right. It needs to be a dash. It's like, yes, but you gave me an under score. And at that point it was like, I'm so sorry for the confusion. That's like, Yep. So that's like, that was on.
It was on me because I was just being a little bit lazy. And I could have found the problem the solution with about 5 minutes worth of extra searching to get the right URL, but I was feeling lazy and I wanted to move on to something else and it cost me about an hour.
It's humbling. It's very humbling when that happens, and I think that people need to admit it when that happens because you hear all the hype and you hear that this is such a phenomenal thing and it's going to make us all so much better. But if you're not critical enough, then it's going to hit you really hard. Yeah, exactly. I mean, I I see it being and I haven't been in the seat, but I would love to try being very
powerful for early projects. Basically everything Greenfield, everything we need to get something up and running from scratch. Incredibly powerful because you need many, many lines to get something small done usually, and that's what it's good at, it's generating lines. It's great for that. I think as long as you're doing that and you're using it as a proof of concept and not a production system, then I think that is perfect.
You can use it to make a very quick proof of concept, something to just prove your point. Claude makes great interfaces. So if you have an idea for an interface, well what if we did this? It can make you an interface in like 2 minutes. Like a using React. It can write and display React. So it will write C#, but it won't display a Razor component, but it will display a React component.
So you can actually iterate and then it gives you the code and then you can improve that code and take the bits that you need to make it work. That I find really powerful. But the code that it gives you, if you use that off of the bat, it's not going to be maintainable code because I read it then I generally like refactor at least half of it, make it into something that is going to last a longer time. You love yourself. Yeah.
OK, manually. I I do that manually because I rested in the past to do it and it didn't. Even when you're trying to get the perfect prompt, say I spent like 10 minutes trying to get the perfect prompt to could refactor this in 5 minutes. It's just easier to use Rider or Resarper and use their extract methods, renaming, all that kind of stuff. You can take that code and you can improve it in half the time that you can get the system to improve it. So yeah, it it, it's nice, it
works. It does help your productivity, but only if you use it right and only if you recognise that you are going down the wrong Rd. early enough. Gotcha. I feel like I've had conversations with people and usually the thought that pops up in my head is kind of you're playing a game and you have this checkpoint system where you make a safe state where you're happy with whatever is generated and you either manually change a few things, but you, you continuously make little
checkpoints for yourself. Yeah, in essence, those are commits. Yeah, but I have seen people and also from a start up phase go very far with regards to vibe coding, self interjecting, a little bit of refactoring, a little bit of peer review and then making a full-fledged
production system out of this. I have not seen any example to be honest, where there's strong engineering culture in an existing organization, strong conventions where vibe coding then is also very effective because somehow it doesn't adhere to the conventions that you have within your particular organization. It adheres to the general developer on the Internet which does not understand your code base. Yeah, it it.
It's also not necessarily great to be the general developer on the Internet. It's the code is taken from so many GitHub repositories that may or may not be maintained. They may or may not be made by people who understand what they're building. They might be experiments or tests or learning repos. And I think that can be the issue. It's the code that I've seen it
generate. Personally, yes, I could probably make something vibe coding with it, but only if I didn't care about fixing bugs or expanding it in the future. Gotcha. Yeah, let's zoom into some of the AI projects that you've been working on because I I want to get into productivity and I want to get into kind of results from a project based. So let's start there. What have you been building with regards to AI as well?
The first thing that we built was a conversation summariser because basically I managed to get the role of product lead AI. I'm a huge AI skeptic, which I thought was hilarious. Speaking to the guy that's been helping us, he just said no. Being an AI skeptic is awesome because you're not just going to go down the this is cool path. You're going to think about what you can use it for. And that that's what we were
doing. So we had a meeting like we don't just want to do a really bad chat bot, which is obviously what everybody, oh, you've got to put a chat bot on your side and we need customers to just be able to chat with a chat bot and get a mortgage. I like, have you ever tried that? It's chat bots are not designed for that type of interaction because everything here is known. So I didn't want to go down that
street with the first project. And we had three ideas that I gave a presentation on last year at the VIP Congress for the mortgage market. There was a chat summariser, there was the chat bot because. It has to be there. To have a chat bot. Of course. It's I, I really hate to given that part of the presentation, but it needed to be there. And I think we had another one for e-mail generation. Gotcha.
When you've had advice from an advisor, we're expecting people to want the second one, but apparently the conversations that the team had with people who attended said the chat summarizer sounds awesome because we spent so long making notes that this would be such a boon for us. It's like awesome, allowing the fact that if you're online, most software these days comes with a transcription.
We needed to see what value add we can put on top of that because there's no point in doing something if they say, well, like get it over here for free or in an existing thing. So what are you bringing to the table? So what we wanted was a conversation that you can start inside of a customer's dossier so that everything is recorded in context with that customer. So you don't have to switch between applications if you ever want to get the transcript, you can all do it inside of 1 system.
So no context switching. And also at the end of the conversation, make a financial summary of what was in the conversation. So not just a, a baseline, tell me what was spoken about, but really from the point of view of the financial world, what do we need out of this conversation? We managed to build that in 3 1/2 months, which I was really pretty. I have a tiny team. It's it's me and I don't work many hours and I'm stretched
thin. I have a back end developer working with me who is just superb, compliments me perfectly because he really wants to put the cross the TS and dot the IS which I want to do. But I also know where my head is most of the time. So we complicate. We complement each other really well there and when I need it, I have a front tender to help us and three people 3 1/2 months getting our first product live I thought was really quite cool. It's live now. It's live now.
Awesome. So we managed to get that one live and now we're working on a number of things. So taking what we learnt from the implementation of that, we have some longer term things that we need to build that we know we can't roll out quick releases because it's not going to have value until there's a system in place. But we don't want to stand still because you can't stand still in
this market. So we're trying once a month to release extra functionality so that we can carry on doing the cadence and it stays fun to program because you're always doing something new as well as the long one where you're not going to get that dopamine until later on in the year. So we just released the second piece of functionality and that is generating emails based on that conversation.
So you can open the conversation, you can generate an e-mail and you can style it to the customer that you're talking to. So some customers want something that is really I forgot the English to hankluk accessible and they they just need an accessible short description because that is where their knowledge of the financial world is, which is a fine thing. It's you don't need to have that knowledge. Why do you? You can also make it more formal.
You know, some people prefer more formal communication. Some people prefer more friendly communication. Because of my history and the things that I've built, I have a reasonable understanding of what I'm doing. So I want some of the technical details in there because I know what I need to be looking out for.
So being able to do this just in a couple of seconds for your conversation tailored to your the needs of your client, and then being able to edit it for anything that you think, OK, that's this can be better worded. Yeah. So you can save a lot of time in sending these emails that way. Awesome. So that went live this week. That's really nice. That's very recent, actually very fresh, yes. Yeah. So it's we really started in
November trying to do this. We talked about it before then and November was when the really big push started and we've kind of learnt that doing an amount of experimentation before you start building saves you so much time. Interesting experimentation. To what degree? Like to make sure whatever you do is achievable, or the format in which you're trying to achieve it, or. A little bit of both. So the way that did, we did the experimentation in the first month, we ignored all of our
deploy things. It's like, no, first of all we want to know how we can get data out of the two or three models that we have to interact with in order to get our summary, because it both summarises die rises, so you know who is speaking at what time. And then it does a summary at the end. I was like, OK, so how are we going to make sure that we can do this in a viable way? So we started off with just a console application. You just gave it a link to a web file, a video file, whatever we
had available. And we could run through each of the steps and check that we were getting the data out that we needed for each of those steps for the LLM, for the summary, because that's the one where you really need to prompt the other two, it's solved problems. We can just interact with existing infrastructure. So we use Azure Speech Services for that, because why should we struggle when Microsoft's
already done it for us? For the summarisation, that's where the LLM and your system prompt really come into play. So we must have spent a couple of weeks experimenting, trying to find a system prompt that was good enough to get the information that we needed, but also produced good enough results without hallucinations. Because it is financial data. The last thing that you want is hallucinating something in the conversation that can throw off anything in the future.
Yeah, of course. We also learnt the hard way for that. Our original idea was when we have this summary, it has all of the financial subjects belonging to that conversation. So your job, your salary, your mortgage, what loans you have, what savings you have. We have all of this information. We can take it out of that conversation and we can pre fill our data model with it and that would be awesome. We gave up on that after about half a day. Oh, that's quick.
Because we saw the dead end that it was walking into, the amount of context that you need in order to correctly translate these things is huge. So we set up a very quick demo we gave that was with Claude, by the way, that was actually in the Claude browser. We had a fake conversation, recorded it, transcribed it, gave the transcription to Claude and Claude made a fake. I think we just started with a job, so a job financial subject form and it got most things
wrong because. From the conversation. From the conversation, yeah. When you're talking about a job, you don't always talk in the same context. So do you earn 3000 a month? Does that include a 13th month? Does that include vacation money? Is that 4 weeks or is it per month? And these things were just so hard to get correct. And the worst, the worst one that we tried with it during that morning was actually it. We had a conversation with I think 5000 was a salary new 15000A year.
I was like, OK, how are we going to get it to fix this right? Yeah. This isn't supplying the value that we want it to. It's going to take us too long to fight this. There are better ways of getting information into a dossier of a person. We have source data in the financial world now. We have companies like Octo where you can log in with your Digi ID and you can give documents and information to
your financial advisor. So trying to guess it from a conversation, we realise that that was going to take more work and more checks by the advisor then it was going to save. So we struck that one off. It's we didn't run down that alley too far because we could see that it it wasn't going to supply value. And if it's not supplying value, what's the point in stressing over it?
Even something as simple as this, like extracting information from a conversation and then the topic of it being financial loan based, I was like, yeah, that should definitely be doable, right? But that's a huge assumption.
And if you take that and if you run with it and in the end you come to the same result that you came in half a day, then that's a problem because you already have so much sunken cost in all we've been trying to do this, we've designed the user interface like this is all in basically that you would then
get into this cycle. My assumption it's the cycle of trying to fix it. Continuously you get into the sunken cost fallacy, and I am sure before people put it in the comments, I am sure that it is possible. It's definitely doable. But the question is what is the cost going to be versus the value you can extract from it? Because this is only valuable if people are prepared to pay for it. How much are they going to pay you a month to use this? Is that going to cover your
development cost? Is it going to cover your LLM cost? That's the other big one here, that people. Do you want to think about that? Three. No, we figured it out for our conversation summary. It costs about €1.50 per hour. OK, using Microsoft's resources. Yeah, sometimes more, sometimes less. And that's a spoken conversation. That's a. Spoken conversation. That's a lot, to be honest. That it, it sounds like 150. Well, people are going to pay
150 for their transcription. But if you're having three calls a day, five days a week, then you're up to 15 per week, you're up to 60 per month. You need to cover your development costs, your overheads. Are you going to be able to get somebody to pay an extra €100 a month for this? So that's where you have to start figuring out, OK, at what point can we say this bit doesn't provide value and we can approach it a different way and save money for us and the client.
And that's a lot of what we've been doing as we've been working on this. Gotcha. Now with the features that you've put into production, you explained to one that went live very recently and the conversation analysis that you've done, how do you measure then success? Is it by how many times people go in and manually change something? Is it by costs with regards to the output and the outcomes? We are measuring it on a couple of ways at the moment.
It's because we're in total experimentation. We don't have anything solid in place that we're going to have by the end of this year. So our original idea is how many people are going to be using it. It's currently in beta. We're giving people free access whilst it's in beta so that we can see what they think to it. And we have asked for feedback from our customers on top of that. Yeah, there's the cost. How much is this going to cost in the real world, in a real life situation?
Are there places that we're going to see that we do need to improve this with regards to making it affordable? And we're working on that, and we're improving it all of the time so that we can, yeah, get the best value proposition out into the world. But it's all things that you really have to take into account when you're starting this. We started, I started pessimistically.
And yet still at the end of that first brainstorming session, we had such ideas which very quickly proved to be non starters based on what we could see coming out of it. And I think that's something that a lot of people miss. Yeah, Yeah, it might be in the the market that you are as well.
Like the financial domain things are very, very much they end up being perception based like as a customer if the information if, if the information is factually incorrect, then we just had a dialogue and I gave you all the correct information. Like that can be killer with regards to brand and. Recognition, something that we spent a lot of time on. It's we we would rather that we say that there is information that we can't give, then we give incorrect information.
That's the worst. Incorrect. Yeah, it's better to say Nope, sorry, than yeah, give something wrong. This can have it can have serious consequences if the advisor takes it again at face value. You're using AI, don't take anything at face value. So the advisors still need to be in control here. Gotcha. But if you continually give them bad stuff, then they're not going to trust it. They're not going to use it.
Yeah, that's it then. How did you measure then what the accuracy of your system prompt was? Because you mentioned OK this for us, getting the minute details of a system prompt correctly was actually a decent chunk of work. It was a decent chunk of work. We measured it. We, we have an amount of conversations recorded by our financial advisors because our product owners and consultants are all qualified financial
advisors. So they knew what the conversation should be. So we had a number of different conversations that we could feed through the system. My team can look at those and pretty much figure out if it's right or not. But we also gave it back to the financial advisors. So is this what you would expect out of this conversation? And we tuned it based on that. Got you. Yeah.
Are you? Because I'm working now also in the financial domain, we're trying to make people more productive by taking something manual they're doing and then generating it through AI, kind of pre filling it, keeping a human in the loop for validation. And one of the benefits that we have is that this process is already 2 years old, which means we have two years worth of historical data. So we know the PDFs, we know the outcomes and now we put an out of them in between.
So we have a very easy way of measuring if what the output of our prompts are equal to what the historical information was that was actually captured. Like that's how we are doing kind of the validation and analysis. And I really like that from a surface level, sounds very good. We have ground truth. We can work with that. The problem there was that we have the human factor of error is also in there and they're not manually corrected.
So if we're going to fine tune towards 100% with regards to our prompts, then the same human error factor will be in there. So we actually had to go into, OK, what are the documents that are valid qualitatively with analysts being like, OK, this is valid and this was actually incorrect and we should probably make an amendment for this. Yeah, yeah. But did you have something similar with regards to historical information or was this very much qualitatively
done? This was, we didn't have access to historical information for this. Yeah. It's we don't have access to our clients data obviously because you shouldn't be bad. So we we don't have access to that data to run against. So we are running against our own tests and that's the point now putting it into beta for the financial advisors so that they can try this and they can get back to us and we will ask them proactively for feedback on what they the issues that they're
having. We've already seen one issue which we managed to solve that was really odd, but that's that's why you have things in beta and the people using it can then also accept that. OK, it's 99%. It's certainly not 100%. You need to be careful, but it is still saving them time. So we're happy. How was it with regards to their adoption? Because you're mentioning, OK, they're happy now.
Were they happy from the start? Were they kind of fighting against this change or what was that adoption curve like? The adoption curve, yeah, it was very low. OK. Because yeah, it it is AI coming in to do stuff. And do you trust it? So we have a lot of communication that we are building up regarding this functionality. Gotcha. Yeah. So we're we're going to start to do a video series on it as well. Oh, that's nice. Because telling somebody about
something is one thing. Showing them it in action is something entirely different. Yeah, exactly. So that's, yeah, that's what we're currently working on. It's a little bit of content creation just to just to make the job more interesting and a little bit different for a while, so. Yeah, it's I like that a lot actually. Like it's something I didn't
necessarily realize, right. You're building something to make people more productive, but the only way they are going to be more productive is if they actually embrace whatever you've built regardless on if functionally it does the right thing. Like especially with something so delicate as generating the e-mail conversation, like the content of things, the intent might be exactly the same but
the phrasing can be different. The nuance and that's all like personal either preference or perspective. That's so personal. That's the reason why for the e-mail generation, we wanted to give them the option to use different terms of voice. Yeah, I like. That I imagine at some point we're going to be able to give them the option to upload their style of writing because that would be perfect for it. But what we're trying to focus on is not the things that make their job fun.
And I have the same as a developer. I don't want AI to do the bits that make my job fun. I want AI to do the bits I don't like doing. So I will use it to scaffold. I will use it to, I used it at one point to transfer a load of things from JavaScript to C#. And it's great because you can give it a nice prompt, it would do that work and it saves you 2 weeks worth of boring typing. And that's not interesting to me. Fighting the new functionality that is interesting to me.
And I wanted to do the same thing for our customers. I want to make sure that we are not taking away the bits that make their job enjoyable. I want to take a bathe the bits of their job that detracts from what they're supposed to be doing. That is incredibly smart.
Like if there's a good use case where there's a lot of time spent on something that people really don't enjoy, and they want to actually spend their time on the tiny bit that their time is now going towards that they really enjoy, then that's the perfect use case. Yeah. And we're trying to do the next step of what we're doing at the moment is taking that also to their clients.
So it's when you buy a house, one of the most annoying things I found was uploading all of my documents because it's not enjoyable. You've got to find everything. You've got to upload everything. Make sure you're uploading the right file for what you're trying to do. And it, it is cognitively a high load, but it's also very boring,
which is not a good combination. So what we want to do is allow people to just see that they have to upload 6-7 documents, however many it is. And all they have to do is just grab the right documents and just upload them. They don't need to say what these documents are, they just need to upload them. And then we can recognise what documents they are. Then we can validate them.
Because once you know what type of a document it is, you don't need AI for the validation them because you can just get the values. And that is a prescribed process. Now you don't need any generative process in there. In fact you don't want any generative process in there that is an explicit, if it is a different number to this, then you know it's not right. But the idea that we have there is to reduce the lead time in a
mortgage. So I know that for our documents, that was an extra week added to the process because the advisor has to send you the request, you have to upload everything that then has to get validated. If there's any more documents needed, then you have to go through the same process again. And what we want to do is allow customers to just upload their documents, have them instantly
validated. So they're going to be told straight away this document you've uploaded is invalid, Explain why it's invalid and then let them upload the correct 1 so that instant feedback saves one or two days. And if we can make it easier for the client to upload it, do the pre validation so that the advisor, they have to be in control and they have to approve it because again, financial data, you don't want unknown things being approved.
But it goes from them having to go through the document, extract the data themselves to a this is a document, this is the information in it. Do you agree? So they can go from an hour's process to 5 minutes. And again, as far as I'm aware, that's not the bit that makes their job fun, checking documents and copying over numbers. They are more into the how can I get the right mortgage for this person? How can I help this person achieve their financial dreams?
And yeah, so get rid of the boring stuff and let them focus more on that. Exactly, Something you mentioned kind of stuck with me with the regards to, OK, we have a document, we need to retrieve the values of this. A lot of people go rag, at least that's what I've seen. Right. Take all the contents of something, give it into an LLM model and then generate the output. But you mentioned specifically you don't want to generate, which, yeah, in essence I agree
with, right? It's black and white. Either it's in there or it's not. So you can also retrieve it. Yeah, I would say if the. This is one of the things that I'm going to say. I'm not sure I'm allowed to say what we do. OK, that's OK. Yeah. But I was wondering though why people are going the generative route. Like I understand it, if documents are different and if they're all images then it becomes more complex, but if indeed the format is the same,
that's a solved problem. Then the issue is formatting is not always the same gotcha that that I can say it's. There are very few documents that have fixed formatting in the financial world, so ID is an awesome one. More than that, there are models on the market that you can put ID into and it specifically looks for ID information, so that is perfect. But for something like a salary slip, I have not worked for a single company that has a salary slip in the same format. That is a big issue.
It always has the same information, but it doesn't have it in the same format. So that that is what makes that bit quite tough and difficult. And the same for the one that we had. That was really interesting because it really showed the the edge cases that make or break something employers intent which you need to buy a house. Your employer has to say, yes, we're going to carry on
employing this person. When asked about that, financial advisors came back to us. Our internal ones said, well, there is a form that is used nearly always. So that one, we know exactly what it's going to look like. I've never used that form. Well, yeah, it's, it's 90 to 95% used. It's like that. It's like you've just said that form is not valid for all of them. So unless you're going to say we only support this form, it's not valid to say that this is what
it has to look like. So you you have to catch all of those edge cases, even when it looks like it's something that's really simple. I mean, that's in the end, that's programming. Like that's that's engineering. It's like these are hypotheticals, but I have to cover them. Yeah. Like that's the thing here. Yeah. I want to get into more of a day-to-day.
What's in your tool belt nowadays, because you mentioned Claude, you mentioned copilot, what do you use to make yourself more productive or what is being used generally within your team? Within the team, we have a choice out of copilot or Clod so people can use the one that makes them what they're most comfortable with and what helps them the most, or they can use both. You don't have to be limited to one or another. copilot is better at some things, Clod is
better at other things. We had a training morning or training day where we had somebody come in, help us how to prompt, and then we had an exercise in the afternoon of pick something that you need to do and try and just do it with AI to try and get people to really see the value that it can bring. This is also a really tough process because people have their ways of working. As much as humans are all about change, humans hate change. Hate. Yeah, yeah, I fully agree.
And so it was a little bit of just trying to show people what was possible. And we have a nice uptake, I think not as good as it could be, but we have a nice uptake and people are starting to learn how to use it and starting to learn that you can use it for more things than you originally think. Yeah, so except I will use it for when I got started in Python, I did the absolute vibe coding of I need to do this. Yeah, give me and it gave me a
Python script. It's like, OK, so that's Python And I ran it and it failed and it's like, OK, failed with this error. And I got it to the point where it worked. And then I looked at the structure and tried to learn a little bit of Python whilst I was doing it. And from that point on, it was kind of like I would ask it about AP is as much as anything else. It got it right 50% of the time and it saved me time because it got it right 50% of the time.
And the other time it got me close enough that I could track down the issue myself. So that's one way. Obviously the coding weighs an obvious one, but we also use it for brainstorming. Something that I love about using generative AI is helping you think through a problem. So rather than going and asking it explicitly for something, I tell it to ask me. So I have this problem, I'm trying to figure out how to solve it.
Interview me and ask me questions and tell me or help me figure out what I need to do. At the end of the day, the AI doesn't tell me anything, it just helps me think a problem through. So it's a rubber duck. That one I have found super useful. I helped it. I had to go to the doctors for one of my medical issues and I needed to get all of my symptoms down on paper because of the way my brain works. That was really hard because I always forget something.
So I got it to interview me and it never put any symptoms in itself, but in asking me questions I could tell it scenarios, I could tell it things that were happening. And at the end of the day I had a three page Word document that I could take to my doctor with all of my symptoms organized by criteria. Sodium, this is neuro, this is gastro, this is musculoskeletal. And the doctor is like, I apologise to the doctor saying I hope you don't mind me bringing this. So this is awesome.
Can I have it in electronic form so I could send in the Word document? And it was so much easier for him to then gather my symptoms. So using it for that type of thing was a wonderful experience. I also used it to buy my new car. Oh, really? Yes. I was like I I couldn't figure out which one to get so I told it the two options and then told it to interview me about what I
want from a car. Fun thing is I brought the opposite car to the one that it told me to buy, but because I went through the process of doing it, rather than it just going round and round in my head, I could actually lay it down piece by piece and it helped me realise which one I wanted. Interesting. So there's that use of AI that I think people haven't quite got around yet that can really improve your not only improve your productivity, but also allow you to think things
through more deeply. That's what I'm thinking as well, yeah. Like it's a, it's a way for you to bounce ideas off of something and it's, it's very strange because it could just be you on the other side. In essence, because there is nothing, it just generates kind of statistically whatever sentence structure is the outcome here. And it loves generating. So you also have to limit it. But if you use it for self reflection, I, I don't see how that could be anything other
than being beneficial. And especially in something in engineering, you just need to kind of get out of your own head to a certain degree. So to then talk through a problem with something as kind of a mirror is, has always been helpful. That's why we have the concept of Robert Ducky in the 1st place. So then using this for that I think is genius. Yeah, it is a really nice use case.
I think it's a slight tangent, but one of the things that you said there reminded me of something that is so important to remember about AI, the fact that it's predicting statistically. Yeah, something that a lot of people really don't understand about AI. Even if the model is called a reasoning model, it is still a prediction engine. It will do cycles of prediction which allow it to give a more accurate answer, but it is still a prediction engine. This thing is not thinking, it
is not reasoning. It is predicting text, taking that as extra input and using that with your prompt in order to try and drill down and get to the correct answer. If you ever use Claude, turn on reflection and you can see the air quote thought processes that Claude goes through to get to your answer. And you can also see how it switches as it produces results. And that triggers it to think, OK, this is what they're saying. And it allows you to have much more fluent conversations with it.
But it's fun watching that. OK, you predict one thing and then it goes back to like, Nope, that wasn't the right thing to do. Yeah, exactly. Yeah. I mean, in essence, like I, I understand that in the end it's just predictions, stuff like the OK, I'm going to toggle and I can see reasoning that doesn't help because in the end it's still statistics.
But the argument is always, OK, we're going to do statistics so well to a point where it's going to equal reasoning, but in the end, it's still a prediction model. Yeah, yeah. I I think that the thing that humans still have as a huge advantage is our ability to think laterally. The fact that you can just go from one topic to something completely different in a way that works in the human mind that can really help you move forward with a problem is still something that I don't know if
they will get to the point. Yeah, I think we're a lot further along than I thought we would get by this point. We're going fast, very fast. When I was at university in the 90s, I did a semester on AI. As part of my course. We had an AI professor. He was involved in so many projects and showed us the things that it was being used
for in awesome use cases. So whether or not that was facial recognition for security systems or I believe it was Germany, you might need to Fact Check this, but I think it was Germany for their breast cancer screening in the 90s. They didn't like using mammograms because you had to use X-rays and X-rays are bad, so they used ultrasounds. Ultrasounds are much harder to read than X-rays.
So they trained a model to read the ultrasounds and then that gave extra input to the doctors and false positives were filtered out by doctors. But they found cancers in a way earlier stage than they would have done. And that was back in the 90s. So our professor was absolutely not anti AI. He he was full on with this. But he said something that has stuck with me since the is it AGI? Yeah, is probably never going to
happen. And the example that he gave to us of why you cannot make a thinking engine is because we have qualia in our thoughts. I never heard the word before that time and I've never heard it since. But it is things which are unexplainable. So if I look at the bottle cap that's in the middle of the table, it's blue. You also know that it's blue, but you don't know that I'm seeing the same thing as you. I I've had this conversation also because I'm partially
colour blind. Like it's, it's very strange for me to be like, what colours do you see? Yeah, yeah, I can appreciate that one and think, what does a strawberry taste like? Yeah, describe the taste of a strawberry using something that somebody else can have an absolute concrete grounding on without having to reference other things which are only experienced inside of your head. And to make a fully autonomous AI, you have to solve this
problem. And he didn't see it being done in our lifetimes of our children's lifetimes. He didn't say it's impossible. He just didn't see it happening then. And if I look at the engines now, just being prediction based, no matter how good they get, you're still not going to get to that essence behind it. Yeah. It's it's not just having the knowledge of the world, it's also, yeah, the experiences as in whatever do you see and is it equal to what else someone else
sees or with taste as well. And then touch. Yeah, it's like all senses. I I never thought about it like. That it's a very strange thing that's I'm a phantic, which means I don't see images in my head. OK, if I the the whole Apple test that went around the Internet a couple of years ago, close your eyes, I see black. I don't see anything. I thought that was normal.
So, you know, imagine you're on a beach and I would have a rolling commentary in my head, you know, it's like you're on a beach, blue sky, maybe some clouds. I didn't realise that people would actually see this and I don't, not at all. I can describe it, I can't see it, and I think that itself shows the fundamental lack of understanding that we have of how a brain and a mind works, that getting to that point of an actual autonomous AI is still a pipe dream for us. It's a huge leap.
I I can see it getting better, I can see it getting more accurate, but I don't see it making that jump. To finally then circle back because we touched on what you've been building, what you've put out in production, and also what has made you more productive.
Do you think with the introduction of AI and kind of the tool suite that is available to software engineers now, is it going to make software engineers better at what they're doing or is it more harmful with regards to kind of their thought process intellectually and also their output? I see people being more productive, but I also have definitely been humbled by how lazy I've become for certain problems. It's like, yeah, I just want it and hear the error and just give you the outcome.
Yeah, I think it it all depends on the developer as well. I saw a talk given month, month and a half ago. If you give junior developers AI, they don't improve, they get worse. If you give media developers AI, they can stay pretty much as they are.
If you give seniors AI then you can start to get the benefits because they can see the good and the bad of the output and work towards that and use it for the things that are going to save them time rather than chasing their own tail trying to get a prompt to produce something that they could have coded in less time. I had the same issue. Sometimes I kind of feel like, OK, you could have just done that. I was like, yeah, but I did this in like 5 minutes and I didn't want to type it.
Yeah, exactly. Yeah, I got the result I wanted and I didn't do a thing. Yeah, so it's. I see it for that, but I think it can also help for the senior developers on checking things as well. I've done that both for code and for non code. You can have your code, you can have your ideas and you can ask the AI then for improvements and then you can decide which improvements you want, which improvements you don't want. You can also ask it to describe your code and I found that far
better. Rather than just saying, what improvements do you see with this code? Just giving it code and saying what does this do? And if it comes back and says what you want it to do, then you're probably on a reasonable path. And if it comes back with something else, then you probably know that that's not going to be readable in six months. I think it helps with maintainability. That's interesting. I, I also use this for my for my talks. I used it in writing the book as well.
The book is on me. I did not use AI to write the book, but every chapter that I finished I gave to Claude just with a simple prompt to tell me what this is and if it told me what I wanted to convey in that chapter. Then I would do a follow up question of what is the target audience and if it came back with this is a beginner's book and taking people through XYZ. These bits are really good for that. Then again, I know OK, that
chapter's done. Sometimes it would come back, like that's not really what I meant, or that's too advanced and I would have to rewrite that bit to make it better in the future. That having that as a first editor was just phenomenal because it just meant that I could pass it on to my publisher and to their editors and the technical editors with a lot more confidence. Yeah, that I was doing the right thing.
Yeah, exactly. I mean, already when you're writing something to an audience for you're preparing a presentation, you always do a self validation, right? But that's your perspective. Absolutely. And then I love how this is in a safe space. You can kind of give it to something that is AI and then be like, OK, yeah, I could do another pass myself. Yeah, I could do another pass myself. And I did a blog post once and I rewrote it from scratch for the
feedback that I got. Yeah, it's, I thought it was kind of OK, but it threw up so many, not technical errors, but contextual errors. So it was like, readers are going to struggle getting through this. Oh, wow. I was like, OK, maybe you've got a point because yeah, I've read it, but I just wrote it, so of course I know what it means. Of course. So having that feedback allow me to write something much better.
Yeah. I wonder if, because then probably you haven't done this, if you could then AB test in a in a perfect world and say, OK, this is the 1st result. And then this is what whatever the changes were after AI validation, let's say. Because from your perspective, probably the feedback that it gives, it seems plausible, right? But that's also the job. It's to give you a plausible feedback and be like here's
something. I think that's why I try not to tell it. I try not to ask it to validate what I'm saying. I try and keep it as vague as possible as I like. Not yet. I like that. Is this a good chapter? I was like, no, it's like what is this chapter? Who is it for? The only time that I would try and get it to do self validation is when it's find some things. It's like, I kind of agree with it, but I don't think it's that important. And at that point I will do the
95%. Unpublished is better than never published for 100%. Yeah. And then it's OK if that's what you're after. Sometimes it's like, this is fine. It's like, yeah, that's nitpicking. It's fine. And other times I say these bits are fine. This bit's really important, Yeah. And then it's like, OK, so I know to look at that bit again. Gotcha. It's I found that so useful also because I know I'm a perfectionist. So actually asking it, OK, I'm a
perfectionist. Is this acceptable without actually saying that question was was also a huge boon because I know that I spend way too long writing my chapters at the best of times and they're said like I could improve that. I know I can improve that. And it's like, no, it's if it says 95 and all the examples work then I know I'm there. Yeah, I, I love those insights because my mind usually goes to, OK, I'm trying to confirm something or I'm trying to deny
something. I ask it in in that way. But then it's such a, a yes, Patrick's so on and so forth. It always says yes. Or if I'm saying that, well, what about this then of course, you're right. And it gives me X wines that the, the vagueness or at least the open endedness of your questions in kind of validating. I think that's super important. Yeah. Yeah. Awesome. I've really enjoyed this conversation. Stacey, thank you so much for coming on.
It's been 3 1/2 years, I think, something like that, Something like that, yeah. Thanks for having me again, this has been fun. Awesome. Then I'm going to round it off here. If you're still here, let us know in the comments section what you thought of this episode. It's the best ways to support the show and we'll see you in the next one.