AI Startup CEO Reveals What Really Kills AI Projects

Hi everyone, My name is Patrick Akio and joining me today is CEO and Co founder of Deploy Marta Stock. We discussed a few things, some of which you might think are controversial. First one, bottom up initiatives are meaningless without a strong vision, A dot on the horizon top down from leadership. Another one, the AI Act that the European Union has put into place is actually beneficial to

all parties involved. And lastly, Marta shares how he's building this AI hub here in the Netherlands, in Amsterdam and what the benefits are going to be. So enjoy what made you start this AI Hub or AI community in the first place. I didn't really, I wouldn't say start. I think those things start to

exist. So not giving myself all the credits or so, but if you talk especially about the physical location, I promised them as I started together with my car furnace to other companies and I promised to myself and also to my car furnace. I'm not going to spend a lot of time on on this. If there's no demand for anything, if it is a hub or something else and there's no

proper demand. If nobody really wants this, I'm not going to spend any time on this because I have plenty of other things to do. And that's already hard enough. But then we found that more and more people recognise that in the nodes, we don't achieve a lot when it comes to AI at the moment or data or infrastructure in general. So for me, that was the reason to, to, to start talking with a lot of founders, like what is the, what are the challenges

you're facing at the moment? And in January, we decided to, to host a dinner to get up with a few other founders who were happy to spend some time on this. We found Techie Peer to also be part of it and that was clear that everyone is in the end doing a bit their own thing in Eint, over in Rotterdam, in Amsterdam, in Delft, in here in, in so many different places and others.

Everyone is kind of doing their own thing and that doesn't really work for AI because AI successful AI companies are scars I think. And you really need the top 0.001% of people that are both really smart and passionate about something related to AI and entrepreneurial enough.

And if that's Canada across a pretty small country, then you won't really find each other. And I strongly believe that if you, if you create one place where we are all together, where we can help each other, where we can, where we can help each other finding investment, where we can maybe invest in each other a bit, where successful start-ups will see spin outs out of the company. They create that flywheel and with that flywheel, we'll create bigger and bigger companies.

Without that flywheel, everyone keeps on doing their own small things. And yeah, that's not going to work. So the, the, the, the polar model in that sense is and, and, and and and, and, and, and trying to do this in a fair and, and, and, and distributed way. I think that's not going to work for AI that we keep on thinking too small. We keep on acting too small. So for me, that was the main reason. And then a lot of people got quite unjust about it.

And that's good to see. So that that that that motivated me to spend more time on this to, to, to together with those other founders to yeah, start searching for location, to negotiations, find out that the real estate world is a completely different world than the AI world. I'm not a big fan of it. Yeah.

So I'm not sure if I want to do it again, but but that was the starting point for us. And what helped a lot for me is that with one of the companies we founded, Engines, we're based in both in Berlin and we are based in the AI campus. So we actually see how it works to be with a lot of AI companies in the same space to do meet ups together to basically help each other out, become each other's customers. We actually see it working there.

So for us, that was like a great blueprint what we can achieve here and on us as well, I think. Yeah, I like that a lot. I mean, I like to start of how your thinking works in the 1st place. If there's no demand then why spend any time on it at all? Like what's the point? What's the purpose? I learned that the hard.

Way and in general, I think one of the things that I love being in tech as a software engineer is people share a lot with regards to knowledge more so on the implementation side more than anything else. Look, I there was this problem. This is the context this is how we solve it.

They do that for free articles, blogs, podcasts, videos, conferences, meetups, everything like that and this is kind of in a same vein, but then for AI startups with regards to challenges, funding, what problems are we solving? Learning from each other, being each other's customers sounds

really cool man. I think we need it. I think we, we have seen it before that it actually works not even on purpose. Like we didn't create per SE a fintech ecosystem in them. So there was not, there was not one person who said let's create an ecosystem here. But it happens. It's whenever there are a few successful companies, you've got spinners, you've got, you got more money in the ecosystem as well. I think that's what we, that's what we lack in and out now.

But if you talk about cloud infrastructure, data and now AI as well, we, we kind of missed that. I can't name, I can't remember a lot of companies at least that started here that have a lot of spin outs in that, in that vertical and we see now the the results of that I think. Got you. And you said NL is behind with regards to that flywheel and that ecosystem.

Are there any countries that you say, OK, those are actually quite far ahead, those are companies or countries to look up to with the rest of their ecosystems? Yeah, I think if still work ecosystems is more cities or yeah, basically cities or the the, the, the large area around it. Of course, Paris and Berlin are the most obvious one I think in Europe, maybe next to London. You do see other successful companies in other smaller cities sometimes. So it doesn't have to be a

physical ecosystem I guess. But especially Paris is fascinating I think. And it's good to realise that it didn't start like five years ago, it started 25 years ago. Szechuan F was funded in the end by the founder of Scaleway and Scaleway exist for like 25 years, I believe. So you see that flywheel already started 2025 years ago with investments and successful companies in cloud infrastructure didn't start yesterday.

And also don't believe like if we, if we're going to shout really loud that we should, should, should push now that doesn't work. There's a lot of groundwork. There's a lot of, it takes decades of them to really become very good in, in a certain industry as a country that that

doesn't come overnight. And that's why I, I strongly believe in that the ecosystem, if we, if we get things together, if we start building more successful companies, that will lead to more successful companies that will lead to another wave. So you need a flywheel to create new waves and become better in a certain industry, in this case

AII. Feel like you need a very strong vision on a long term horizon to kind of realise what you need to do now and plant seeds with people, talk to people and really start creating this. Is that usually how you think? Like more in the bigger picture, long term vision. I'm also very good to to worry about things yesterday. The day-to-day, yeah. Definitely. And I do think you do need to realize that it takes long. I don't think it needs to be one person with a vision or so.

I think especially these kind of things can be organically multiple people working towards something fakely in like 5 or 10 years, and then new people will come and they work for another 10 years or something in the same direction. I think in that way you create a strong ecosystem. Yeah, yeah, you get that. What's your take on kind of how Netherlands and even EU is handling AI? Let's start with the A i.e. U Act. EUAI Act. It's a difficult 1, isn't it?

Yeah, I'm probably one of the fewer engineers who is actually quite a favour of the AI Act. I've seen so many organisations

struggle to get beyond POCMPP, innovation, PowerPoint, blah, blah. And the reason for that often is that someone in the organization at some point is going to say, yeah, that looks great, but you're not going to get get this direction. And it's often because something is lacking, which is actually written down the AI act, it's either oversight, monitoring, it's, it's, it's, it's either way too risky to get things beyond POC.

And I think what we, what we define now in the AIA is quite a good set of rules and standards which are going to lead to, to, to proper safeguards and help us to get AI beyond POC, to get AI to production level software and systems. So that's why I'm actually a big fan of it. It creates standards which help us to get AI in especially more regulated and high risk environments, production and often the high risk service, also the high impact stuff.

So that's the reason I'm, I'm quite a big fan of it that I've seen that having the proper governance and the proper controls in place helps to get things beyond POC into production. Yeah, help me deepen this out for the people that don't know, because the the EUAI Act categorises different types of AI in different fields of industry, right? If it's education is different, if it's risk and financial management, it's different.

If it's consumer facing, spam or anything else, it's mainly the lower risk applications and then you have different classifications with regards to what you need to adhere to. True. Mary, first of all, disclaimer, I'm not a lawyer. Yeah, legal specialist. So we actually give training together with, with, with our partners and often they are more the legal specialist and more the, the, the technical person in the room.

And we often joke that we always avoid doing anything legal because we hate it. And the other way around. They understand shit about engineering. But to summarise it in, in a way, in a way which is probably also more interesting for engineers, You do indeed have, have multiple categories. It's prohibited. Don't think about it. You're not going to do it anyway. I've hardly seen application which you fall in that category anyway.

Then you have high risk and that actually entails quite a few things. And to summarise, it is everything that has an high impact on human rights, on basically the rights you have and, and I have, which are also a strong part of EU regulation in general. And there, there are quite some articles which dictate what you have to do. Basically the standards you have to take into account. So you have to explain for example, where your data is coming from.

You have to, you have to explain and you have to document how you think about bias or ethical considerations. You have to have proper controls in place. That's called human oversight in this case, or effective human oversight. So there are a few things which you need to do if there's a high risk model or high risk system in general. And I believe that those things you should be willing to do anyway.

But it helps now to have a standard of like, OK, these eight points with like sub bullets are the things you have to do. And there's often software for

that. There's a reason we started deploy is to basically make sure that you have all the tools in place to adhere to the things you need to do for anything that has some risk with it. And you should do that anyway because if you don't do it and there wouldn't be an AI act, you would still probably be in the news if you if you basically fuck up with with a certain system. So I think it really helps to create a standard and I don't really think it's going to block innovation.

Is that the main counter argument why people don't like

it? That it's kind of putting the brakes on innovation. Because I agree with what you say, a framework can be a good thing, right? And now this is a legal framework, so it's kind of mandatory and you have to comply to it, but it's still a framework. And if those are a means to an end to create a good system, then it could be a good framework as well. But that's subjective.

Yeah, the the only other thing that's going to happen if you don't have an AI act is every single member state is going to bring up an AI act kind of version. You see the same now in the US If you want to bring something to the US market, you have to adhere probably to the New York State regulation on AI, which is more about bias and ethical considerations of fairness. You have to do, you have to act. You have to comply to the one in Illinois.

You have to comply to the one in California, which actually looks quite similar to the AI. So it's just a patchwork of different regulations, often also conflicting on some points. It's way harder. And the same is going to happen in Europe. Like France probably going to protect tech companies so they will be quite, quite, quite relaxed about a lot of things. But Italy doesn't. So Italy's going to be super strict on things. It's going to be a mess in the end.

And then we're all going to complain like, yeah, it's such a mess in Europe. We need to have harmonise things and it's all over again, the same thing. So I'm really happy we actually have some clarity. We have standards. It's going to help actually also because you will do those things anyway. You know, the opposite is exactly what I explained. It's it's, it's basically what's happening in the US.

Yeah. And I wonder why people are not a fan of it, right, If it's otherwise, kind of, we don't have many other options. And each country is going to do their own thing. There will be something because I feel like people need to be protected towards the applications that they use, they consume, especially when they're more on the human rights side. With regards to the impact and yeah, mainly impact, I guess. So there will be something. So this might be the least of the evil I guess.

Yeah. And I think, I think which, so I do agree that we see large differences between the US and Europe, but I think it doesn't come from regulation per SE. I think it has to do with the way we we deal with regulation. So we can use it in the next use. We can always use regulation as an excuse, like I'm not sure if that's allowed by GDPRARAX, whatever you can think of. Or you can try to comply as good as possible in a practical way and just go ahead with innovation.

And I have the feeling the latter happens in other parts of the world and it first happens here. So if you can just use legislation as an as an excuse, yeah, we don't have to do anything. So it's easier. We have to work less, we have to spend less hours on the office. But that's more cultural thing than anything related to regulation. So if we wouldn't have the AI, we would find another reason or those people would find another reason not to innovate.

So we have to fix the cultural path, which is the the hardest part by the way. But I think that's the thing we need to fix if we want to innovate quicker. But regulation itself, I think it's not the bottleneck. You also mentioned that this might be a means to an end to help go from proof of concept to something that's actually production ready that people can use.

I've been in my previous assignment at a bank where I've seen many proof of concepts, but then they didn't comply to let's say the standards of the environment with regards to

technology and what you can or cannot use. It was just proof of value. All right, we see this value now. Can we do it with the tool set that we have? And then if it's a no, then it's kind of yeah, there's stakeholder expectations there. There's an investment. There's no return on investment because the technology is not compliant. So it's very difficult to then go from proof of concept to something that is running an application in in production.

I don't know if many other or at least from the organizations that that you've been in and out of are struggling with something similar when it comes to actually we see proof of value and then we're trying to go to production, but technology yes or no or we're behind or there's a lot of risk and low risk appetite. Yeah. I, I see those things a lot of course that that that I think it has to do with bottom up is this top down approach.

A lot of organizations are hiring a few smart people also very young, and they kind of have to figure out what they have to do with AI or data products. Just build something, be innovative, experiment, whatever. And I never saw that really working. So if there's no pretty clear picture where you want to go to a top down in an organization,

it doesn't work. It it just ends up in MPPS and PO CS and proof of failure or whatever other names you want to give it. But it's not going to prediction because it's at the end of the day, it probably doesn't really fit in whatever, whatever strategic direction you're deciding to go to. So it's really also the responsibility of board level or whatever you, whatever is leading organization, it's their responsibility to really think

about, OK, this is where we go to in like one or two or three or five years. These are the products we need to get there. This is the reason we are doing it. And that can drill down to the rest of the organization. They can figure out how to build it, but in the end I think you need to really have a good design where you want to be as an organization in like 3 or five years. Same as building houses or building suburbs. You also don't start bottom up,

you start top down. You need to have a proper plan of a suburb, then a proper plan of how you build a house, and then you can actually start building bottom up. But otherwise it doesn't work. Yeah, maybe in Belgium, but in most countries it doesn't work. Especially in some industries, leadership teams have always struggled with the role of tech product, digital products. And now you have AI there as

well. And you mentioned that this vision, this kind of dot on the horizon, where do we want to be as an organization? How do we leverage technology that's now out there? New is now also their responsibility. It's quite challenging to I, from my perspective, kind of see that take shape in some of the organizations or industries that I've been in. I do see people experiment bottom up.

But then in the end, like you mentioned, the effect of that or the outcome of that might be that it doesn't reach production at all. But I do like the

experimentation mindset, just within boundaries I guess. Absolutely. So my advice to to more junior people in the organization, if you feel there's no direction, stop developing and start, stop developing and start asking those questions because it's never going to work. It's going to be very, very energy draining. I think if you work on something for six months and it's going to be thrown away and the other way around.

If you're really on top of the organization, if you're on the board, if you're a director or whatever. And then you have the feeling that for some reason you don't really understand what you're working on or you don't really understand anything about digital products, data, AI. Either find a person and give them a board seat or Start learning how it works and start gaining knowledge about it. But doing neither and hope that bottom up stuff is going to

happen. I've, I haven't seen it working for the past 1520 years. Is not going to work in the upcoming years either if you don't have a vision. Yeah, and I see that also if you talk about other geographies, I see younger people actually leading companies than in most parts of Europe. We tend to have pretty old people in parties, especially outside of the Netherlands I think not doing that bad, but I've seen statistics of that and the average age is just pretty old.

Also here of people running companies. And that's a bad thing, you think? Yeah. Yeah, I think you need a few younger people sometimes as well on the top of an organization. Yeah, I haven't figured out how to get there yet, except like this. Oh, you start something from scratch with the years, and then you're also old by the time you have something of scale and you get there. There are really, really big companies in the US where the CFO is like 35 years old. Yeah.

I think that's going to be really. Helpful. Yeah, very good there. What do you think of when people say culture eats strategy for breakfast? Because I've been in a lot of environments where people really kind of hamper on their culture to a point where indeed they might not even have a strategy. And then sometimes it just feels like anarchy. I don't know. I think it goes hand in hand. If you write something down when it's not a lift, it doesn't

work. Yeah, If you never write something down and it's all captured, your culture, it also doesn't doesn't scale. At least there's a lot of unclarity. We at deploy, we have, we have a hybrid model, so people come to the office and that helps in culture and also work at home. And so we need to write things down. And I have a lot of respect for companies who are remote only because then you have to write things down. Culture's pretty hard.

In the end, you can, you can work around if it's really hard. So you have to write things down. And that strategy I think is super, super important. Vision, mission, strategy, goals, targets, etcetera. And if you're always in the office, I think the culture is really important. I see that here when I walk around and obviously culture is super important and, and both

ways can work. But I think if you if you have some kind of hybrid model, it also goes hand in hand to have both strategy and culture pretty much aligned in that sense. Yeah, Yeah, you mentioned that bottom up initiatives and bottom up experimentation, it, it just doesn't go anywhere without a strong vision, right. And then, at least from my perspective, you can experiment within those boundaries and see if the outcomes are a means to an end.

In the end, if they help the business, the company, the organization and the group of people, because organisations are still just groups of people, reach whatever their vision is,

or at least take a right step towards that. That's super easy I think I just. I just don't understand why it's. It sounds complex to me. You define where you want to be 5 years from now. That should be doable for most board members. I guess you drill down what initiatives need to be taken and you see where AI or data or digital products fit in. But how do you take people along with you, right? If there's a vision where they don't see themselves, you lose them.

That's true. So that's easier. So it's smaller companies, of course. I think, I think that the more you go to done, the more, the more you can involve people. I think in any way you can involve them. But in the end, I think clarity needs to come from a few people. In the end, if you don't provide a clarity, everyone is swimming a bit in the end. Yeah, yeah, yeah. Which is super frustrating.

The thing that I've seen the most, and this is indeed coming from board level and organisations, is we see companies doing stuff with AI. We need to do something like this decent of urgency that just trickles down and there's like, we need to act something with AI on the product sense, embed Gen. AI into whatever we have, but they don't even use Gen. AI. It's just AI. Everything, everything gets under this umbrella of AI. Doesn't matter if it's predictive AI or generative AI,

That's one. And then there's the second one, which is more for the people that already have a lot of software engineering capabilities. We need to be more efficient with regards to how we use AI so we can improve our output regardless of whatever outcomes we're trying to achieve. We need to optimize the efficiencies and stuff like that.

Those are the two. And then especially if there's no good framework, if there's no good story, even the people that communicate that are lacking in communication skills or even sometimes just charisma, they lose the buy in. There's been a bottom up kind of challenge and push back on, OK, we don't even have tools available. How do you want us to work with AI or how are we going to be enabled in this, this mismatch and expectations. And then it just sandwiches and middle management.

It's trying to middle manage. I agree, but maybe to turn it around and make it positive. I've seen really, really good examples as well. That's, that's the, that's the fortunate part of working on something like D players that you're early adopters are the ones that actually have a strong vision, that actually know where they want to be in two years from now, that actually are growing really quickly. That understand that efficiency is super important.

The ones that understand that having a few killer AI features, AI products in the offering is going to be super important. So it's fascinating to see them operating actually, they are actually really growing quickly because they they do define where what they need in like one or two years from now. So I would just turn around and look at the ones that grow quickly what they are doing. And often they're quite open to talk about it. They often share quite a lot of

things. But those are actually really interesting to watch, I think. And you don't have to do that only in the Netherlands, but if you look in, for example, the UK and how Fintech and also share tech now is taking over quite a big market share from the incumbents in the market. You can learn a lot from them. And it's super interesting, I think to see how they are operating at the moment.

Can you elaborate on that? You said already it starts with a strong vision of where do you want to be in one or two years, but what are they doing that is exceptional that allows them to excel in that way? It's not even exceptional.

I think it's it's, it's so if you think about it, it's not even, I think it's not that they actually spend time, I think on, on, on defining where they want to be in one or two or three years. And they do quite some sessions, I think together with their colleagues on discussing and arguing whether it's going to be. So if you talk about banking, for example, and your banks, you see that they basically take the whole process. Every company is a is a little factory in the end.

They take the whole process and define and every step of the process they define here and here and here and here. We feel that with relatively small investment, we can actually make things much more efficient or much more effective for basically make a business case in the end that that skills

and that's what they are doing. They basically make a blueprint of how a bank works and then find all the places where they are more efficient of more effective or can create new products and new offerings which are you know, which which they expect is going to be demand for. So that's what they're basically doing and that's often was

lacking in all organisations. So if you're going to do the same thing, I'm 100% sure that's also we have the right people in the organization, but often you need less people than what you think you can actually achieve the same thing. And that's why I why I mentioned fintech and, and in shirt tech in the UK, they're much further than here. They also have already much bigger market share.

And you can, you can learn quite a lot by looking at the presentations, meet up slides, those kind of things. We're going to learn a lot from that, I think. How much can you still gather from this kind of competitive

analysis? Because you mentioned, OK, look at your own organizations, look at your whether it's process journeys or customer journeys or workflows that you have and see where kind of the most time is spent where you can make a solid business case where you can rely on technologies like these to be more effective in what you do, right? That's very much, I feel like looking at your organization in a nutshell and trying to optimize in that way.

What I've seen other organizations do is they see what someone has created in certain output. It might be chatbot is always the kind of go to example. Someone sees a chatbot, they can be like, OK, well we can actually also do that and it's going to plug and play.

But it might not be the best for their organization or for their consumer base or for the branding in the 1st place, but that's what they do. I feel like what you mentioned in looking at your own organization, that's kind of the better way to do it. But how much can you then still learn from your competitors? I think how they how much time they spent on making the blueprint, there's something you can learn from them. The exact blueprints or the

exact use cases. That's something I think you can better define yourself because every bank is slightly different with different kind of customers, different kind of offering, different market positioning. So you have to make your own blueprint in the end. But the fact that you have to make a blueprint that you have to think about it, that's something you can learn from

them. I think if they have time to spend spend hours and days on on on defining where they want to be in a few years from now, I think also other organizations can do that, especially prioritising innovation and the stuff of the future of everything that happened yesterday. It feels like there's no easy answer and you just have to go through the motions and step by step, kind of figure out where the value is for your own context and for your own organization.

People are looking for a quick win though, especially. That doesn't work. That's OK, That doesn't work. So maybe that's that's the thing I'm trying to tell you is just saying that we have to push something very quickly. It's often a very stupid strategy that I hardly ever sort of working. No. What are some of the implementations or integrations that you've seen that are actually greatly valuable?

Is there something you can share in that aspect, some of the AI features that you have seen go live and actually have impact? Yeah, I think it's good to to to distinguish different kind of AI applications. And so you have a few AI applications which are basically in the current process, often predictive AI, not even generative AI, but every process that is super repetitive, fraud detection or claim handling a certain insurance company or you know, whatever.

So many of those repetitive tasks, you see a repetitive task which is done by people at the moment, which is slightly more complex than normal automation. It's often a very good candidate for machine learning and AI. If you talk about creating stuff which is, which is based on facts, that's also of course

very easy to, to, to, to use. Actually at Deplo we are, we are now working with an AUS company which is doing quite a lot of our development work, which is I think super fascinating to see how the learning curve's going. So also with development work, you can even go beyond Copilot, you can actually work with companies that take over quite large chunks of your whole development work because you have the context, you have the code base, you have especially all the documents you can give

your own developers. You can give that also through AI and with all the cartwheels they already have built and have in place. You can, you can actually automate quite a lot of development work, not as an assistant, but actually taking over your development work. So we see that working on our own organization as well. But all the other more repetitive stuff, I would also have a look at it. But all of them in the end require all the guardrails and the governments we talked about.

If you want to go into the direction, invest in all the groundwork and all the the the infrastructure around it to do it in a good way. I think that's it's never a waste of money. It's the same as investing in cybersecurity 15 or 20 years ago. Whenever you want to go fully digital, you do need those, those cart fields basically to to go beyond. Yeah, just some experimentation work. Yeah, you mentioned there's an ex, Did I understand that

correctly? There's a company helping you with the development work that you have on deploy itself as an application. So of course everyone is using Co pilots of rabbits. That's not really taking off all your development. That's just someone piloting. Yeah. You're, you're still operating in the AI system we actually create.

We we are basically doing the design work, defining what we need and then send it off to another company and they do all the work until merch request and then we accept it or a review of the list. Gotcha. But isn't that your core? Isn't it scary to? Kind of scary. Give that, yeah, like or have that in collaboration with someone else, but it's working. It's also interesting because we have to think even better about the design. Yeah, we have to think even

better about the requirements. We have to review it even better than before, but it actually creates better code. Does it a lot of companies invest in how much their development team understands the business, understands the product, right? Because then you can think along, understanding why we do things makes you challenge what we need to do in the 1st place. And then if the requirements don't make sense from your lens, also looking at diversity, then

you can challenge that. But I feel like with this construction, you might miss that. Yet you're still saying kind of quality wise we have seen increased? For some. So I do get your point. There's also the worry we have. So if it is, if it requires a lot of context and expert knowledge, it's harder, then it often doesn't really work yet. Yeah. Although we can, the more we do with them, the more context they have, so the better they will

understand it as well. But if it is, if it is a feature which is pretty well scout upfront and has less of those business context on it is actually quite doable. And because we're not doing the development work itself for that feature, we actually are more critical in the review, we're actually more critical in the design. So it actually creates better features. Interesting. Is this like an agency or a consultancy that does this construction?

I don't know how to classify it. We'll discuss it actually. Is it a supplier? Is it? Outsourcing is it. An AI tool is that freelance is a combination of those things. Yeah, it's pretty cool zooming in into AI features in production. You mentioned a few things with regards to effectiveness and efficiency. What are some of the things

that, if you're talking about a production system, needs to be there with regards to observability? I talked to Saud up earlier and he has a very distinct view on AI and operations, let's say. But I'm wondering what your take is. What are the metrics that I need to think of or what do I need to have in observability to be in control? That's a good question. It depends a bit again on the on the type of application of the

type of technology you're using. If the generative AI or predative AI. I actually listened to the podcast you you met with sub OPS. I'm trying to recall everything now. I think what is so you can measure all the technical aspects, both the the, the, the, the, the more the metrics around traffic, but also the ones which try to capture performance. There's a bit like you, your your critical question like 2 minutes ago. A lot of things are more require more context and simple statistics.

So we are deeper strongly believe that one of the biggest things you need to measure, one of the most important things you need to measure is some kind of feedback loop, either directly

or indirectly. So indirectly could be that you choose the second option or start arguing with an, with an NLM that's, there's also a kind of feedback loop, but having some kind of feedback loop and monitoring the feedback loop and act on it. Whenever you see that things get worse or when in a certain subcategory you see more feedback could, could mean that the, that the system is biased

on some subgroup. I think that's the most important thing you need to measure next to all the statistical stuff and all the the technical stuff. The real thing you need to measure is in the end, human feedback, because that's, that's basically interaction between humans and machines or human and AI systems. And that's often pretty easy in practice to, to measure. It's if it is a, a predictive AI model, it could be feeding back actuals or feeding back expert opinions or overalls.

If it is an, if there's a generative AI system, it's basically all the arguing that this happening between an LM and a human being or over ruling decisions or trying to correct things. Those, those things are all forms of feedback. And if you get feedback, find an API and start measuring that. Then you can act whenever you see that things don't really work in certain applications or some subgroups, or it's getting worse over time, or it's getting

better. The positive side, I think that's the most important one because a lot of things are not captured in statistics. Yeah, I really like that. Like the only Gen. AI feature that I worked with in the end, it was going to make people more effective. So work that they were doing, spending hours on it, we had something that would pre fill a form instead of them doing the work, they would kind of have their own position of reviewing.

And I saw a lot of people talk about, OK, if the answer is correct, then in the end it's correct, right? And we can even have a model to say if what a person changed, if the intent was exactly the same. And for me is the fact that someone already changed it, if it's not the way they like it or the way they want to kind of have that work be put forward because that goes in a whole chain and other people are going to see it. It's not good enough, right?

It's kind of this user acceptance that also needs to take part of that, which I think is interesting. And I think that we still call it explainability, but it's more than just Shepherd Lyme or whatever. But but but explain ability or support for a certain decision or certain output is super important as well.

So if you talk about it and I'm giving the support for certain outcomes, it's super important because the outcome can be, can be good, can be a good recommendation for something, but the reason behind it could be completely wrong. So I think that support is also important to make sure you capture the feedback about the reasoning as well, which is harder for a Netherland because asking it for an explanation is not an explanation of the model itself or the underlying

technology. No, it's could be just as well hallucination. Yeah, yeah, yeah. Reasoning with regards to why an answer is the answer it is I think is incredibly challenging. But if you've if you have a good way and it also makes sense from a user perspective continuously, then I feel like you have a great outcome. I think that's an it's going to be an an ongoing discussion for

the upcoming years. But it's basically the discussion between closed source systems, AI systems and open source models. I'm not sure which one is going to be leading in a few years from now. So you see open source getting closer to close source. We also see that the business value of it is is pretty small at the moment to be more open about which code you use, which

data you use, etcetera. But if we go more towards or if we demand more transparency than it could be that we in the end move more towards using open source models for repetitive tasks or for certain use cases. And with open source models, LLMS, you can to a certain extent explain why, why a certain outcome was the way it was, like which tokens were important, for example, or saliency of, of of of a certain

outcome. And I think that can help to really understand why the lamps are doing some really strange stuff, sometimes better than just asking for an explanation. So if transparency is getting more important, if control and source code is getting more important, if IP is getting more important, then we might move back towards more open source

systems. If we for some reason decide that it's not important, then we probably stay working with closed source systems and it means that we have less transparency. We have to rely on the explanation we get from those providers. Right now like those are I think big ifs and I could see them, I have assumption on the way they are going, but are there any incentives to use open source versus closed source currently? Yeah, we see this quite a bit in, in in health tech.

You don't want data to be shared to those really large tech companies. You do want to have more controls. You do want to really go deep into the the LLM system itself. You probably want to use smaller language models, still pretty big of course, with smaller language models because you have simply more control on what's happening. So there we actually see quite some companies working with their own open source LLMS which they fine tune and run use case because they simply have more

control on what's happening. Interesting. And I think the same will happen for the high risk use cases. Yeah. Is it because I haven't experimented as much with open source variants? I don't know if they need to be self hosted. For me closed source model is just an API Callaway and that's it. And if it's compliant within the organization then we use it. That's like and we have experimented with other open source models, but I haven't had

any hands on experience. Do people usually self hosted or how much expertise is needed to get that up and running do you know? Yeah, I've seen it with some companies that differs a bit I think, but you do, it is of course way more investment in the end to, to make that work. So the ones I, I mentioned are often focusing on one single use case, which which is basically the company in the end as well. So they provide a certain service in a certain step of

certain process. So if you talk about healthcare, they're doing like one care pathway, 2 care pathways and they optimise a model for that. And that on itself is already a big enough business case to run a company on. So if that's your whole company, then it makes sense to to

provide all the card rails. If you want to scale that to like all the care pathways or use it in healthcare in general, then I think it's going to be super, super complex and it's requiring a lot of money, a lot of investment. But the other way around, just relying on a big U.S. company to do this for you is also another way forward, I guess. So yes, it does require quite some investment in in multiple different ways, but it's also the only way forward, I guess in

those in those users. So especially in healthcare, I think you do want to make sure that you understand why system makes certain decisions. And hence you do need to have access to source code. So that's what what we at least see. But they're not honesty. If I touch card within deploy, I think our developers are going to kill me so. For me, like government healthcare, if you're saying like these are the typical use cases, it's quite specialistic work.

But then the people that want to work in those environments, they might not want to work in healthcare governments because it's also like slow organizations, very slow moving people that have been there for 20 years and a part of the walls, let's say. And they don't have the same

experimental or innovation mindset that other startups might have. So it's kind of this chicken and egg problem. I don't know how to get out of that yet. I feel like we do need innovation in specifically those fields, but the way we've been doing it so far, it's just not effective. Yeah. And like these kind of innovations often start with things being under pressure. So in healthcare we will run into a point where it's simply

not affordable anymore. And I think that will, that will make it easier on the end to innovate. That will, that will help letting organization make decision that has to do with risk. So if we don't want to take risk, then we then it's hard to innovate. At some point we need to take the risk because the other, if we don't make that decision, then the decision basically we can't provide healthcare for anyone if not for everyone anymore.

So I think because both government and Healthcare is probably under pressure of the upcoming decades in Europe, we will make decisions which stand towards innovation and will become easier to innovate and will become easier to both companies and that in that industry.

But I completely realise that at the moment it's super, super, super hard to build a health tech company or Med tech company or company which is providing software to the government, not consulting with actual software, you know, super hard. Which is strange, right? Because in your scenario, there's like, OK, there's this building pressure, which means we don't have an option anymore and we have to.

It's like back against the wall. Whereas now I feel like we still have time, yet it's not working as well as it should. But right now would be the best time. The best time for a lot of things, we wait till it's almost too late and then we start innovating. But it, yeah, it has to do with our personal horizon, I think. Do you know what you're going to do in five years from now? No, I hate that question. Whenever you've been talking about these long timeline horizons and I'm like, Oh my

God, like I have no clue. Most people don't, I think, so they're not optimising for the long term.

Do you like, do you have that strong vision for you as a person or even you as a company? I am my my girlfriend at home. She often finds it very annoying that I'm thinking I had for quite some years. Quite some. Because it makes it less romantic I guess. Yeah. So both personally and professional, it's something you do.

Yeah, I also like it. Mary's a bit of dreaming sometimes about about some stuff as well, but I think it can also be really important to at least have some fake idea where you are in a few years from now. And it would indeed be helpful to do that in, you know, in with the government, in healthcare, with hospitals, in the financial system because they're also going to be under pressure. We talked a bit about fintechs initiatives which are definitely going to take market share.

So if I think it's going to be crucial for them as well to put innovation high on the agenda and not on like on Friday afternoon, also for them, it's going to be critical. But probably you'll see that when it's almost too late, people really start to innovate. Yeah. People will move. And that's happened over the past decades or centuries as well, I think, over and over again. I can see that.

Yeah. I've really enjoyed this conversation, Martha. It's fascinating the way you think, the way you have a strong vision and kind of are driving forward this community and in and now and I'm some specifically, I'm going to look forward to it. We'll, we'll keep in touch. Hopefully it's going to be successful. Hopefully it's going to be successful. We'll see you as we go. Thank you so much for coming on and sharing. Thanks a lot this. Was a blast.

Well, round off here. Thank you so much for listening. If you're still here, let us know in the comments section what you thought of this episode and we'll see you in the next one.