Surveying The Landscape Of AI and ML From An Investor's Perspective

Attend the dev and ML talks at Nodes 2023, a free online conference on October 26th featuring some of the brightest minds in tech. Check out the agenda and register today at neo4j.com/nodes. That's ne0, the number 4, j.com/nodes. Your host is Tobias Macey, and today I'm interviewing Matt Tuerck about his work on the mad landscape, ML, AI, and data, and the insights he has gained on the ML ecosystem in the process. So, Matt, can you start by introducing yourself?

Yeah. Absolutely. Thanks for having me. As you know, I'm a big fan of your work. So, my name is Matt Turk. I am a partner at Firstmark, which is an early stage venture firm based in New York. And, over the last, 10 or 15 years, I've been heavily focused on the world of data, data infrastructure, machine learning, and AI. And do you remember how you first got involved in machine learning? Yeah. Prior to Bigovese, I, was a cofounder at some point in my, career.

And I was working on enterprise search and knowledge management, which actually involved a lot of AI, you know, at the time, Bayesian techniques. Although, at the time, saying that you were an AI startup was not quite as cool as it is today. And so for

context, you've actually appeared recently on my other show, the data engineering podcast, talking about the data infrastructure aspect of the mad landscape. So I'm not gonna dig too deep into kind of what is the mad landscape and some of the history behind that. So for folks who are interested, you can I'll I'll add a link back to that episode. But just for the context of this conversation so people don't have to jump around a bunch, can you just give a bit of an overview about what is the Mav landscape project and some of the story behind how it got started and how it has had kind of a a lasting impact over these past several years.

Yeah. It's an annual project I've been working on since 2012 and, you know, it's always 1 of those, big market maps. The purpose of which is to try to make sense of the complexity of the ecosystem, and I try to capture on just 1 graphic, but also now, interactive application, both the worlds of data infrastructure, machine learning infrastructure, but also ML and AI applications. So you can find it on my blog, which is mattturkdot

com with 3 t's. And the interactive application itself is at matt.firstmark.com. Somad.firstmark.com. In terms of the ML components of that overall landscape and the ecosystem around ML and AI, I'm curious what you have seen as the kind of broad categories of changes that have happened over that 11 year span from when you first started compiling

the the landscape to where we are today, where AI has become a topic of conversation for people who aren't even in the industry at all. Yeah. It's it's been a a really interesting evolution because, you know, as we all know, machine learning and AI heavily depends on data. And especially for a lot of enterprise AI applications, there has been a natural progression towards

being able to, deploy AI in the enterprise that involved first getting your data house in in order. You know, that was Hadoop back in the day, but that's really become a reality with the modern data stack and, you know, the rapid success of, cloud data warehouses. So, you know, that that was really the big trend of the last, few years. And now that a bunch of companies, do have their data house in order and, you know, with the

parallel evolution of all the super exciting stuff that has been happening in deep learning, and most recently in generative AI. Now AI in the enterprise becomes a possibility. So we're exactly at that inflection point, which is just extraordinarily exciting. And with AI applications, an interesting aspect of it too is the kind of relative relative appetites for risk and required accuracy because of the fact that we're not dealing with deterministic prop processes, but instead probabilistic.

I'm curious what you have seen as the general awareness of what that means in terms of kind of risk appetite and actual capabilities and useful applications of these technologies, along with that progression

capabilities of those underlying models to be able to ratchet up that level of accuracy for different problem domains. Yeah. Totally. So we we're jumping right into the central question, for AI in the enterprise, which is that, indeed AI is a predictive kind of technology, and I think people don't always

fully understand that. So a lot of the data world has been focused on the core paradigm of business intelligence, which is really understanding the present and the past where you just, you know, use tools that give you information that's gonna be correct a 100% of the time. When you start getting into predictive technology, which is machine learning and AI, then, you find yourself in solutions where

AI and ML may get it right 80% of the time, 90% of the time, maybe 95% of the time, but not a 100% of the time. So that has all sorts of different implications about how you can deploy AI in the enterprise and, you know, what use cases you may wanna apply it to. There was a first wave of enterprise AI that, really started in, you know, 20 14, 15, 16 around use cases, that were more sort of like numbers based and structured data

based. So I'm thinking applications like fraud detection, like churn prediction, that kind of thing. And those are great examples because if you help a company predict churn, if you get it right, 80% of the time, it's still a wonderful thing. But, if you, you know, and and if you get it wrong 20% of the time, then there's no particular issue there. As we, get in the thick of that, generative AI wave,

then this you know, also the new use cases, but I think, the the central question is is wrapping our minds around, okay, what are the use cases when we can live with AI that is right 80% of the time or 90% of the time. And, you know, we can get into what that means. But that's the sort of the the framework. 1 of the kind of interesting challenges of machine learning and AI in the kind of business and kind of company forming space is that and and in the enterprises you were mentioning

is that it's not a deterministic approach to problem solving. It's probabilistic, and there are variances in terms of the level of accuracy that you're able to get, and there have been obviously progressions in terms of the accuracy for different problem domains, sometimes quite drastic.

And I'm wondering what you have seen as the impact of the kind of general education and awareness of what those impacts are and the kind of true capabilities and applications for m l a I in terms of the appetite for risk and, accuracy requirements in different companies and different industries. I'm glad you mentioned that because,

it's actually a very central problem. That that's not gonna be new news, to, you know, your audience or folks who spend a lot of time thinking about mission running in AI. But as you think about the sort of proliferation of, AI in the enterprise in particular, what what you just stated is, is is a surprisingly misunderstood fact. And I think part of it is that, people have understood

data through the lens of business intelligence. And as we all know, business intelligence is really, ability to look at the past and the present, with 100% of certainty. Like, okay. My sales in the northeast increased by x percent through the performance of those 2 reps. But, when it comes to AI, I think

it it it really does take some education for people to understand that it's a predictive technology and that it's gonna be correct 80% of the time or 90% of the time or 95% but certainly not a 100%. And 100%. And and therefore, it should be applied to use cases where being right, just most of the time but not 100% of the time is acceptable, which, rolls out all sorts of different,

use cases. And then, even for the use cases that you don't apply to, like, accepting that it's not gonna be right, all the time is is important. And I think that's, fundamental

aspect is, getting a new life because now we're clearly in that ascending part of the hype cycle around generating AI where, you know, with AI going truly mainstream. Now everybody seems to be thinking, that AI can do all the things and, you know, ChatJupi in particular being this kind of, you know, all around, kind of AI that can do multiple multiple things in a highly convincing way. Now, you know, clearly, the conclusion that a lot of people,

get to is that, okay, AI is going to be able to do all the things, for all problems very quickly. So then there's gonna be another wave of education that's gonna be needed around, okay, what are the actual enterprise use cases where you can have technology powerful, though it may be right on the, portion of the time. Another aspect of the ML and AI opportunity space is in the kind of consumer market and consumer facing

applications of it where a lot of the earlier attempts at machine learning and AI were, as you were saying, predictive in the business context of, I see that I've made this many sales in the past quarter. I predict that I'm going to need to order this much inventory to be able to accommodate future sales,

But a lot of the more recent popularity around AI have been in these natural language applications of, hey. Look what this fancy thing can do. I can ask it this question, and it knows all these other things about x, y, and z, or in kind of software engineering where it's I can tell it, build me this application that does x, y, and z, and it'll be able to put together something that works, sort of.

And I'm wondering what you see as the kind of consumer grade AI opportunities, how that has changed the business kind of appetite for investing in machine learning and AI and some of the new kind of landscapes that it has opened up in terms of avenues for different types of machine learning applications, different types of data and predictions, and also in that question of risk and accuracy, kind of how that translates into the consumer space as compared to the a the enterprise space.

Yeah. So that I think that's that's exactly right. I think there is, you know, that's a very clear moment that's happening now when, you know, very broad category of population is learning about AI and the power of AI. So, you know, a lot of people say, hey, this is the iPhone moment

of AI, and I think that's a good analogy. Just to to to push the, AI the iPhone analogy a bit further, You know, if if if you recall, there was, when the iPhone came out, there was, like, this interesting moment when the enterprise, was trying to really control

people's phones. And, you know, part of it was BlackBerry. Part of it was just great also constraint about, like, how you could do what what you could do with a mobile phone. And then the the power of the consumer tool that the iPhone was basically overwhelmed any kind of, like, resistance. And, you had this, you know, bring your own phone kind of movement. And over period of time, that was that was pretty short than

the iPhone went from being a consumer tool to also being an enterprise tool. I I I think this this little bit of that happening in the enterprise right now where now, you know, people play with chat g p t. So, the expectation

that they will be able to do the same thing at work is is is already there, and that's only going to increase. Like, you don't wanna spend, you know, all new all new day, you know, a little of your time with JetGPT on as a consumer tool, and that's gonna answer your questions all your questions. And then you, you know, you you go to work or you turn on your, you know, you you you start, working from home, you work a day at home, and then, you start using clunky tools that require, like, all sorts of, like, different interfaces. So I think I think that's happening. That's creating very high level of expectation, and I think the enterprise is going to need to,

adapt to this and evolve. You know, that's certainly a whole avenue of discussion around, data tools that, you know, you and I and other people listening to this love. There's certainly this interesting trend around, okay, what what does that mean in terms of BI tools? You know, do you need still to be a SQL expert to query them, or do you even need to be, like, a Tableau Looker expert to do it?

Why would you have a natural language interface that enables you to ask any any kind of questions whether it's simple or complex, and have the data to all return in natural language exactly what you want to know? So this, you know, this,

certainly that and there's a bunch of companies that are seeing the opportunity and starting to to, to work on this. So I think I think that's, interesting. I'm sorry. You you were asking me about, what was the The kind of impact, on the the kind of the way that AI becoming kind of consumer grade and consumer facing, how that has impact the list of kind of opportunity spaces for bringing ML and AI to bear on different types of problems and the

the way that risk and accuracy need to be considered in that consumer facing space versus,

how it's thought of in the enterprise, and and and also the level of kind of education that's necessary as we bring AI and ML into a broader community and into kind of general use? What are some of the kind of responsibilities of the people who are building it to be able to actually provide some of those guardrails and understanding of these these are the actual limitations, you know, particularly in that space of chat gbt is starting to hallucinate again.

So there's certainly another part of the consumerization of of AI, that creates new use cases in the enterprise. So if you use, chat gpt to, you know, write an email to your friends, you certainly wanna be able to use, ChatGPT or the equivalent to create marketing copy, or internal communications

or, like, all sorts of different things. So there's certainly, all sorts of different enterprise tasks that used to be done a certain way for a long time that people are going to want and expect to be able to do using, using those tools. The the key difference between, most of the consumer use cases and the enterprise use cases is that a little bit to the earlier point,

there's a number of enterprise use cases where you just cannot afford to get it wrong. And therefore, the well known and well publicized issue of, generative AI hallucination becomes a particular problem. That's particularly true in circumstances, where you are customer facing, where you are real time, and where not getting it right, can yield,

if you didn't and and and and serious consequences. So, you know, like, simplest example, you don't want your enterprise chatbot to start, saying racist things to your customers. So, you know, again, like, we are in that descending phase of the generative AI hype cycle where everybody is like, okay, I can do all the things.

But, what I'm seeing in my daily conversation with entrepreneurs, including the founders I work with that actually are in the trenches of deploying some of the things, in production in the enterprise is, okay, what do you actually do about the hallucination problem? And, it's a very unsolved problem, and that's what is so fascinating

about, you know, how quickly the this whole generative AI space is is is evolving. It's like, you know, week by week, you see, different lines of thinking, different strategies. Seems that, most of the companies trying to deploy generative value. The enterprise, end up with a combination of different models. So they use g p d 4 card, to, to, you know, a bolt of the, you know, the problem space or the interaction, the prompting, and all of the things, and then they combine that with

specific models that they have. And, typically, those models are trained very specifically on the, data that each enterprise has. So it becomes a sort of a ground truth for what is true, what is not true, and they combine all of that. So there's, again, gbd4 with, you know, which gets fine tuned on enterprise data and then separate models, and then they combine them with this little bit of a sequence

to get to the to the final result. But all of that is just experiment that's happening, literally as we as we speak. And it's just crazy and and fun times. Yeah. But, you know, until we we truly, solve that, we actually gonna be a lot more limited in, the actual use cases of Internet and the enterprise than the current narrative on Twitter and elsewhere would would let you believe. You know, it will be great to to help build stuff that, you know, doesn't matter that much like marketing copy.

It it matters, but, like, it doesn't matter if you get it wrong the first time. I think it will matter a lot, for, coding. I think that's a particularly sort of juicy problem space where, just talking to a bunch of developers. So they already use chat gpt in particular, gpt 4 in their,

day to day, coding. So that that is super interesting. But then, again, for anything that requires the AI to be right, a 100% of the time, especially in real time, that's not just not gonna be a visibility until we figure out how to solve that hallucination problem systematically.

There are a few topic areas that I wanna explore. 1 of them is because of the rapid news cycle around these generative AI capabilities in terms of the kind of video generation, image generation, text generation, wondering what kind of an impact that has on the investment market of, you know, VCs wanting to be able to kind of ride this rising wave

versus the kind of due diligence that's necessary of understanding what are the actual capabilities and limitations as you were referring to, and then also what are some of the ways that you see this generative AI hype maybe occluding some of the other more fun kind of fundamental applications of ML and AI or ways that are they're being ways that AI and ML are being used in less kind of flashy industries or scenarios?

In terms of the, you know, investment and the VC landscape, there's a couple of thoughts. So 1, there's a point of moment of a gold rush that's happening right now. I think for both, the the right reasons and arguably sort of the the wrong reasons, the the wrong reasons being that, you know, it's been a pretty dire in the,

VC and startup world for the last couple of years, and junior AI happens to be the the 1 bright spot. So there's a little bit of pent up energy, from, you know, venture capitalists to just go really hard at the 1 space that seems to be moving. So, you know, arguably wrong or not wrong, but certainly circumstantial. And then there's the right reasons which is, you know, a 100%. There is that, really obvious, inflection points and really impressive,

you know, real time compounding that seems to have the happening where, as we all know, like, every 2 days, it's like a crazy new thing that's happening. You know, last week was auto g p t. Before that was baby AGI and, you know, it's like every every week seems to be, exceeding the the the prior week. Now if you if you talk to people,

in the VC world, there is, you know, different schools of thoughts. And I think the reality is everybody's trying to figure it out in real time. There's so many people that say, hey, you know, that's a little bit of a first mover advantage that's still in today's world world matter.

You know, there's only so many of those fundamental frameworks that are going to matter or those, vector databases or whatever. So you better make your bets now because, it's gonna be too late. And then you have a whole different scroll of thoughts, that says, well, if you think of the, you know, big winners of the prior waves, very often, they were latecomers, you know,

the late commerce. The obvious example would be Google, which was search engine number 27 or whatever the number was, or Facebook. That was certainly not the first social network. So, you know, everybody's trying to figure out it in real time. As a second thought, I I I, you know, it it very speculative, but, I was actually just just, sort of jokingly, but also not so jokingly tweeting, the the other day that the structure, of companies, especially early stage companies

today and going forward was going to be different. So I was basically saying, hey, you know, standard startup team now should be CEO, CTO, 2 engineers, and GitHub Copilot.

And then I updated that tweet, just a couple of days ago saying, well, you know, on second thoughts, actually, no, the default team, should just be CEO and, auto g p t. And, look, it's, you know, it's it's it's Twitter, so it's meant to be a thought experiment in in real time rather than profound and definitely true statements.

But I think there is some some interesting food for thought behind that. And particularly, since we're talking about the VC in terms of what that means, in terms of, like, how you not just build but also finance those companies. And, if you truly end up in a situation where you can have much smaller teams, do those teams

need that much money to get started? You know, just the way, when, the cloud came up, you know, suddenly you did not need to spend $5, 000, 000 just to have some servers in your backroom, and you could get started with,

AWS for, like, you know, a $100, 000 or whatever. I think I think that is, you know, something comparable to this is Apple is is happening in AI now. So look, I I don't know, but I think that's really interesting and and that was truly going to be the case. I think that has pretty massive implications,

in terms of the overall structure of the venture capital industry, which is coming from, years of, raising very large, large funds. You know, my sense is that, like, the the as as always, the reality will be somewhere in between, but I think that's something directionally interesting. And, you know, since from the the entrepreneur perspective, I'm being even more provocative here, but, you know, it used to be that you would need to be hyper focused

on 1 thing, and you could only do 1 thing at a time because you had finite resources of time, money, and people. But in a context where, you know, you can get a lot more done, certainly doing, you know, more with AI when you have, like, a work on a very specific problem and you're super focused on it. That will never not be true. But could you also build companies where you do multiple things at same time? And you're a seed company and you try multiple, you know, avenues

in your path to get to product market fit. So you actually launch 4 products, not 4 MVPs, not just 1 MVP, and then you kill the 3 that don't work and you work with the 1 that do work. So, yeah, I I really think that we we're, you know, in the possibly early stages

of, a brave new world where a lot of the ways we've been building a financing company may may be changing quite a Yeah. It's definitely an interesting aspect. And there there's another question buried in there that I wanna get to, but there are a couple of leading questions that we need to go through first. And the first 1 is we've been talking a lot about kind of generative AI because that's the 1 that has all of the interest right now. But in terms of the overall landscape of machine learning and AI, I'm curious if you can just talk to the core screen divisions that you see in terms of the different categories

in the landscape and kinda some of the ways that that manifest that are not necessarily even tangentially related to generative AI. Yeah. Absolutely. And, you know, it's again, we're like in that moment where generative AI has has brought the whole AI field, mainstream. So a lot of people seem to be discovering AI all of a sudden, but, the reality is that, this whole current wave of AI restarted in 2012 when, the power of deep learning, became very apparent.

So, there's actually been a whole generation of companies that have been building and deploying AI in the enterprise at scale. I'm an investor in 1 of them, a company called Dataiku, which has become the global market leader for, for enterprise AI currently. And, you know, certainly Dataiku is being, very active on on generative AI now, but,

it's also rich very significant scale. I think that's public, you know, there are 200, 1, 000, 000 in revenue and, you know, across the world and, like, 100 and 100 of, of of customers doing AI for today's, you know, enterprise problems as opposed to, basically, you know, the the the new use cases that are being, created by generative AI. And, those are some of the things I was alluding to. Like, you know, trend prediction, fraud prediction, supply chain optimization.

So the whole world of what if you want the sort of structured data kind of, kind of AI. Now there's been, you know, a whole wave, which is now getting accelerated by generative AI, but of of of use cases in the enterprise computer vision, you know, audio recognition, for things like, you know, manufacturing, surveillance, and

security and and and all the things, which, you know, companies like Dataiku and others have been have been handling. But but now, engineering AI is accelerating this whole, like, new world of of different use cases. And I suspect that, companies ultimately will end up doing both. That's certainly the the path of a company like Dataiku is, is going down. If you if you think of, companies like,

Databricks, which have become, over the years, less of an AI company and and more of a data infrastructure company. But now it's like, you know, going back to building data infrastructure with DALL E. You know? So I think all enterprise AI companies will be will end up being that combination of, like, hey. We do AI on structured data. We do AI on unstructured data. We, enable you to leverage outside models. We'll lever we enable you to host your own models. So we,

you know, all forms of AI will be sort of, like, unified, through those companies. And then for the companies that are building the ML and AI infrastructure for being able to train and build and deploy and monitor and fine tune these models, what are some of the challenges that they are facing in terms of the engineering capabilities, customer acquisition,

figuring out because of the velocity of the AI space, what are the actual problems that need to be solved as new problems get discovered? Yeah. I think I think I think that's exactly right. That's exactly the problem. It's, you know, there's a whole generation of of companies emerging, that are calling themselves, LLM ops, which is, you know, the new take on MLOps. As we all know, the the world of MLOps over the last few years has been both exciting but also challenging.

Exciting because, you know, again, the the world of enterprise AI has been accelerating dramatically over the last few years, even before generative AI. And therefore, there's been demand, and and a very clear market need for

ops, kind of, you know, tools and and and platforms. At the same time, MLOps became very crowded very quickly because, you know, a bunch of smart founders saw the opportunity and then a bunch of VCs or etcetera by the companies. As a result, you have just, like, too many MLOps companies. So now you add on a new layer, which is, like, all the LLMs LLM companies and, you know, a lot of people have have seen how many generative AI companies that were in the most recent class at YC, which was,

you know, partly, something to to, like, joke about, but also, really interesting in terms of of trend. But the the the question for El Alarm of companies and challenges is exactly what you're describing, which is every week is different. So, you know, nobody knew what the auto gbt, you know, recursive agents were 2 weeks ago. I mean, of course, the concept itself is not completely new, but I've implemented

is new. So what does that mean? Is that what you need to support? You know? And what will the world look like in 2 weeks from now? So there's that. And then there's the other part, which is, you know, a lot of people talk about generative AI, especially in the enterprise. But, you know, as always, the the hype is a little ahead of the reality of it. And I think there's a lot of people that are trying to ponder

what they need to do and figure out what that means for them and trying to get smart and educated about the space. So I think the the demand for LLM ops at this stage is very new and experimental and untested. So look. I mean, you know, it's it's great. Like, every new wave sees, a a group of new infrastructure companies,

and, great companies will be built. And, you know, and a lot of other companies will be will turn out to be less, less necessary. But, like, you know, you you see certainly the emergence of a of a stack there, which is, nascent but interesting, and, you know, some of it is vector databases, so there's whole space of, you know, the the pine cones of the world that are already commending, you know, valuations that are

reportedly well ahead of, the current reality of the business, but that's fine. And then you have emerging frameworks like langchain that, you know, like everybody, that sort of joke that, you know, however many years ago you wanted to sound smart, you would say, hey, blockchain. And now, like, today, you wanna sound smart, so you say, hey, link chain. You, like, even if you don't necessarily know what link chain means,

but that's certainly coming up in all sorts of, kind of conversations. But, like, be beyond the joke, like, the the, there's certainly, new tools and new platforms that are emerging, and, it's both challenging, but also a little fun. And so now circling back to the question that I wanted to lead into when we were discussing kind of the the impact that these generative models will have on accelerating business capabilities is that for companies where the model is the business,

there is a a different kind of barrier to entry where they need to have access to enough data and, compute resources to be able to build and train the models. And I'm wondering how you see companies addressing that challenge, particularly if they are in an even remotely adjacent space to some of the the big tech kind of incumbents who already have all of the engineering talent, troves of data, you know, top tier kind of machine learning, compute clusters.

Kinda what do you see as they the challenges that these earlier stage companies are hit running up against as they try to build models as the core component of their business or as a component of their business against these kind of existing moats? There's 2 different, cases here. This,

discussion around the the providers of models and the discussion around the users of models. So from a provider of model standpoint, if your business is to provide a a model as a service, then I I do think, that it's going to be increasingly hard to provide something that's truly general, unless you have access to massive amounts of capital. I think that phase where you saw not just OpenAI, but also, you know, the anthropics and the adepts,

you know, come up on the surface. I think the I I I don't know how many more of this, there's gonna be appetite to fund, because, those are very big ticket, investments where, like, every round needs to be, you know, starting at, like, 40, 50, 60, you know, $1, 000, 000, and I don't know how much, the market will sustain. So, look, I don't wanna say that game has been played because,

you know, famous last words and the next, big thing comes. And, you know, there were some people that thought that, you know, DeepMind, had killed the game back in the day, so that was certainly not the the case. But at least for this current market phase, like, I I don't know I have any more. I I I I, think there's plenty of opportunities, to create an offer as a service, industry specific models and that we just, starting to scratch the surface

of those. I thought that the Bloomberg, 1, Bloomberg GPT, I believe it was called, was a a particularly interesting example. So, you know, Bloomberg being a company that has access to tons of financial data, created their their own engine there. You know, my sense is that there'll be a bunch of those, you know, for health care and for transportation and for, you know, subsegments and all the things where of companies that do have access to data

will be able to create models and offer the service. So that's sort of the provider side of things. And then from a user side of things, I actually don't think, especially in the enterprise, that the OpenAI and GPTs of the world are gonna be the final state of affairs

and that all enterprise AI applications will just be a wrapper on top of that 1 big model. Actually, I think it's gonna go, at least in part, pretty much the opposite way where in lieu of of or or possibly in addition to, those very large models, you're gonna have a whole zoo of smaller models. I actually think that, you know, open source is going to play a an incredibly important role there.

And, the reason why there'll be a bunch of models will be partly cost, partly not being completely dependent on 1 company, like OpenAI platform risk kind of thing. But also, you'll you'll need to be able to heavily customize those things to your specific needs. And, arguably, if what you wanna do is, something that's going to, automatically respond to your employee, inquiries and you in the US, the fact that, g p d 4, speaks, Polish and Indonesian,

doesn't matter that much. And you actually want something that's much narrower. So, obviously, you know, this concept of distillation and to give very large models and and turning them into smaller, more more relevant, models. But, you know, I think I think there's some directionally, multiple small models, again, in lieu of or in parallel with the large models is is pretty much what the future is gonna look like

at at this stage, I would I would think. Yeah. I I actually just recently read a Wired article talking about Sam Altman saying that

they're hitting the point of diminishing returns. We're just throwing more data. The problem isn't going to produce the GPT 5, and that we're actually at a point where we need to start thinking about what are the different architectural aspects of these models that we need to be considering. And I definitely think that the open source models is a really interesting space, and I'm wondering too whether we're starting to get to the point where we need to actually have these

more narrowly scoped models, and then have a kind of meta model on top to manage orchestration across those different subcomponents, and and that maybe that's the next approach versus just this 1 monolithic model that does everything within a particular problem domain. Yeah. Yeah. Absolutely. I mean, that's the LMM ops space that we started talking about, which, I think, you know, is going to need to exist despite the challenges that we mentioned, but that's certainly 1 aspect of it.

Certainly, you know, chains of model sends the whole, like, lang chain thing, multi prompting, recursive prompting, you know, observability, and then all the stuff that's coming from the, you know, more traditional MLOps world to such extent there is such a thing, of, you know, model fairness and bias and, you know, deployment in production. Like, all of this is going to need to exist in the generative AI world just as much as, in, you know, other forms of, of, of AI.

And then, you know, just, building on the first part of your point and the Sam Altman thing, there's there's that big question in in in research as well, which, you know, may or may not be in the scope of this conversation. But, you know, indeed, that's the debate that has been raging. It's, you know, are you going to throw just more data at these models and get, you know, better better results? Or, do you need other forms of AI to be added to to this?

You know, and it's like the whole question around the return of, like, symbolic AI or good old fashioned AI, you know, and helping AI understand concepts and introduce concepts of reasoning. Because as we all know, right now, as, you know, amazing as it is, and others, like, I've just no idea what's going on and, just produce the next most likely word given, you know, a prior sentence. So it works magically, but, like, there's just no reasoning, whatsoever.

Yeah. The so so many different interesting conversations to be had on that. And digging a bit more into the question of open source models with things like chat gpt holding the limelight and being kind of sucking up a lot of the oxygen in the room and being a proprietary model, I'm curious what you see as the most interesting movements in the space of open source models and some of the future development or areas that you would like to see more activity.

Yeah. I think open source is usually important, not for just, like, business reasons and technical reasons, but also most, like, society level kind of reasons. So, you know, the the the reasons why, OpenAI doesn't open up their models are, well, not publicized. You know, there's debate there, but, you know, certainly, that's that's rational for saying that there's some risk involved there. But we do need open source as a counterbalance

to whatever power is going to be, accumulated by OpenAI and even, you know, further power accumulated by the big tech company. So I think that I think that's I think that's really important. That's certainly why, companies like Hugging Face are going to be immensely important to the ecosystem, even more so than they are now as the home of open source

ML and AI. And then there's, you know, there's a whole separate discussion about, okay, what does it actually mean to be open source when it comes to the world of AIs, at the, you know, model, the weights, the algorithms, the data, and all the things. So there's all sort of variation and sort of shades of gray around this.

But, you know, there's certainly an argument for, like, radical open sourcing and, you know, equally a very obvious, argument against that because, completely open source model, also opened the door to just also for ill intention people to be leveraging that, you know, which leads to the whole discussion around regulation and and all and all the things, which is a whole separate, kind of kind of worms kind of discussion, but, absolutely essential.

Yeah. Definitely brings up shades of the kind of cryptography export controls aspects as well as in terms of the open source space. You know, what is the OSI equivalent for machine learning models, and what are some of the licensing considerations? Like, can you use Apache 2 on a model? What does that even mean? Yep. That's that's that's either way right. And so in terms

of your experience of working in the space, investing in companies, talking to entrepreneurs? What are some of the most interesting or innovative or unexpected ways that you're seeing ML and AI used in business and social contexts? Yeah. We we're really in that phase where, you know, everybody's trying to apply AI to, like, all things. But, you know, I think we're just at the beginning of, like, that massive wave of, experimentation.

Look. It's certainly, you know, the applying AI to the data world, which we talk about that I find interesting. There's, AI applied to just complete product creation, website creation that I find fascinating where you can create, like, an entire app, just through commands. I think that that that just has, like, absolutely profound consequences and not just on the no code world, but, in in general, if you think of, the proliferation of software in the world as having been gated by,

the scarcity of talents and the, you know, really, really small number of people in the world that know how to build the things. It's like a whole new world that opens up if you can just build software product for everything, and everybody can can can do it. You know, and just like you guys can be, like, unexpected unexpected user cases of, you know, generative and all the things. So, like, I'm, you know, I'm I'm I'm I'm very much,

out of that world, but it's been just fun to hear. So my, colleagues and and younger friends, talk about using, chat gpt in a dating context. And, it enables them to basically scale their efforts. So, as opposed to conversing with, however, you know, whatever the number is, like dozens of people on across multiple apps at any point in time, you are doing it manually. Now they just use, chat gpt to create conversations, which, of course, is hilarious when you think that, people on the other side,

are probably or will soon enough, do the same. So you'll have, basically chat GPT, talk to chat GPT, or whatever system, instances, chatting to 1 another. And, like, I'm I'm curious to see what comes, on the other side of it, whether you get, you know, better matches or or worse matches. But, it's, yeah. I would I did not spend a lot of time thinking about it. I thought that was a really funny and surprising use case.

Yeah. Or we just end up with another, another version of I don't know if you ever saw the movie, Jexi. Yes. Yes. Would be. Yes. Yes. With the jealous AI, in your phone taking taking control of your life. Yes. Yes. All all of this is, is a 100% going to happen, by the way. And, again, like, I don't know if that's beyond the scope of this podcast, but I got, you know, all the her thing. And we were talking about, you know, hugging face

a minute ago, but the the part of the reason why Hugging Face is called Hugging Face is that the the first version of the company before it pivoted to the current model was to be like a her like, not her of the movie, like her like kind of, AI friend to whom you would speak and build a relationship with, which was, you know, a recall ID, arguably

a bit strange, but, certainly ahead of his time of where the technology was at the time. But the technology is here now, and, that's absolutely be going to be a thing. And, you know, humans developing feelings for, AIs or feeling that, AIs understand them better than any human being would be that's not 20 years away, and now that's, like, next year or even possibly right now.

So it's, this, you know, like, this is what we all experience sort of this incredible acceleration of the future where, what felt like tickets away is just, acting right now.

Yeah. It it also brings up a whole other interesting conversation that we don't probably have the time or the expertise for of security in an ML context where rather than just being network or application security, we're now having to think about security in terms of combating malicious AIs for on behalf of somebody or, like, flagging potentially malicious AI activity for somebody maybe interacting on social media and kinda saying, this isn't actually what you think it is.

Yeah. Yeah. Yeah. And, so I think that's a problem that we're going to resolve need to resolve as a society very quickly the, you know, the proliferation of, like, it takes and and all the things for security reasons or others. So, yes, if you can spin up

some website in a in a second, you can, you know, create phishing attacks like you've never were before. You know, used to be, you know, sadly because I'm active on social media. I have a lot of, sort of impersonators and, like, fake accounts, fake, metric,

accounts on, especially on Instagram where I'm not active. But, anyway, And, you know, people have been building remarkably complex schemes for, like, crypto fraud where, they would have very precise website, like, a workflow and all the things, and it's just extraordinarily hard to figure out, that, you you know, it's it's a it's a scam unless you pay really

a lot of attention. And so now we say, I like, if you can build those websites in a second, you can, like, spin up, like, you know, thousands of them, you know, completely customized to every single person and, you know, target. And that's that's incredibly scary. You know? And then there's a whole whole discussion around,

I I don't think people say, well, you know, maybe we should tag deep fakes to make it clear there are deep fakes. And, I was telling you the other day with, Victoria Parbelli, who's the CEO of Synthesia, which is a fascinating generative AI company, that, I work with and I'm on on the board of. And, you know, we're sort of joking about, Web 3 and crypto applications.

But he he was saying, and I thought there was some real insight here that, you know, instead of trying to identify all the deep fakes, maybe what we should identify are the real images and songs and because, you know, those are gonna be very soon much fewer. And, you know, the the web 3 crypto joke was, like, maybe, actually, that would be a real use case for, a blockchain where, you know, each actual real image or or, you know, again, songs or creation would have a unique,

sort of digital idea that could be verified against the the blockchain. But I think there's there's something really really interesting here, around, okay. Well, let's maybe, focus on on on defining the truth as opposed to defining, what is not true. Yeah. That's definitely an interesting and scary proposition. And, if we're gonna use the blockchain, we definitely have to work on figuring out how to scale it, both in terms of actual

throughput and the kind of environmental impacts as well. Yeah. Well, that that's a different podcast. Yeah. Absolutely. So many different interesting kind of rabbit holes to dive down anywhere you look. And so in terms of your experience of working on the mad landscape and digging into the space of ML and AI over the years? What are the most interesting or unexpected or challenging lessons that you've learned personally? So there's there's a couple of lessons.

So, you know, 1, is a little bit, what we said earlier, which is that, as an entrepreneur, you need to truly be thoughtful about applying AI to use cases where AI can succeed. And, I have a little bit of a mental model around this, which is, like a, you know, 3 by 3 kind of matrix, where on 1 end of the spectrum, you would have, use cases where, the AI needs to be right, a 100% of the time or the product needs to be right by a 100% of the time.

And then that needs to be true in real time. You need to have that 100% correct answer in real time. And 3, it's a high stakes kind of, kind of environment. So, like, an an example of this would be, you know, a a product that would operate in the context of an ER, room where it would be, like, a life or death situation where you need to make a real time decision, or or or self driving, I guess, would be another example of this. So that's that's 1 of the spectrum.

And then the other end of the spectrum, and that's kinda like 3 by 3, you know, mental model would be use cases where, yeah, if you get it right, that's great. But if you don't, you know, nobody dies. And and or 2, you can, you don't need to do that in real time. So, basically,

you know, before you need to provide whatever the the final output is. And, yeah, it's it's low stakes and nobody nobody dies if you if you get it wrong. So you don't need to get it wrong all the time. It's done in real time. And if you get it wrong, then it's not a life or death, kind of situation. So, in that middle model, the best use cases of AI tend to be towards that latter end of the spectrum.

And, there's a lot of them, and we talked to, you know, about some of them around, you know, creating content in the enterprise that's that's, you know, that's not real time, that's low stakes, and that's okay if you don't get it 100% of the time. But answering customer request in real time is something that's, super important. That's, much more towards the the former and, you know, using generative AI in particular with the hallucination problem to

do this, becomes, becomes trigger. So that's that's 1, you know, first half of the answer to the question. That's what I've learned. It's, make sure that you apply AI capabilities to the right problem. And the second aspect of this is that still to this day, I think, there is a little bit of misconception around how how much of a solved problem, AI truly is.

And, you know, for all the APIs and all the tools and all the things, I think people, depending on the use case, of course, that people misappreciate to which extent, AI, is still a little bit of a deep tech kind of endeavor, especially if you're a true AI company. And by true AI company, I mean that you, do a lot of model work. So either you create your model or you train a model or you, you know, fine tune or significantly customize an existing model.

As soon as you start doing that, then you need people who are able to do this. There's not that many people around who have, those kind of capabilities, and there's gonna be experimentation. And so the company ends up looking like the deep tech company where where where the first, you know, x period of time, whether that's a year or 2 years or 3 years, depending on, what you are trying to do, that that that's not gonna be your, like, you know, super agile SaaS MVP,

you know, screw it, ship it kind of thing. It's gonna be much more like, okay, heads down. Let's train the AI and build the product. And that has all sorts of different consequences in terms of, you know, funding, in terms of, expectations of, pace of development and and and all the things. And look, we're we're sort of joking earlier in this conversation around, okay, maybe the companies of the future will be just, you know, to GPT or whatever comes next and and 1 person.

But, I think the the reality as of today is is very much not that. The reality as of today is very much, either team of experts and take time to get it right. Again, if you do model work and if you're if you're a, sort of, an applied ML company that doesn't do any model work and basically, is a wrapper on top, of,

OpenAI, you know, to be the model or or whatever, then, you know, ultimately, you're very much a SaaS company, in my opinion, as opposed to an AI company. And you have the, you know, the pros and cons of of a of a SaaS company, but you certainly have a level of velocity, that's, that's different from those kind of, like, deep tech, AI companies.

Yeah. And also in terms of the generative space, it'll be interesting to see once we get to the point where the gen the the output of that generation is other models, and we're actually just using generative AI to build more AIs, which I I know that we're probably there in some edge cases, but once that becomes the mainstream, that that'll be an interesting inflection point as well. Yeah. No. No. Yeah. Absolutely.

Absolutely. And, yeah, I think we are in the toy part of the cycle where, you know, we'll get super excited about whatever is coming out. And, you know, it's all magnified by, or accelerated by by Twitter and all the social media. But the reality is, like, all these frameworks, are extremely early and and not working. But, yeah. I mean, given the exponential pace and the compounding, like, all of this, is going to go from the toy phase to reality very, very quickly.

I think we've probably already addressed a bit the question of when ML and AI is the wrong choice for businesses. I don't know if you have anything to add to that before we go to the next question. Yeah. Just that, businesses should be careful. And I I think they are you know, in all my conversations, I think people wanna use, generative AI but, are in the learning phase. But, yeah. You know, as for the above, yeah, the the reality is that right now,

for a bunch of use cases, we're just not there in terms of generative AI being ready for the enterprise. So people should, somehow resist getting completely caught in the excitement, you know, certainly, staying on top and, you know, free acknowledging it's it's it's wonderful, but, pausing a little bit before, truly, trying to deploy any of this, in any kind of, particularly high stakes, way as the you know, as we're just discussing.

As you continue to explore this space and figure out what are the possibilities, what are the risks, what are the pitfalls, what are the areas of ML and AI that you're paying closest attention to in your own work? Yeah. All all of the, all of the above. I think, you know, what while wanting to be cautious, high cycles,

look, I'm I'm like everybody else. I'm just super excited about what's going on. So I I, you know, have it's very simply simple classification of, of of companies because of the question of, like, what is an AI company comes up all the time. So I, the other day, I came up with, just like a very simple categorization, you know, which I I don't mean to be, like, a definitive

statement or anything, but more the sort of, like, mental model. But I'm excited in all 3 categories. Like, 1 is, you know, AI companies. And by AI companies, I mean, companies that, you know, play with, work closely with models, whether they build them, find to them, or, you know, customize them or whatever. To, applied ML companies, which is the category we were mentioning earlier, which, in my opinion are more SaaS companies, but basically companies that sit on top of somebody else's model.

And 3, companies that create, tools and frameworks, and those are, infrastructure companies. So you end up with AI companies, applied AI companies, or AI application companies,

and then, yeah, infrastructure companies. And, I think there's certainly a moment right now where great companies are being built in all 3 of those categories. And as an investor, I'm certainly very actively looking at at all of this. But I'm I'm looking at them differently with different use cases. Right? Different, sorry, criteria for the

first category of, AI companies. I pay particular attention to, okay, do those people have a very deep bench of, ML and AI talent that can, be truly, innovating at the model level? For the second category, which is AI applications, I'm also thinking about, defensibility. Can they build something that is, you know, meaty and thick enough on top of somebody else's model that, they can be interested in companies in the long term? And I'm thinking of, collaboration.

I'm thinking of, like, workflow, but, again, more sassy kind of things as opposed to AI. And then for the 3rd category of, infrastructure companies, I'm I'm thinking about the the needs of, tomorrow and whether this company are, addressing them and whether they can go through being whatever little tool they are by definition at the beginning to something that feels more like, an AI

infrastructure platform doing multiple things. These are enough blue ocean for those companies, to go through that journey and then being the place where, people will build, especially enterprise driven AI companies going forward. Are there any other aspects of your work on the mad landscape and the overall ecosystem around ML and AI that we didn't discuss yet that you'd like to cover before we close out the show? No. It's just, like, I need AI to, update this, AI landscape because,

you know, it's, like, every minute. Like, I'm sure during this conversation, while we were recording this, so there's been, like, 2 major breakthroughs in AI that were that were announced. So just like, you know, keeping track, of, stuff is, just remarkably

challenging. You know? And I have this fantasy which, of course, will never happen. It's just, like, taking, you know, just a week, to, like, of doing nothing else but, like, consume all the AI newsletters and the AI podcast. I guess, like, this 1 and, you know, I would feel so much, more on top of things, but it's it's just, look, I I I love it. I've been in this space for a very long time and,

you know, people ask me often, well, you know, how do you feel about it? Because now there's, like, a bunch of tourists that's sort of waltzing in, and, you know, claiming deep expertise and they haven't really done the work. And, like, look, I mean, that's that's a little bit of that, but, like, ultimately, who cares? And it's, you know, it's perfectly fair and, like, people just, you know, go seek the heat to work wherever it is. You know, but my my my my, sort of deeper

sort of reaction is that, look, I'm I'm I'm super happy. It's so fun. It's so exciting. It's such a moment. And, I actually don't see enough people in the general AI ecosystem, like, truly celebrating this. Like, there's a lot of debate, and I got back and forth and, like, some beefs and some people, like, disagreeing, but it's it's, you know, we live we live at this exact moment through an incredible period of time, and, like, we should all be super super thrilled and passionate and and just,

you know, be grateful for it. Well, for anybody who wants to get in touch with you and follow along with the work that you're doing, I'll have you add your preferred contact information to the show notes. And as the final question, I'd like to get your perspective on what you see as being the biggest barrier to adoption for machine learning today. Well, you know, I don't wanna repeat some of what I said above, but I, you know, I I think at this stage, for enterprise application hallucination,

as we discussed, is probably the biggest problem. Like, this stuff needs to work, right, for the

being truly deployed in the enterprise, and we're not quite there yet. So, unfortunately, that's a less sort of sexy thing to talk about, and you probably get less likes on Twitter if you focus on that too much. But I think that's the heart of the problem right now. And I'm I'm very curious. I know a lot of very smart people working on that exact problem right now, and I'm I'm curious to see if that's something that just disappears overnight or whether that ends up being quite sticky.

So we'll see. Alright. Well, thank you very much for taking the time today to join me and share your perspectives on the ML and AI ecosystem. Appreciate the work that you've been doing on the mad landscape over the years. It's great to be able to have those snapshots

and your opinions on what this all means in terms of the ways that we can think about those broad categories. So appreciate all the time and energy that you and your team are putting into that, and I hope you enjoy the rest of your day. Thanks. Thanks, Tobias, for having me. And, you know, if and when we do this again in the future, I'd be curious specifically for the AI world,

how different everything will be from the conversation we had today. It's it's fascinating. Absolutely. And and and are you going to be training chat gpt to generate the bad landscape for you? Yes. Well, you know, I already have a, I was already, joking the other day. I have this, Synthesia powered avatar, and I was saying that, I mean, like, you know, like a fake grand reveal where I was saying, well, you know, for the last few months,

my AI avatar has been sitting on all my boards and having doing all my Zoom meetings while I was couch surfing. So, you know, joke for now, but I don't know. We'll see in a couple of years what ends up being true or not true. Absolutely. Alright. Well, thanks again, and have a good rest of your day. Yeah. Thanks, Thank you for listening, and don't forget to check out our other shows, the Data Engineering Podcast, which covers the latest in modern data management,

and podcast dot in it, which covers the Python language, its community, and the innovative ways it is being used. You can visit the site at the machine learning podcast.com to subscribe to the show, sign up for the mailing list, and read the show notes. And if you've learned something or tried out a project from the show, then tell us about it. Email host at themachinelearningpodcast.com

with your story. To help other people find the show, please leave a review on Apple Podcasts and tell your friends and coworkers.