How Investors are using AI - [Business Breakdowns, EP.240] - podcast episode cover

How Investors are using AI - [Business Breakdowns, EP.240]

Feb 05, 202649 minEp. 240
--:--
--:--
Download Metacast podcast app
Listen to this episode in Metacast mobile app
Don't just listen to podcasts. Learn from them with transcripts, summaries, and chapters for every episode. Skim, search, and bookmark insights. Learn more

Summary

David Plon, founder of Portrait Analytics, joins to discuss the tangible ways investors are integrating AI into their workflows. The conversation covers using AI to filter news, streamline pre-buy research, generate ideas, and perform quantitative analysis. Plon also shares insights on crafting effective prompts, the importance of experimentation, and the evolving capabilities of LLMs, including context windows and agentic AI.

Episode description

Today we have a special episode breaking down how investors are using AI. 

This is a question I get from many of you, and while there is no shortage of content on the implications of AI, I know there's an appetite to learn more about tangible use cases, how to make sure you're getting the most out of these tools, how to think about advancements in the technology, and ensuring that you're keeping pace with the innovation curve.

So my guest today is David Plon, Founder of Portrait Analytics. Now, David and Portrait have been partners of Business Breakdowns since last year, but I specifically asked David to do this episode because, one, he is really front and center to how investors are using and applying AI. But two, and maybe more importantly, he and his team come with a background in investing. 

So while the conversation doesn’t really focus on Portrait, you’ll hear references to what he and his team are building and how they’ve shaped it for investors, you’ll very much understand when you hear David talk that he is someone who understands the pain points of an investor.

I think everyone will find something in this episode that will benefit them in their day to day, and I would love to hear the feedback. 

For the full show notes, transcript, and links to the best content to learn more, check out the episode page⁠⁠⁠⁠⁠⁠⁠ here.⁠⁠⁠⁠⁠⁠⁠

This episode is brought to you by ⁠⁠⁠⁠Portrait Analytics⁠⁠⁠⁠⁠ - your centralized resource for AI-powered idea generation, thesis monitoring, and personalized report building. Built by buy-side investors, for investment professionals. We work in the background, helping surface stock ideas and thesis signposts to help you monetize every insight. In short, we help you understand the story behind the stock chart, and get to "go, or no-go" 10x faster than before.

Sign-up for a free trial today at ⁠⁠⁠⁠⁠portraitresearch.com⁠⁠⁠⁠⁠

Business Breakdowns is a property of Colossus, LLC. For more episodes of Business Breakdowns, visit ⁠⁠⁠⁠⁠⁠⁠joincolossus.com/episodes⁠⁠⁠⁠⁠⁠⁠.

Editing and post-production work for this episode was provided by The Podcast Consultant (⁠⁠⁠⁠⁠⁠⁠https://thepodcastconsultant.com⁠⁠⁠⁠⁠⁠⁠).

Timestamps 

(00:00:00) Welcome to Business Breakdowns

(00:04:05) Meet David Plon: Investor to Founder

(00:05:54) Pain Points in the Investment Process

(00:09:08) AI as a “Smart Filter” for News

(00:11:54) AI’s Role in the Pre-Buy Process

(00:14:34) How AI Enables Specific Quantitative Analysis

(00:17:24) Using AI for Investment Idea Generation

(00:21:42) How to Write Good Prompts for LLMs

(00:25:27) Structured vs. Creative Tasks

(00:27:47) The Value of Experimentation & Real-Time Feedback

(00:32:28) Best Practices for Deploying AI in an Institutional Setting

(00:35:57) Why Documenting Decision Making is Essential

(00:39:19) How Models Have Improved at Using Provided Context

(00:42:02) Memory in LLMs: Near-Term Limitations, Long-Term Potential

(00:46:24) Applying Agentic AI in Investment Research

Transcript

Welcome to Business Breakdowns

C

This episode is brought to you by Portrait. It's the AI research system that I used to prepare for today's episode and for all business breakdowns episodes. Portrait was built by former buy-side investors. And they understand great investing isn't just about having more information from low quality sources. It's about having the right information organized the right way. And if you listen to the show, you appreciate diligence consists of many things.

Diving into the history of a business, framing the nuanced competitive dynamic. Tracking key signposts around your thesis. And historically, that would take up material time that you do not have. But Portrait is basically like adding an army of analysts to your team. It's powered by an AI system specifically designed for investment research workflows.

So you get nuanced idea generation. Portrait assesses the same types of qualitative attributes that we discuss on this show, and that can help identify businesses which fit your framework. Portrait also customizes research report generation. And I used Portrait to generate a primer and lay out full bear cases ahead of today's episode to help frame the conversation.

And third, there's intelligent thesis monitoring. And that's where Portrait assesses thousands of data points across value chains each day, extracting the insights, driving the business. Again, all this work would typically take Hours and hours and hours. It's at your fingertips now. Visit portraitresearch.com to start your free trial today.

🎵 Music

D

This is Business Breakdowns. Business breakdowns is a series of conversations with investors and operators diving deep into For each business, one history, its business model, its competitive advantages, and what makes it a good idea.

🎵 Music

D

We believe every business has lessons and secrets that investors and operators can

A

And we are here. To find more episodes

D

Downs, check out Join Colossus. All opinions expressed by Helena. are solely their own opinions. Their employers or affiliates may maintain positions in the securities department. This podcast is for informational purposes only and should not be relied upon as a basis for investment design.

🎵 Music

C

Today, we have a special episode breaking down how investors are using AI. This is a question I get from many of you. And while there is no shortage of content on the implications of AI, I know there's an appetite to learn more about tangible use cases. How to make sure you're getting the most out of these tools, how to think about advancements in the technology and ensuring that you're keeping pace with the innovation curve. So my guest today is David Plon, founder of Portrait Analytics.

Now David and Portrait have been a partner of business breakdown since last year. But I specifically asked David to do this episode because one He is really front and center to how investors are using and applying AI, but two, and maybe more importantly, him and his team come with a background in investing.

So while the conversation doesn't really focus on portrait, you'll hear references to what him and his team are building and how they've shaped it for investors. You'll very much understand when you hear David talk. that he is someone who understands the pain points of an investor. I think everyone will find something in this episode that will benefit them in their day to day. And I would love to hear the feedback. Now on to the episode.

Meet David Plon: Investor to Founder

David, I am excited to have you here to finally do this recording. I think over the holiday break. Many investors were spending time trying to figure out how they can get more out of AI tools. It's constantly evolving, but At least for a snapshot in time, I think you'll be able to give us some really good perspective on use cases, some of the skill sets that you can use to leverage the most out of the technology. But I actually just wanted to start out with.

your background, because I think this is really unique and can speak to why I find you such a valuable resource when it comes to thinking about this through the investor mindset. So Can you bring us back in time prior to Portrait, what you were doing and how that led up to starting this business?

B

Yeah, absolutely. Really a pleasure to be on here. So I spent my career as an investor in a few different seats before starting Portrait. I started at Barclays on the trading floor in the special situations group. was then at a long short hedge fund called Slate Path Capital, where I was a generalist. And most recently was at Bow Post, which is a large hedge fund based in Boston where I was a generalist on on the public markets team doing both public equities and distrust credit.

I think I probably first got bit with the idea of using AI within the investment research process when I was at business school at at Stanford. And this was like twenty fifteen and twenty seventeen. So deep learning was very much in the zeitgeist. We were still pre transformer right up until I graduated, but it just seemed inevitable to me at some point that this technology would be really impactful for a lot of the research workflows that

I went through and after spending a lot of time studying and exploring and prototyping, I got the conviction to throw myself full time into this project. So that's my background leading into starting portrait.

Pain Points in the Investment Process

C

When you think back to some of those stints, whether it's bow post, sleep path, even potentially on the trading floor, can you think of any scenarios where you were spending a significant amount of time that were pain points that are maybe easily solved by AI today. We'll get into some of the very specific use cases. But I'm just curious when you look back at that time, some of the workflows that you had, where you could see the biggest lift today versus where it was ten years ago.

B

It's very different today. It's interesting because the output of an investment research process is ultimately a decision and hopefully a high quality decision. It's very unlike other fields where there is actually a widget or a service being provided. When I thought about the investment research process, there were aspects of it that had friction, but that friction was helpful because it helped me build conviction personally. I was one of these guys that could

never really outsourced model building. I always had to build a model myself in order to feel conviction in recommending a position based on that company. But there were certainly aspects that I'd say limited me in terms of how productive I could be in finding and researching the best ideas. Essentially, I would bucket it all into the category of there's just a ton of information out there that I could potentially consume.

Consuming it in a way that was efficient and additive to the research process was challenging just given hours in a day. So I would put it in the three main categories. There's certainly idea generation. So finding the ideas that best fit my mental models. I knew there were probably dozens out there that were totally in my sweet spot, but at any given time it was very hard to know if I was working on one of those.

There's certainly the initial context building that goes into a idea, especially as a generalist, not even developing a differentiated insight, but just getting the table stakes on how a company And an industry works and the narrative and business breakdowns has always been a very useful resource. Whenever I'm working on something that where there's an episode, that was always a huge help.

And then I'd say lastly, just monitoring an investment idea. As a generalist, it was often challenging to stay on top of not just the companies, but the surrounding ecosystems, the customers, suppliers, competitors.

And if someone lived in a given sector, they usually were able to put together a real-time mosaic based on all those data points that they're absorbing about what was happening in the industry and so forth. And I always found that really challenging as a generalist. There was more than one earnings season when I got smacked.

Because I was the last person to realize that and market demand was weakening or suppliers were pushing prices through and those types of things. So ultimately, I would say those are the types of workflows, high-level information processing to help triage new data points.

Broadly speaking, are the types of things where AI can be really useful. And we can get into some specific applications of that. But that's how I would think about where AI can fit in a research process without replacing the parts of the process that are important for building conviction.

C

Yeah, I think that really resonates and seems to align with the people that I've seen getting the most out of it.

A lot of it is productivity and efficiency, which on the surface level sometimes people can poo-poo. But when it comes to making decisions and having clarity of thought and being able to take in the most valuable information, I think there's a lot more to it. So I liked the idea of getting into some of the use cases and I actually wanted to start backwards in terms of the buying process.

AI as a "Smart Filter" for News

And reference what you just brought up there in terms of position monitoring. What are some of the tangible use cases of AI today that you think investors are using or you would recommend they use? to just monitor what's in your portfolio and stay on top of it as effectively and efficiently as possible.

B

I think there are many good solutions today for staying on top of what is happening with an individual name. There's plenty of ways you can add it to a watch list on any number of services. And staying on top of all the new filings and transcripts and news related to that specific company. What I think historically has been hard, but now is much simpler. is being able to cast a wider net of pulling in relevant data points and a much sparser set of data. So for instance, let's say

You own Expedia. One of the key investment factors for Expedia is always going to be what's happening in the hotel ecosystem and how volume is trending, how pricing's trending, or what OTAs are doing with their distribution strategy and how it relates to OTAs.

The challenge historically is if you're an internet analyst or you're a generalist following Expedia, you probably aren't gonna want to subscribe to every piece of news of what Marriott is saying. You don't care what their profitability necessarily looks like or If they're adding new units to take share from Hilton, what you really care about is any specific data point which would add to the mosaic around what is happening in the hotel landscape.

with respect to consumer demand for travel, or if they're seeing trends in market share shift with respect to OTAs. But historically, it was impossible to pick up all those data points unless you really consumed every piece of news. There was no smart filter on top of all that data. I think with AI today, and obviously the company I run Portrait, we have solutions that target this.

But even outside of Portrait, there are plenty of ways today to now much more efficiently pick up those data points across that surface area such that where today a certain level of understanding was unique to someone who just covered travel, you're now able to pick that up much more efficiently as it relates to your investment thesis.

C

Yeah, I can definitely speak to that as a transport analyst trying to monitor truckers and knowing that there was valuable information coming out of CPG companies talking about freight costs. I spent many late Thursday nights while the rest of the world was enjoying Manhattan Happy Hour control F through

various transcripts and earnings releases. So I think that is a major one that has evolved and just the ability to gather all of that information in a much more efficient way and not control Fing the heck out of things.

AI's Role in the Pre-Buy Process

When you next move into the building up a research on a name or the pre-buy research process, getting up to speed, this is another one where I think it's the widest net of possibilities. But when you think about maybe through the lens of how you used to approach it versus how you're seeing other investors approach it.

What are some of the use cases that you're seeing people implement to really get the most out of it and maybe build better conviction or gather more information, whatever it might be?

B

It's interesting because I was someone who and still do love doing the pre-buy work. My wife likes to joke that my favorite Saturday morning activity is reading a 10K. And The thing that's interesting, what I was mentioning before, where to me that's an important part of the conviction building process is building up that context and feeling like you've read all the material and you can start piecing together the various data points.

Where I think AI can be really helpful in that process is two things. One is getting you enough information to know whether an idea should be killed. I used to do this accounting at the end of every year when I was an investor. How many ideas did I really look at? A remarkably low number that made it through to the deep research stage.

And a lot of times I could have killed an idea much quicker once I'd surfaced enough information to understand there's maybe an existential risk here. I'm never gonna be able to get past. Or the management compensation is such in a way where I'm just never gonna feel comfortable that their incentives are aligned.

And so I think where AI can be really helpful in that process is giving you enough baseline context to be able to triage the idea of, oh yeah, this is passing the initial sniff test. I want to spend more time on it and maybe spend 12 hours reading through all the historical content. And I'd say the other thing that has been really helpful is just expanding what

gets moved up earlier in the pipeline in terms of types of analyses. So for instance, I mentioned earlier CEO compensation. One of the things that I would do if I was really getting into a name is go through the last five proxies. and try to map out what are the metrics that the CEO is being comped on and how have those changed and the weighting of those changed. A lot of times that's a useful signal into what the board is thinking about. That's a bit of a pain.

And now on portrait is a click of a button. So things that historically would be more in like the deeper dive category, now I can move up. And that to me is one of the most powerful use cases of AI. So as an analyst, you're able to turn over far more rocks in a given period of time than you would otherwise and spend more of your deep research creative time on ideas that you know have already passed your initial sniff test and are worth that time.

How AI Enables Specific Quantitative Analysis

C

When thinking about really tapping into that, would that be having a list of non negotiables in terms of whether it's compensation elements or something else that you can screen through quickly, or is it you don't necessarily know what you're going to hit that might break it, but you're just gonna get there faster.

through the various iterations of getting up to speed. I know it's a little bit technical, but I'm curious mostly if there is a way that you think that you could templatize that versus not.

B

Oh yeah, there's two categories. There's definitely things that that you can templatize. So for instance, I mentioned CEO compensation. There's certain accounting things I would look for with respect to like aggressive revenue recognition or changes and certain assumptions that would inflate figures. And then I'd say there were patterns that Or if I saw that would turn me off. So one analysis that now is very trivial but took a lot of time historically was I would go back through the last

three, four years and lay out every piece of guidance that the management team had given. Both obviously the hard specific guidance, but also anything qualitative or soft, like, oh, we expect revenue to accelerate sometime in the second half of the year. And I think building up a picture of management's guidance style and credibility is really important. And that's the type of context that someone who's followed a name for a long time intuitively has that being fresh to a name I wouldn't have.

That's a situation where if if my thesis is based on some sort of turnaround execution plan that the management team is undertaking, but I can do the work and say, Oh, actually though over the last three years, every year they've said they expect margins to expand and it never happens.

that would be enough to maybe kill the idea depending on other circumstances. So I think that's another cool example of it helped I think where AI can be really useful is surfacing patterns that historically were pretty painstaking to put together and now can happen far more easily.

C

Yeah, one hundred percent. It's a v very valid point too on the credibility. I think there's a trust level and once investors show they don't really trust a management team or a stock, it could be a free fall and you see that quite frequently.

B

Yeah, and it can be really subtle too. There are times where if you just look on the Bloomberg screen, it says management beats guidance every quarter and then you actually dig into it, you realize when they give four year guidance in the Q one call.

they end up revising it down quarter after quarter after quarter. So by the time they get to Q four, sure, they're gonna beat their numbers. But if you're building your model looking out a year, you probably should shade whatever management's saying at that point, subtly.

C

Yeah, there's effectiveness in just kitchen sinking and taking all the pain in one quarter and that can be better than just consistently revising down. So there's all types of things that to your point feel they're definitely quantitative in nature, but the ability to extract that information was not always easy. And I think that's an entire category where AI is showing up. The last section is in that sourcing of new ideas.

Using AI for Investment Idea Generation

And maybe something you alluded to before where you have certain frameworks or characteristics of businesses that you really like. And it's not always easy to find all the companies out there that fit that or to know if you're looking at a company that might be fitting that. So where in the idea sourcing, idea generation are you seeing

AI be most effective. And I will just mention to me personally, it feels like there's always a step before you get to sourcing. You're usually introduced to a name for other reasons, but I'm curious where you're seeing this coming into play.

B

There's two main ways I'm seeing investors use AI for this type of use case. One is understanding companies that are exposed to a given trend or a given development.

So for instance, when tariffs were first announced, I guess April of last year, there was a lot of investors, especially on Portrait, trying to figure out which companies are gonna be most exposed and maybe the second order thinking is even trickier, which companies, for instance, have a predominantly US based supply chain and their competitors have an international supply chain, like that type of analysis.

And AI is really useful for that, even out of the box. There's just a lot of world knowledge built into a modern state of the art. model about first and second order effects and who might be affected and so forth. The more nuanced and I'd say difficult case and where we spend a lot of time has been working with firms to

Find ideas that fit like a nuanced definition of a mental model. So for instance, in my past life, I had this mental model of companies that were previously high performing and there were really attractive elements of the business model that you could believe in and underwrite. And there was some sort of hiccup. It could be something macro related or really bad product cycle, some sort of execution mess up.

And the market is now revaluing the franchise value of that business. And a lot of times I could isolate the research to that one specific headwind and figure out if it's temporary or not. And if it was temporary, then you could buy it. That's like a pretty nuanced. thing to find. It's really qualitative in nature because the headline numbers are going to look bad whether it's temporary or not in the near term. And so finding ideas that fit the mold of that is challenging for two reasons. One is

That just requires a lot of qualitative reasoning along with the figures to do that. But two, I'd say even enunciating that clearly is hard. When I ask a lot of investors, what are the mental models that you think about when finding a really attractive idea? Some of them have been very explicit about writing down, I'm looking for attribute A, B, and C.

But a lot of times it's more of a gestalt feeling, right? You know, when you see it. And actually figuring out how do you translate that into a query for lack of a better term is challenging. And that's one area where we spend a lot of time at Portrait as well is helping folk.

look at their history and figuring out like, okay, what does a 10 out of 10 idea look like for you given your historical trading context and firm and all of that? So I think the combination of those two things is difficult, but when it works, it's an incredible

They're few feelings as exciting in this business as getting served up a pitch where you're reading it, you're like, oh my God, yeah, this is really exciting. Clear my calendar. I'm gonna spend the next week just figuring this out. That's a really magical moment and something AI has been really helpful with.

C

Yeah, I could tell you from doing the show, I get to experience all these different businesses and I'm trying to draw the patterns and match them up. And some of them come more simply than others. The mission critical part of the supply chain that

a small percentage of the overall cost, but they're a dominant player. They have all the market share, pricing power should be there. That's one that's emerged and I think gained a lot of appreciation over time. And then there are the more subtle ones like what you described before, which

really articulating them can be more challenging. But that is an interesting approach and actually something that I've seen a few investors who have a differentiated framework try to lean into finding ways to take that proprietary knowledge or pattern recognition that they developed over time and get this tool that can get the most out of it in terms of uncovering new opportunities.

I think one of the things that you've made clear to me and other appearances that you made, and as we've talked, is there are skill sets.

How to Write Good Prompts for LLMs

To get the most out of AI. And I don't think they're always that obvious, or at least they weren't obvious to me. But I was hoping we could just talk through some of the things that. seem to make a massive difference in terms of whether investors getting the Google search. better quality output versus really building on what they're doing, becoming massively more efficient. And the first is just the prompt writing. So how would you articulate how important prompt writing is?

Some of the basic things that you think benefit investors from thinking about when going about prompts and just feel free to wax poetic on it.

B

It's an interesting question in that the answer changes every three months in terms of what makes an effective prompt because the model capabilities change as well as the tools available to the models are constantly changing. In the investment research context, and I think this is true for other contexts as well, but particularly research based workflows with large language models, the mental model that I think still works really well is imagine you are writing an email.

to somebody maybe overseas who's gonna work overnight and is gonna be doing a task for you. And assume they're smart, but maybe lack a lot of context on you. What information would you want them to have to be able to do a good job? Then whatever you end up writing is probably a pretty good starting point for an effective prompt.

And to dig a little bit deeper, some of the subcomponents, when I write prompts today, I usually outline a specific task and why the task is happening in the same way where if you give an analyst build this cost curve, they say, okay.

If you say, hey, build this call stuff because I think it might be shifting and that could imply something about future changes in the pricing, that's helpful. So I usually provide some background context, I provide a task. Depending on the nature of what I'm looking for, I might specify an output.

I might say, hey, I want your output to look like this. Have an intro paragraph, then outline these data points, show me a table that has X, Y, and Z. That sometimes is useful. Sometimes that can be hurtful in that you're constraining the model's output. The same way with a human analyst.

You might want to give them some slack on the rope to be able to be creative and how they want to synthesize the information. And then I usually have a section of my prompts where I have specific task guidelines and then also just general domain knowledge. So for instance, in terms of guidelines, these are things that I would just think about. Hey, maybe I'm doing that guidance prompt. Make sure you capture.

soft guidance that doesn't have a specific figure on it. Cause otherwise if you say just capture all guidance, it's just gonna capture specific quantitative guidance. And in terms of transferring domain knowledge, I have like a set of bullet points that I try to encapsulate. Here's how I think about being an analyst and some of the things you might learn through your experience.

One of those is I always remind the model, like, look, you're gonna largely be reading commentary from management teams. They are always biased positively. It's important you take a skeptical eye to anything they're saying. And even something that simple really helps.

Because the models have been trained to want to be helpful. And that usually influences them to be more positive than they otherwise should be. So just little things like that where you can imagine taking an eager college student who's excited and believes in the good. And trying to impart a little wisdom. Hey, actually, management teams tend to inflate things a bit. Just to wrap that up, defining the task, the context behind the task.

some specific outputs if I want it, some guidelines and some kind of domain context, I think is a really good starting point. And if you can send that to a a human and they read it and understand the task, you're probably going to do pretty well with the model as well.

Structured vs. Creative Tasks

C

Yeah. The level of instruction, if you almost humanize it, that was one of the initial things that I think you mentioned. months and months ago that made a material change for me. And then on the idea of context and where we'll use an LLM. assume off the shelf L L M for this question, but when you're thinking about when to just give them massive freedom versus when to load in documents How does that change the scale and the output quality in your experience?

B

So I think it's very task specific. There's a few different dimensions of which it depends upon. One is is the nature of the task something defined and structured and maybe more quantitative, in which case, yeah, it can be a pain if you're using an off-the-shelf solution, but upload the documents. Cause otherwise, generally speaking, an off-the-shelf model is gonna have to

Go crawl the web. And that's an imperfect, inefficient way to gather a bunch of structured information and you'll most likely get hallucinations, which is why, of course, companies like Portrait exist because all that stuff's preloaded. But to the extent that the goal is

something more exploratory or creative, it can still be really useful to upload the relevant documents, but I would explicitly instruct the model, hey, I don't really know which way this answer is going to go. So pull on threads, follow it, chase down leads. And in that case, you can steer the model, especially modern ones that are quite responsive, to do a lot of web searching. Where this matters as well is.

how important the task is with respect to accuracy to usefulness. There are certain tasks where if you're 99% accurate, you're zero percent useful. And that's like self-driving cars, where the cost of an error is exceptionally low. So I think of model building, like if I presented a model that had an obvious flaw in it to my boss in any of my past roles, that would be pretty bad. I certainly did that on occasion.

Yeah. Maybe you're just doing an industry survey. What's the history of this industry and how has it evolved to these three players? If there's a factual error here or there and it's earlier in the research process and it's more of a triage exercise, that's okay. I'm fine with having some calibration depending on where the task is in the process, the importance of the accuracy and how structured you want the output to be, I would vary in experiment.

The Value of Experimentation & Real-Time Feedback

C

Yeah. It's quite interesting to see where things matter based on precision versus being directionally right is where all the value comes from. On your point on experimentation, I'm curious how much this matters. And this is something that I mentally struggle with sometimes as well. You mentioned the technology is constantly changing. So by experimenting or getting better with

certain skill sets that eventually the models are going to be able to do on their own. How much difference do you see in investors or yourself when you experiment with different things in creative ways? when you know that there's a standard, decent quality way that you can solve it already. I know it's a broad question just about experimentation as a skill set. But I am curious if you've noticed any patterns with the clients that you work with or you yourself and some of your teammates.

B

In an evolving technology like this, I think spending some fifteen percent of your time on experimentation is really important because Unlike past technologies. One, the underlying models are changing, but two, the capability frontier is evolving as well and is quite jagged. And there are a lot of undiscovered, for lack of a better term, capabilities that you can pick up on well before others do.

And so having a process by which folks experiment and really spending some time pushing the models can be one helpful in a certain way, but two gives you a better intuition of where the models are today and maybe where they're going. So to add a little specificity to that. I and we at Portrait have plenty of tasks like this. And even me personally, I have some tasks where I know the models can't do this today.

Or maybe they can do a small percentage of the time. But anytime there's a new model that comes out, I run my suite of 10 different exercises across it. And I one can get a feel for how much of a leap this model is, but two, I might now need to update where I think the edge of that capability frontier is.

And a lot of times once you start doing that process, it kind of builds on itself where you might have an experiment of maybe it is building a cost curve and how that's shifted. That's a type of task that can take a long time and be challenging.

And now the model can start doing it. Well, great. Now you start having ideas of well, I wonder if I layer in a long term cost curve versus a marginal cost curve. And there are different things you can start coming up with for ways you could actually apply it within the research process today.

So I would say for anybody, certainly companies do this, but individual investors have a suite of ten different things where if the models could do it would be great and just constantly run these against the model. And once they aren't able to do it. Save that template and reuse it as part of your research process when working on a name. So I think those are some of the things that where experimentation can pay a lot of dividends.

C

I would buck at this an experimentation, but one of the things that you laid out when you talked to Brett from Fundamental Edge was just writing prompts in different forms. So start out with the most simple prompt that you know is probably gonna give you a very broad, mediocre answer, then take it layer down in more detail and specify more detail, and then see how the responses change based on how you're instructing it via prompt.

I did it a bit and it made me appreciate how it's sensitive to maybe the ordering of questions or if if I specify the ordering of questions and when I'm loading into the context window and it doesn't just impact that single exercise. It makes you appreciate This is a technology that really doesn't have many buttons that tell you, Oh, this is four wheel drive that I use in this condition. It is just a chat and you can have to play around with it yourself.

B

And just on that, I think one thing that's really cool that I mentioned earlier with the way I write prompts is I think about sending an email to someone overseas, the difference with an LLM is that the response is, for all intents and purposes, instant. The real insight, I think, comes from doing exactly what you just described, which is iterate on this. The cost of sending a single query is is trivial.

Start simple, start adding complexity as it's helpful. It's really like a two way dance to end up in a spot where the AI really is acting like a useful analyst and You don't need to wait overnight to get the content to then see how I get feedback. You can give that feedback real time and adjust things. And that becomes super, super powerful.

C

your prompts are loaded into portrait for whether it's uh primers or whatnot and you see the layer the levels of detail that go into those. And it's actually quite helpful. It made me feel a little down on myself in terms of what I was prompting prior to seeing those. But I think, yeah, it definitely reigns true. Taking it up a level

Best Practices for Deploying AI in an Institutional Setting

I have seen quite a few individuals who are getting a lot of value out of AI, but it almost feels like it's siloed. You could have two individuals at the same fund that use entirely different tools. And they're getting value. Obviously, the enterprise or the fund is benefiting from all of it, but it doesn't necessarily feel

strategically aligned or there's too much collaboration in regards to it. Have you noticed anything in terms of best practices for funds just to get some type of adoption that's aligned or collaborative?

B

Because investing is ultimately a lot of the research is building conviction and people build that in different ways. It's challenging to mandate changes in a research process because if I was at my last job and the head of the firm said, Okay, now you must use only cell side models for your models, I would really struggle. I can certainly sympathize with

I know a lot of firms have have been trying top down to say, hey, we don't want to fall behind on AI. You need to start using such and such tool. And I think that can sometimes work in certain ways, but in other ways it can backfire and people push back. What I've seen the most Adoption and firms that are taking the most advantage of this, I'd say is finding the right balance between having firm-wide initiatives.

that don't force people to change their behavior while letting individuals experiment and figure out where AI is gonna be additive and reducing friction without negatively impacting conviction. So for instance, just using Portugal as an example, there are firms where at the firm level, we spend a lot of time building idea generation. screens that are the super bespoke to them and their process. And we just run those like an outsourced analyst.

And that's really value add because all we're doing is we're not asking anyone to change the process. They just get a bunch of really interesting pitches that now they can spend time on if they want. And similarly, the thesis monitoring has become a really big thing. And the nice thing about thesis monitoring is it doesn't require a change in process.

All we're doing is we take a ticker and maybe your investment thesis and then we're just pulling in data points that at least our system believes could be really helpful and incremental. to your thinking. And again, that doesn't require anything new. It just requires looking at a new insight when it arrives and ignoring it if it's not relevant. Whereas I think for the actual day to day asking queries, building output,

It really just depends on the user. And obviously working with a firm that specializes in this and has built software specifically for this vertical can be really helpful for driving adoption because unlike a general tool like ChatGPT. someone building software in their space can customize the UX around the specific workflows. But ultimately I think that adoption has to happen bottoms up and people need to feel comfortable with it.

to be able to continue making high quality investment decisions. And to the extent people have changed their process, and many people have, they've done so because they feel trust in it. And I think building that trust has to happen on the individual level.

C

Yeah, it's actually an interesting perspective that I somehow hadn't thought about. But when it comes to any type of investment research and investment process. People do it all differently within the same font. They might have the same core things that they look for, but the path to get there is quite different and even model formats.

I can remember many of arguments over having a single specified template and people moving off of that. So yeah, I think there's a lot to that. And maybe that opened my eyes a little bit. On the

Why Documenting Decision Making is Essential

practices that firms or individuals can do in terms of investing in things that maybe aren't obvious in value today. But as the technology evolves, maybe they'll be able to leverage more. And I mentioned before we talked about there's the famous example of the sports analytics teams that had been tracking data well before sports analytics were a thing and they were only able to really benefit from all those data.

practices decades into the future, but the teams that never have kept up with it, it's really hard to go back in time. So are there exercises or practices that you think funds or individuals can start to implement that will pay off in time?

B

Obviously everything we spoke about earlier with experimentation certainly applies here, but I think one thing that is going to be really important and already was historically, but I think carries a lot more weight going forward, is just the importance of documentation of thinking and decisions and research.

Essentially, these models, the usefulness of them rises exponentially with the amount of context they're given. As the model context length expands and as their agentic reasoning gets more heavily utilized to do longer running tasks and they ultimately shift from being a helpful research tool to something that lives within a firm and can execute a research process. Their ability to do it is going to be a function of how much data they have on how you and your firm operate.

So I think firms where they have documented memos and even write up short paragraphs on why they made a trading decision, it's hard to know ex ante which data is going to be used and how. But I think it's a pretty reasonable bet that having that data at a minimum is helpful for the humans, but will certainly be helpful for the machines because I think a moment ago we're speaking about adoption challenging is loading in that context.

It's the same challenge you goes when you hire a junior analyst, needing to teach them everything about how you think and your past trading history with a given name and all that information just takes a ton of time to convey. And so to the extent that that data can be captured in real time, that forms an enormously valuable source of IP that will make AI unique to you.

And again, it's a little hard to know up front how that's going to be used. And it probably takes some creativity to figure it out, but hard to imagine two or three years from now that every piece of data isn't being used within a model that is operating within the context of a fund.

C

It's a perfect example. I could speak to on ramping at places where all they had were the memos, which were the buy and sell decisions for positions. Which are valuable in some ways, but they're usually overly polished versus those that after earnings they would have their quick blurb.

Maybe Amazon entered the space and it was something to monitor. And then a month later they had a meeting with the management team and it made them feel a little worse off. And then another quarter comes out and it's clear Amazon's impacting them. And the memo might just say Amazon's movement, but to see the thought process and how the team makes decisions. There was

something very valuable in me getting up to speed. Again, I think it goes back to that lens of almost thinking of it like a team member in the same way that you would give them the most ammo to align with how you work and what you do.

How Models Have Improved at Using Provided Context

Going back to some of the quick hitting questions, thinking about down the road and where we're going, I already brought up context windows. We'll use LLMs again, just thinking about some of the most used tools. When you think about where investors or where you personally find the context window advantage has the most impact or

How you adjust for the challenge that I think many investors face, which is I want to get as much information in there as possible, but sometimes I'm limited by it. How do you approach that? And how much do you think you should worry about that as an investor?

B

It's super dependent on what tool you're using and in what context you're using it in the task. to go a little deeper on that. The context windows obviously have been expanding, although really in the past year they've been flat with Gemini at around a million, GPT, I think around four hundred thousand, and Opus or Claude around two hundred. Where the models have gotten more capable is using that context well. I think historically models struggled if you load it in

I don't know, the last five, ten Ks and let's say a ten K is like a hundred thousand tokens, so five hundred thousand. The models historically be really good at say if you ask, say, what was S G and A in twenty twenty three? They call that a needle in a haystack problem. They would struggle if you said, build a three-statement model using these.

the model needs to apply uh to tension across lots of different pieces of the context window simultaneously. I think that still is a roughly helpful rule of thumb, which is The models are generally, if you're using a ton of context, we need to do better if you're loading that in because you're looking for very specific data points. That being said, I think a change versus if we had this conversation three months ago is models have gotten very good at using the context intelligently.

If you're thinking about building a project in Portrait or Notebook LM or something like that, I wouldn't really hold back. I would give it pretty much the entire corpus. And you're seeing this a lot in software engineering where clawed code and other agency coding tools are very capably exploring super long context.

in a way that is efficient and focused on the task at hand. So in the same way, I wouldn't give a junior analyst a task and say, these are the only four documents you can use. You'd want them to ultimately have the agency to choose what content they're going to use. And I think that holds true. All that being said, if you're going to use the raw model itself, you do not want to overload the context window and use 70% of the context window. You will see a degradation in any complex task.

In that case, you should break it down. But hopefully the much easier way to do this is use a tool that manages that complexity already. So you don't need to worry about, okay, what percent of the context window am I utilizing and so forth is how I think about that today.

Memory in LLMs: Near-Term Limitations, Long-Term Potential

C

On memory, I think it's obvious where as memory has now been implemented in a lot of these tools, it customizes to you in that iteration, again, I'm gonna overuse the lens of the employee oversees that you're creating a better, more aligned working environment. Are there any non obvious impacts that you think memory will have?

B

I think the interesting thing about memory is Today as it's implemented, it's far worse than than a human's memory. It is super limited. It's expensive in the sense of having deep memories takes up a lot of context.

it lacks a lot of the nuance of our memories are really useful, both for specific facts, but more so for the abstract concepts that they create and that we learn. When you think of those memories, like a lot of times that that connection is where they become really powerful and that's become pattern recognition for investment.

So in the near term, I think memories is a useful tool for all the obvious reasons that you can think about in terms of just adding more context that persists in the context window. Longer term, everyone knows all the fallibility of human memory. And this is a little bit above my pay grade in terms of the actual ML research and how memory is being incorporated in different ways.

But you could imagine any one of the architectures that are being experimented today with memory and having short term memory and longer term memory and the ability to look up memories, if implemented correctly, both in terms of how the models are trained as well as the hardnesses around them. Imagine having a model that has lived every single one of a firm's investments and maybe things they even passed on. There's a world where it becomes so much more powerful than any given human.

I look at some of the investors I've worked for and they just have this insanely eerily intuitive judgment about things. A lot of that is not just memory specific facts, but memory that comes from an experience and scar tissue. There's a world where That certainly could exist in models. And I think in some sense, people talk about AGI being the ability to be generally intelligent. I think a really interesting test for a model if it's AGI is can it predict the future?

You need a really good world model and understanding of lots of variables to predict the future. And that's what an investor does. And memory is clearly a really important ingredient to that. So I'd say near term, I'm a little bearish on just how big of an impact it's gonna be relative to just copy pasting that context into a window. But longer term, I think that's the unlock where this goes from being the useful tool to the real core driver of a lot of the research a firm is doing.

C

Well I'll I'll separate near term I'm almost taking away that it might be more valuable to limit the memory, therefore reducing the context window and investing more into the prompt that I'm using. I know they're not like for like, but ultimately memory is customizing to you. And if you could just put that in the prompt, that might be more beneficial than sucking up memory.

B

It's a near term convenience. So in other words, if there's knowledge you want to impart and every single time you interact with the model, yeah, that should live in memory. Or if there are aspects of working on a name and you've maybe done 20 queries on a particular name.

You maybe want some sense of the memory in living in the model. So if you reference something back or it already knows it's did XYZ analysis, it can do that. I think that's where it's useful. It's a bit of a shortcut for just loading in context that you don't have to repeat. I definitely would recommend spending a lot of time being thoughtful around to the extent you can manage the system prompt in off the shelf tool or anything like that. I think we're just very early innings.

from a technical perspective, tapping what it means to really utilize memory in a thoughtful way. And just to illustrate that. And these models have such incredible world knowledge by essentially reading the entire internet in their pre-training. So they clearly have the capability to store in their weights a lot of knowledge and then learn important concepts.

from that knowledge that leads to things like reasoning. So it's just a matter of time before that same paradigm applies to continual learning on the job. And that to me is gonna be really, really fun.

C

Yeah. I do think about if Bridgewater's really been recording all those meetings for all of this time, what all of that that input could do eventually And what could happen with that technology. I don't even have an idea of what it could be, but it just feels pretty unique.

And seeing how people are testing this even with their own personal recordings and creating their own things. The last thing I wanted to talk about, because it gets brought up, it's a buzzword. It has very practical applications, but

Applying Agentic AI in Investment Research

agentic AI. The definition gets a little bit fuzzy sometimes, but just thinking about agenc AI as a topic and as it applies to investors. If you could just talk about where that's going, how you would even categorize that within the various tools, and anything else you would mention on that.

B

So there's lots of definitions of agentic AI. I think of it essentially as the model has the capability to reason, reflect, take action, reflect on those actions ultimately in the pursuit of a goal. And there's obviously different levels of scaffolding and constriction around what the model can and can't do. What's really exciting, especially now, is

Agents have started to work. And interestingly, when we first started Portrait, we tried and we did actually have an agent that was running using GBT4. And it was

so hard to get to work. And the way you got it to work was you had to be so prescriptive about what the model could and couldn't do. I think our system prompt was thirty thousand tokens or something crazy. You really had to limit it because at any given time the model if it started going down the wrong direction and it didn't have the ability to self-correct and be thoughtful about what it's learned and how it should adjust its plan.

And that required just a ton of prompting. What's cool today, and I think what what's happening in software engineering is the leading edge of what's gonna eventually happen in investment research and in really any other knowledge field is the models have now become smart enough To do longer running tasks through using tools and updating its thinking process as it goes. And you're seeing this with things like Claude Code and Codec.

where people are leaving these models alone for multiple hours and it is dynamically operating like a software engineer to understand a code base, make changes to it, test things out, see an error, go look at the trace and figure out why there's a problem, make a fix. That type of thinking is certainly really challenging and that now works. The nice thing about code is it's the perfect.

environment for these models to figure this type of capability out because one, it's all just text files at the end of the day. It's all contained usually within a repo. So all the context is locally available and it's super verifiable. The code either runs or it doesn't. Investment research is obviously much different. None of those things are true. The context is in many different forms and very broad and some is more accessible than others. It's also much more qualitative in terms of output.

But the form factor of can these models reason iteratively and arrive at complex answers that require on the go changes of plans? The answer is yes. And it's just a matter of the engineering work to get those models into the right context so that they can operate like a junior analyst and ultimately like a senior analyst doing long running research. It's definitely still a buzzword, but there's a lot more meat behind that buzzword than there was maybe a year ago.

C

The point on code and I often bring up chess, these constrained environments that are still very complex. but they're constrained by certain things, you tend to see more of a clean iteration cycle and more productivity that comes out of it and the more complex and adaptive the system.

the harder it is to control the evolution. So it's one that I think will be interesting to see how it all evolves. But this has been fascinating. Thank you for going on some more philosophical tangents in addition to the very tangible use cases. It's been a pleasure. So thank you for sharing the knowledge with us.

B

Thank you. It's been really fun.

D

To find more episodes of breakdowns ranging from Costco to Visa to Moderna, or to sign up for our weekly summary, check out Join Collaborative. C O L O S.

🎵 Music

This transcript was generated by Metacast using AI and may contain inaccuracies. Learn more about transcripts.
For the best experience, listen in Metacast app for iOS or Android