Today I am delighted to host Desi Ivanova, a distinguished research fellow in machine learning at the University of Oxford. Desi's fascinating journey in statistics has spanned from quantitatifiance to the frontiers of Bayesian experimental design, or BED, BED. In our conversation, Desi dives into the deep world of BED where she has made significant contributions.
She begins by elucidating the core principles of experimental design, discussing both the theoretical underpinnings and the complex computational challenges that arise in its application. Desi shares insights into the innovative solutions she's developed to make BED more practical and applicable in real-world scenarios, particularly highlighting its impact in sectors like healthcare and technology.
Throughout the discussion, Desi also touches on the exciting future of BED, especially in light of recent advancements in AI and machine learning. She reflects on the critical role of real-time decision-making in today's data-driven landscape and how patient methods can enhance the speed and accuracy of such decisions. This is Learning Vision Statistics, episode 117.
recorded September 26, 2024. Welcome to Learning Bayesian Statistics, a podcast about Bayesian inference, the methods, the projects and the people who make it possible. I'm your host. Alex and Dora. You can follow me on Twitter at Alex underscore and Dora like the country for any info about the show. Learnbasedats.com is Laplace to be. Show notes, becoming a corporate sponsor, unlocking Beijing Merch, supporting the show on Patreon. Everything is in there. That's Learnbasedats.com.
If you're interested in one-on-one mentorship, online courses or statistical consulting, feel free to reach out and book a call at topmate.io slash Alex underscore and Dora. See you around, folks, and best patient wishes to you all. And if today's discussion sparked ideas for your business, well, our team at PMC Labs can help bring them to life. Check us out at PMC-labs.com. Hello my dear vegans, today I want to welcome our two new patrons in the full posterior tier.
Thank you so much Ivy Hwing and Garrett Clark, your support literally makes this show possible. I am looking forward to interacting with you guys in the LBS Slack channel. Now, before we start the episode, I have a short story for you guys. A few years ago, I started learning machine stats by watching all the tutorials I could find that a teacher I really liked was teaching. That teacher was no other than Chris Fonsbeck, PMC's creator and BDFL.
And five years down the road, senior unpredictable road, am beyond excited to share that I will now be teaching a tutorial alongside Chris. That will happen at Pi Data New York from November 6 to 8, 2024. And I would be delighted to see you there. We will be teaching you everything you need to know to master Gaussian processes with IMC. And of course, I will record a few live LBS episodes while I'm there. But I'll tell you more about that in the next episode.
In the meantime, you can get your ticket at pine.at.org slash NYC 2024. I can't wait to see you there. Okay, on to the show now. Desi Ivanova, welcome to Learning Bayesian Statistics. Thank you for having me, Alex. Pleased to be here. Yeah, yeah. Thanks a lot for taking the time, for being on the show. a to Marvin Schmidt for putting us in contact. He was kind enough to do that on the base flow stack where we interact from time to time.
Today, though, we're not going to talk a lot about advertised Bayesian inference. We're going to talk mostly about experimental design, Beijing experimental design. So BED or BED, I like the acronym. But before that, as usual, we'll start with your origin story, Daisy. Can you tell us what you're doing nowadays and also how you ended up working on what you're working today? Yeah, of course.
So broadly speaking, I work in probabilistic machine learning research, where I've worked on a few different things, actually. So the audience here would be mostly familiar with Bayesian inference. So I've worked on approximate inference methods, namely, know, variational inference. You mentioned Marvin, right? So we've actually collaborated with him on some amortized inference work. I've also done some work in causality.
But my main research focus so far has been in an area called Bayesian experimental design, as you correctly pointed out, BED for short, a nice acronym. So BED, Bayesian experimental design was the topic of my PhD. And yeah, will be the topic of this podcast episode. Yeah, really, really keen on discussing. and very, very close to my heart. You know, how I ended up here. That's actually a bit quite random.
So before, before getting into research, right, so before my PhD, I actually worked in finance for quite a few years as a, as a quantitative researcher. At some point, I really started missing sort of the rigor in a sense of, you know, conducting research, you know, being quite principled about, you know, how we measure uncertainty, how we quantify robustness of our models and of the systems that we're building. And right at the height of COVID, I decided to start my PhD back in 2020.
And Indeed, the area, right, based on experimental design, that was originally not the topic of my PhD. I was supposed to work on certain aspects of variational autoencoders. If you're familiar with these types of models, they're not as popular anymore, right? So if I had ended up working on variational autoencoders, I guess a lot of my research would have been, I mean, not wasted, but not as relevant. not as relevant today as it was, you know, four or five years ago.
And how I ended up working with Bayesian experimental design specifically, basically, approached my supervisor a few months before starting my PhD and I said, Hey, can I can I read about something interesting to prepare for a PhD? And he was like, yeah, just with these papers on Bayesian experimental design. And that's how it happened. Really? Yeah. Okay, cool. Yeah, I love these.
I love asking this question because often, you know, with hindsight bias, when you're a beginner, you like it's easy to trick yourself into thinking that people like you who are experts on a on a particular topic and know that topic really well, because they did a PhD on on it, like they They have been doing that since they were, I don't know, 18 or even 15 or it was like all, all planned and part of being big plan. But most of the time when you ask people, it was not at all.
And it's the result of experimenting with things and also the result of different people they have met and, and, and encounters and mentors. And so I think this is also very valuable to, to tell that to beginners because otherwise they can be very daunting. 100 % Yeah, I would 100 % agree with that. And actually experimenting is good. You know, again, we'll be talking about experimental design, I think.
Yeah, many times, you know, just by virtue of trying something new, you discover, you know, I actually quite liked that. And it actually works better, you know, for whatever purpose it might be, it might be your commute to work, right? There was this very interesting research. You know, when there is like a tube closure, right, if the metro is getting closed, you know, some people, like 5 % of people actually discover an alternative route that actually is much better for the daily commute.
But they wouldn't have done that had the closure not happened. So almost like being forced to experiment may lead to actually better outcomes, right? So it's quite interesting.
Yeah, yeah, no. mean, completely agree with that and that's also something I tell to a lot of people who reach out to me, know, wondering how they could start working on Bayesian stats and often I'm like, you know, trying to find something you are curious about, interested in and then start from there because it's gonna be hard stuff and there are gonna be a lot of obstacles.
So if you're not, you know, really curious about what you are studying, it's going to be fairly hard to maintain the level of work that you have to maintain to, to in the end enjoy what you're doing. So experimenting is very important. I completely agree. and actually do you remember yourself? so I'm curious first how Bajan is your work. And also if you remember when you were, were introduced to, to Bajan stance. When was I introduced to Bayesian stats?
That must have been probably in my undergrad days. I remember I took some courses on kind of Bayesian data analysis, but then I didn't do any of that during my time in industry. Yeah. And again, as I said, I ended up working on Bayesian experimental design a little bit, a little bit randomly. The work itself is, you know, it does use Bayesian principles quite a lot. You know, we do Bayesian inference, we do, we start with a Bayesian model, right? So the modeling aspect is also quite important.
You know, it's very important to have a good Bayesian model for all of these things to actually make sense and work well in practice. So I would say overall, the work is quite, quite Bayesian, right? Yeah. Yeah. Yeah. Yeah, for sure. so actually, I think that's a good segue to introduce now, Bayesian experimental design. So it's the first time we not talk, not the first time we talk about it on the show, but it's a really dedicated episode about about that.
So could you introduce the topic to our listeners and basically explain and define what Bayesian experimental design is? Yeah, of course. So can I actually take a step back and talk a little bit about experimental design first? Yeah. yeah. And then we'll add the Bayesian kind of the Bayesian aspect to it. So, you know, when, when I say, I work on Bayesian experimental design, most people immediately think lab experiments, right?
For example, you're in a chemistry lab and you're trying to synthesize a new drug or a new compound or something. But actually, you know, the field of experimental design is much, broader than that, right? And to, you know, give a few concrete examples, you can think about surveys, right? You may need to decide what questions to ask.
Maybe you want to tailor your questions as, you know, the survey progresses so that, you know, you're asking very tailored, customized questions to each of your survey participants. You can think of clinical trials, right? So how do you dose drugs appropriately? Or, you know, when should you test for certain properties of these drugs, things like absorption and so on? So all of these things can be, you know, cast as a as an experimental design problem, as an optimal experimental design problem.
So in my mind, designing experiments really boils down to optimal or at least intelligent data gathering. Does that make sense? So we're trying to kind of optimally collect data in order to kind of learn about the thing that we want to learn about. So some underlying quantity of interest, right? And the Bayesian framework, so the Bayesian experimental design framework specifically takes an information theoretic approach to what intelligent or optimal means in this context.
So as I already mentioned, it is a is a model based approach, right? So we start with an underlying Bayesian model that actually describes or simulates the outcome of our experiment. And then the optimality part, right? So the optimal design will be the one that maximizes the amount of information about the thing that we're trying to learn about. Yeah. That makes sense? can actually give a concrete example. Maybe that will make it easier for you and for the listeners, right?
So if you think about, you know, the survey, the survey example, right? kind of a simple but I think very easy to understand concept is you know trying to learn let's say about time value of money preferences of different people right yeah so what does that mean imagine your a behavioral economist, right? And you're trying to understand some risk preferences, let's say, of people.
Generally, the way that you do that is by asking people a series of questions of the form, do you prefer some money now or you prefer some money later? Right? So do you prefer 50 pounds now or you prefer 100 pounds in one year? Right. And then you can choose, are you going to propose 50 pounds or 60 pounds or 100 pounds now? how much money you're going to propose in what time, right? So you're going to do a hundred pounds in one month or in three months or in one year, right?
So there is like a few choices that you can make. And there is a strong incentive to do that with as few questions as possible because you end up paying actually the money to the participants, right? So basically, we can start with an underlying Bayesian model that sort of models this type of preferences of different human participants in this survey. There's plenty of such models from psychology, from behavioral economics.
And at the end of the day, what we want to learn is a few parameters, right? You can think about this model almost like a mechanistic model that explains how preferences relate to things like, you know, are described by things like a discount factor or sensitivity to various other things. And by asking these series of questions, we're learning about these underlying parameters in our Bayesian model. Did that make sense? Yeah. Yeah. I understand better now.
And so I'm wondering, it sounds a lot like, you know, just doing also causal modeling, right? So you write your causal graph and then based on that, you can have a generative model and then, and fitting the model to data is just one part, but it's not what you start with to write the model. How is that related? Right.
The fields are, in a sense, closely related in the sense that, you know, in order for you to uncover kind of the true underlying causal graph, let's say if, you know, you start with some assumptions, you don't know if X causes Y or Y causes X or, you know, or something else, the way that you need to do this is by intervening in the system. Right.
So You can only, in a sense, have causal conclusions if you have rich enough data and by rich enough data we generally mean experimental or interventional data, right? So you're totally right in kind of drawing parallels in this, right? And indeed we may... design experiments that actually maximize information about the underlying causal graph, right?
So if you don't know the graph and you want to uncover the graph, you can set up a Bayesian experimental design framework that will allow you to, you know, select, let's say, which nodes in my causal graph should I intervene on, with what value should I intervene on, so that with as few experiments as possible, with as few interventions as possible, can I actually uncover the true, the ground truth, right? The true underlying causal graph, right?
And, you know, kind of the main thing that you're optimizing for is this notion of information content. So how much information is each intervention, each experiment actually bringing us, right? And... And I think that's part of the reason why I find the Bayesian framework quite appealing as opposed to, I guess, non-Bayesian frameworks. You know, it really centers around this notion of information gathering.
And with the Bayesian model, we have a very precise definition of or a precise way to measure an information content of an experiment. Right. If you think about Imagine again, we're trying to learn some parameters in a model, right? The natural, again, once we have the Bayesian model, the natural way to define information content of an experiment is to look at, you know, what is our uncertainty about these parameters under our prior, right? So we start with a prior.
We have uncertainty that is embedded or included in our prior beliefs. We're going to perform an experiment to collect some data, right? So perform an experiment, collect some data. we can update our prior to a posterior. So that's classic Bayesian inference, right? And now we can compare the uncertainty of that posterior to the uncertainty of our prior. And the larger the drop, the better our experiment is, the more informative our experiment is.
And so the best... or the optimal experiment in this framework would be the one that maximizes this information gain. So the reduction in entropy, we're going to use entropy as a measure of uncertainty in this framework. So it is the experiment that reduces our entropy the most. Does that make sense? Yeah. Total sense. Yeah. Total sense. That's amazing. I didn't know. So yeah, I mean, that's, that's pretty natural then to include the causal framework into that.
And I think that's one of the most powerful features of experimental design, because I guess most of the time what you want to do when you design an experiment is you want to intervene. on a causal graph and see actually if your graph is close to reality or not. So that's amazing. And I love the fact that you can use experimental design to validate or invalidate your causal graph. That's really amazing. Correct. 100%.
But I do want to stress that The notion of causality is not necessary for the purposes of describing what Bayesian experimental design is. I'll give you a couple of other examples, actually. So you may... You may want to do something like model calibration. Let's say you have a simulator with a few parameters that you can tweak, right? So that it, I don't know, produces the best outcomes, right? Or is optimally calibrated for the thing that you're trying to measure, right?
It is like, again, I don't think you need, you know, any concepts of causality here, right? It's you're turning a few knobs. And you know, again, you can formulate this as an experimental design problem where, you you are trying to calibrate your system with as few kind of no turns as possible. Yeah. Yeah, yeah, yeah. That makes a ton of sense. Something I'm curious about hearing you talk is, and that's also something you've worked extensively on, is the computational challenges.
Can you talk about that? What are the computational challenges associated with traditional bed, so Bayesian experimental design, and how they affect the feasibility? of bed in real world applications. Yeah. Yeah, that's that's an excellent point. Actually. I Yeah, I see you read some of of my papers. So, all right. So all of these kind of information objectives.
So what I just described, you know, we can look at the information content, we can maximize information, and so on, like, it's all very natural. And it's all very mathematically precise and beautiful. But working with those information, theoretical objectives is quite difficult in practice. And the reason for that is precisely as you say, they're extremely computationally costly to compute or to estimate, and they're even more computationally costly to optimize.
And the careful listener would have noticed that I mentioned posterior inference. Right. So in order to compute the information content of an experiment, you actually need to compute a posterior. Right. You need to compute a posterior given your data. Now, where the problem lies is that you need to do this before you have collected your data. Right. Because you designing an experiment and then only you will be performing it and then observing the outcome and then you can do.
the actual posterior update. Now, what we have to then do is look at our prior entropy minus our posterior entropy and integrate over all possible outcomes that we may observe under the selected experiment. And we have to do that for a number of experiments to actually find the optimal one. So what we end up with is this sort of nesting of expectations. So we have an expectation, we have an average with respect to all possible outcomes that we can observe.
And inside of our expectation, inside of this average, we have this nasty posterior quantity that, generally speaking, is intractable. Unless you're in a very specific case where you have a conjugate model, where your posterior is available in close form, you actually don't have access to that posterior. which means that you will need to do some form of approximation, right? Whether it's exact like MCMC or is going to be a variational posterior computation.
Again, there is many ways of doing this. The point is that for each design that you may want to try, you need to compute all of these posteriors. for every sample of your potential outcome, right? Of your possible outcome under the experiment. So what I was just describing is what is known as a doubly intractable quantity, right? So again, this podcast audience is very familiar with Bayesian inference and how Bayesian inference is intractable in general.
Now computing... the EIG, the sort of computing the objective function that we generally use in Bayesian experimental design is what is known as doubly intractable objective, which is quite difficult to work with in practice, right? Now, what this means for sort of real world applications is that you either need to throw a lot of compute on the problem. Or you need to do, you know, some, you need to sort of give up on the idea of being Bayesian optimal, right?
You may use some heuristics or something else. And what this problem really becomes limiting is when we start to think about, you know, running experiments in real time, for example. So the survey example that I started with, you know, you know, asking participants in your survey, do you prefer somebody now or somebody later?
You know, it becomes quite impractical for you to, you know, run all these posterior inferences and optimize all of these information theoretic objectives in between questions, right? So it's a little bit, you know, I asked you the first question now, let me run by MCMC. Let me optimize some doubly intractable objective. Can you just wait five minutes, please? And then I'll get back to you with the next question. Obviously, it's not something that you can realistically do in practice.
So I think, historically, the computational challenge of the objectives that we use for Bayesian experimental design has really... limited the feasibility of applying these methods in kind of real-world applications. And how, so how did you, which innovations, which work did you do on that front? That make all that better. Right. So there is a few things that I guess we can discuss here. So number one, I mentioned posterior inference, right?
And I mentioned we have to do many posterior inference approximations for every possible outcome of our experiment. Now, I think it was the episode with Marvin. right, where you talked about amortized Bayesian inference. So in the context of Bayesian experimental design, amortized Bayesian inference plays a very big role as well, right?
So one thing that we can do to sort of speed up these computations is to learn a... to learn a posterior that is amortized over all the outcomes that we can observe, all the different outcomes that we can observe, right? And the beautiful part is that we know how to do that really well, right? So we have all of these very expressive, variational families. that we can pick from and optimize with data that we simulate from our underlying Bayesian model.
So this aspect of Bayesian experimental design definitely touches on related fields of amortized Bayesian inference and simulation-based inference. So we're using simulations from our model to learn an approximate posterior. that we can very quickly draw samples from, as opposed to having to fit an HMC for every new data set that we may observe. That makes sense. Yeah. And so I will refer listeners to the episode with Marvin, episode 107, where we dive into amortized patient inference.
put that in the show notes. I also put for reference three other episodes where we mentioned, you know, experimental design. So episode 34 with Lauren Kennedy, 35 with Paul Burkner and 45 with Frank Harrell, that one. focuses more on clinical trial design. But that's going to be very interesting to people who are looking to these. And yeah, so I can definitely see how amortized patient inference here can be extremely useful based on everything you used it before.
Maybe do you have an example, especially I saw that during your PhD, You worked on policy-based patient experimental design and you've developed these methods. Maybe that will give a more concrete idea to listeners about what all of these means. Exactly. One way in which we can speed up computations is by utilizing, as I said, amortized variational inference. Now this will speed up the estimation of our information theoretic objective, but we still need to optimize it.
Now, given that we have to do after each experiment iteration, right? So we have collected our first data point, we have collected our first data point with a need to... update our model and with this new model under this new model, updated model, we need to kind of decide what to do next. Now, this is clearly also very computationally costly, right?
The optimization step of our information theoretic objective is quite computationally costly, meaning that it is very hard to do in real time, right? Again, going back to the survey example, you still can do it, right? You can estimate it a little bit more quickly, but you still can't optimize it. And this is where a lot of my PhD work has actually been focused on, right? So developing methods that will allow you to run Bayes Bayesian Optimal Design in real time.
Now, how are we going to do that? So there is a little bit of a conceptual shift in the way that we think about designing experiments, right? What we will do is rather than choosing the design, the single design that we're going to perform right now, right in this experiment iteration.
What we're going to do is learn or train a design policy that will take as an input our experimental data that we have gathered so far, and it will produce as an output the optimal design for the next experiment iteration. So our design policy is just a function, right? It's just a function that takes past experimental data as an input and produces the next design as an output. Does that make sense? Yeah, yeah, yeah. I can see what that means. How do you integrate that though?
You know, like I'm really curious concretely. Yeah. what does integrating all those methods, so a multi-spatial inference, variational inference to the Bayesian experimental design, and then you have the Bayesian model that underlies all of that. How do you do that completely? Yes, excellent. When we say the model, I generally mean the underlying Bayesian model. This is our model that we use to train our let's say, variational amortized posterior.
And this is the same model that we're going to train our design policy network. And I already said it's a design policy network, which means that we're going to be using, again, deep learning. We're going to be using neural networks to actually learn a very expressive function that will be able to take our data as an input, produce the next design as an output. Now, how we do that concretely?
There is, you know, by now a large number of architectures that we can pick that is suitable for, you know, our concrete problem that we're considering. So one very important aspect in everything that we do is that our policy, our neural network should be able to take variable size data sets as an input. Right? Because every time we're calling our policy, every time we want a new design, we will be feeding it with the data that we have gathered so far. Right?
And so it is quite important to be able to condition on or take as an input variable length sequences. Right? And so concretely, how can we do that? Well, you One kind of standard way of doing things is to basically take our experimental data that we've gathered so far and embed each data point. So we have an X for our design, a Y for our outcome. Take this pair and embed it to a fixed dimensional representation, right, in some latent space. Let's say with a small neural network, right?
And we do that for each individual design outcome pair, right? So if we have n design outcome pairs, we're gonna end up with n fixed dimensional representations after we have embedded all of this data. Now, how can we then produce the next sort of optimal design for the next experiment iteration? There is many choices, and I think it will very much depend on the application. So certain Bayesian models, certain underlying Bayesian models are what we call exchangeable, right?
So the data conditional on the parameters can be, the data conditional on the parameters is IID, right? Which means that the order in which our data points arrive doesn't matter. And again, the survey example. is quite a good example of this precisely, right? Like it doesn't really matter which question we ask first or second, you know, we can interchange them and the outcomes will be unaffected.
This is very different to time series models where, you know, if we design, if we are choosing the time points at which to take blood pressure, right, for example, If we decide to take blood pressure at t equals five, we cannot then go back and take the blood pressure at t equals two. So the choice of architecture will very much depend on, as I said, the underlying problem. And generally speaking, we have found it quite useful to explicitly embed the structure that is known.
So if we know that our model is exchangeable, we should be using an appropriate architecture, which will ensure that the order of our data doesn't matter. If we have a time series, we can use an architecture that takes into account the order of the data. So for the first one, we have... kind of standard architecture such as I don't know how familiar the audience would be, but you know, in deep learning, there is an architecture called deep sets, right?
So basically, take our fixed dimensional representations and we simply add them together. Very simple, right? Okay, we have our end design outcome pairs. We add them together, they're all of them are in in the same fixed dimensional representation, we add them together. Now this is our representation or a summary of the data that we have gathered so far. We take that and we maybe map it through another small neural network to produce the next design.
If we have a time series model, then we can, you know, pass everything through an LSTM or some form of recurrent neural network to then produce the next design. And that will keep sort of the order in and it will take the order into account. Did that answer your question in terms of like how specifically we think about these policies? Yeah. Yeah, that's fascinating.
And so basically, and we talked about that a bit with Marvin already, but the choice of neural network is very important depending on the type of data because if you have, many time series are complicated, right? Like they already are, even if you're not using a neural network, time is always complicated to work with. because there is an autocorrelation, right? So you have to be very careful. So basically that means changing the neural network you're working with.
then so concretely, like what, you know, for practitioners, someone who is listening to us or watching us on YouTube, and they want to start implementing BED in their projects, what's practical advice you would have for them to get started? Like how, why, and also when, you know, because there may be some moments, some cases where you don't really want to use BED. And also what kind of packages you're using to actually do that in your own work.
So that's a big question, I know, but like, again, repeat it as you give the answers. Yeah, yeah, yeah. Let me start with kind of... If people are looking to implement BASE in their projects, I think it is quite important to sort of recognize where BASE experimental design is applicable, right? So it can be applied whenever we can construct an appropriate model for our experiments, right?
So the modeling part, like the underlying BASE model is actually doing a lot of the heavy lifting in sort of in this framework. simply because this is basically what we're to assess quality of the designs, right? So the model is informing what a valuable information is. And so I would definitely advise not to gloss over that part. If your model is bad, if your model doesn't represent the data generating process, in reality, the results might be quite poor.
Now, I think it's also good to mention that you don't need to know the exact probability distribution of the outcomes of the experiment, right? So you can, you know, as long as you can simulate, right? So you can have a simulator-based model that simply samples outcomes of the experiment, given the experiment. which I think, you know, it simplifies things a little bit.
You know, don't have to write down exact probability distributions, but still you need to be able to sample or simulate this outcome. So that would be step number one, right? So ensuring that you have a decent model that you can start sort of experimenting with, you know, in the sense of like... designing the policies or like training the policies or sort of designing experiments.
The actual implementation aspect in terms of software, unfortunately, based on experimental design is not as well developed. from software point of view as, for example, amortized Bayesian inferences, right? So I'm sure that you spoke about the baseflow package with Marvin, which is a really amazing sort of open source effort.
They have done a great job of implementing many of the kind of standard architectures that you can basically, you know, pick whatever works or like pick something that is relatively appropriate for your problem and it will work out, right? I think that is like a super powerful, super powerful framework that includes, know, latest and greatest architectures in fact.
Unfortunately, we don't have anything like this for Bayesian experimental design yet, but I am in touch with the Baystow guys and I'm definitely looking into implementing some of these experimental design workflows in their package.
So I have it on my to-do list to actually write a little tutorial in Baseflow, how you can use Baseflow and your favorite deep learning framework of choice, whether it's PyTorch or JAX or like whatever, TensorFlow, to train sort of a... a policy, a design policy along with all of the amortized posteriors and all the bells and whistles that you may need to run, you know, some pipeline like that. Right. So I mentioned the modeling aspect, I mentioned the software aspect.
I think thinking about the problem. in like other aspects of like, you going to run an adaptive experiment or are you going to run a static experiment? Right. So adaptive experiments are much more complicated than static experiments. So in an adaptive experiment, you're always conditioning on the data that you have gathered so far, right?
In a static experiment, you just design a large batch of experiments and then you run it once, you collect your data and then you do your Bayesian analysis from there. Right. And so I generally always recommend starting with the simpler case, figure out whether the simpler case works, do the proof of concept on a static or non-adaptive type of Bayesian experimental design. And then, and only then, start to think about, let me train a policy or let me try to do an adaptive experimental design.
pipeline. I think this is a bit of a common pitfall if I may say, like people tend to like jump to the more complicated thing before actually figuring out kind of the simple case. Other than that, I think, again, I think it's a kind of an active area of research to, you know, figure out ways to evaluate your designs. I think by now we have pretty good ways of evaluating the quality of our posteriors, for example, right?
You have various posterior diagnostic checks and so on that doesn't really exist as much for designs, right? So what does it like, you know, I've maximized my information objective, right? I have collected as much information as I can, right? According to this information objective. But what does that mean in practice, right? Like there is no kind of real world information that I can... test with, right? Like if you're doing predictions, can, you can predict, can observe and then compare, right?
And you can compute an accuracy score or a root mean squared error or like whatever makes sense. There doesn't really exist anything like this in, in sort of in, in, in design, right? So it becomes much harder to quantify the success of such a pipeline. And I think it's, it's, it's a super interesting area for development. It's part of the reason why I work in the field.
I think there is many open problems that if we figure out, I think we can advance the field quite a lot and make data gathering an actual thing, principled and robust and reliable so that you run your expensive pipeline, but you end up with you kind of want to be sure that the day that you end up with is actually useful for the purposes that you want to use it. yeah, did that answer the question?
So you have the modeling aspect, you have the software aspect, which we are developing, I think, you know, we will hopefully eventually get there. Think about your problem, start simple. Try to think about diagnostics. And I think, again, I mentioned, you know, it's very much an open an open problem, but maybe for your concrete problem at hand, you might be able to kind of intuitively say, this looks good or this doesn't look good.
Automating this is like, it's a very interesting open problem and something that I'm actively working on. Yeah. Yeah. And thank you so much for all the work you're doing on that because I think it's super important. I'm really happy to see you on the base flow side because yeah, those guys are doing Amazing work. There is the new version that's now been merged on the dev branch, which is back in agnostics. So people can use it with their preferred deep learning package.
So I always forget the names, but TensorFlow, PyTorch, and JAX, I'm guessing. I'm mostly familiar with JAX because that's the one. and a bit of PyTorch because that's the ones we're interacting with in PymC. This is super cool. I've linked to the Baseflow documentation in the show notes. Is there maybe, I don't know, a paper, a blog post, something like that you can link people to with a workflow of patient experimental design and that way people will get an idea of how to do that.
So hopefully by the time the episode is out, I will have it ready. Right now I don't have anything kind of practical. I'm very happy to send some of the kind of review papers that are out there on Bayesian Experimental Design. hopefully in the next couple of weeks I'll have the tutorial, like a very basic introductory tutorial.
you know, we have a simple model, we have our simple parameters, you know, what we want to learn, here is how you define your posteriors, here is how we define your policy, you know, and then you switch on base flow, and then you know, voila, you have you have you have your results. So yeah, I'm hoping to get a blog post like of this of this sort done in in the next couple of weeks. So once ready, I will thank you. I will thank you with that. Yeah, for sure. yeah, for sure.
Can't wait and Marvin and I are gonna start working on setting up a modeling webinar Amazing which is a you know another format I have on the show So this is like, you know, I'm like Marvin will welcome and share his screen and show how to do the the amortized patient inference workflow with base flow also using pinc and all that cool stuff now that the new API is merged, we're going to be able to work on that together and set up the modeling webinar.
So listeners, definitely stay tuned for that. I will, of course, announce the webinar a bit in advance so that you all have a chance to sign up. And then you can join live, ask questions to Marvin. That's going to be super fun. And mainly see how you would do Amortize Bayesian inference, concretely. Great. Amazing. Sounds fun. Yeah, that's going to be super fun. Something I was thinking about is that your work mentions enabling real-time design decisions.
And that sounds really challenging to me. So I'm wondering how critical is this capability in today's data-driven decision-making processes? Yeah. I do think it really is quite critical, right? In most kind of real world practical aspects, practical problems, you really do need to run to make decisions fairly quickly. Right?
Again, all the surveys as an example, you know, you have anything that involves a human, and you want to adapt as you're performing the experiment, you kind of need to ensure that things are you're able to run things in real time.
And honestly, I think part of the reason why we haven't seen a big explosion or based on experimental design in practice is partly because we couldn't until recently actually run these things fast enough, both because of the computational challenges, now that we know how to do amortize inference very well, now that we know how to train policies. that will produce designs very well. I am expecting things to, you know, to improve, right?
And to start to see some of these some of these methods applied in practice. Having said that, I do think and please stop me if that's totally unrelated, but to make things successful in practice, there are a few other things that in my opinion have to be resolved before, you know, we're confident that we can, you apply, you know, such black boxes in a sense, right, because we have all these neural networks all over the place.
And it's not entirely clear whether all of these things are robust to various aspects of the complexities of the real world. Right. So things like model mis-specification, right? So is your Bayesian model actually a good representation of the thing that you're trying to study? That's a big open problem again. And again, I'm going to make a parallel to Bayesian inference actually. For inference purposes, model mis-specification may not be as bad as it is for design purposes.
And the reason for that is you will still get valid under some of the assumptions, of course, you will still get valid inferences or like you will still be close. You still get the best that you can do under the under the the the assumption of a wrong model. Now, when it comes to design, we have absolutely no guarantees. And oftentimes we end up in very pathological situations where because we're using our model to inform the data collection.
and then to also evaluate, right, fix that model on the same data that we've gathered. If your model is misspecified, you might not even be able to detect the misspecification because of the way that the data was gathered. It's not IID, right? Like it's very much a non-IID data collection process.
And so I think when we talk about practical things, we really, really need to start thinking about how are we going to make our systems or the methods that we develop a little bit more robust to misspecification. And I don't mean we should solve model misspecification. I think that's a very hard task that is basically unsolvable, right? Like it is solvable under assumptions, right?
If you tell me what your misspecification is, you know, we can improve things, but in general, this is not something that we can sort of realistically address uniformly. But yeah, so again, going back to practicalities, I do think it's of crucial importance to sort of make our pipeline in diagnostics sort of robust to some forms of mis-specification. Yeah. Yeah, yeah, for sure.
And that's where also I really love Amortized Patient Inference because it allows you to do simulation-based calibration. And I find that especially helpful and valuable when you're working on developing a model because already before fitting to data, you already have more confidence about what your model is actually able to do and not do and where the possible pain points would be. And I find that. super helpful.
And actually talking about all that, I'm wondering where you see the future of Bayesian experimental design heading, particularly with advancements in AI and machine learning technologies. Wow. Okay. So I do view this type of work. So this type of research is a little bit orthogonal to all of the developments in sort of modern AI and machine learning. And the reason for this is that we can literally borrow the latest and greatest development in machine learning and plug it into our pipelines.
there is a better architecture to do X, right? Like we can take that architecture and, you know, utilize it for our purposes. So I think you know, when it comes to the future of Bayesian experimental design, given, you know, all of the advancements, I think this is great because it's kind of helping the field even more, right? Like we have more options to choose from, we have better models to choose from, and kind of the data gathering aspect will always be there, right?
Like we will always want to collect better data for the purposes of, you know, our data analysis. And so, you know, the design aspect will still be there and with the better models, we'll just be able to gather better data, if that makes sense. Yeah, that definitely makes sense. Yeah, for sure. And that's interesting. Yeah, I didn't anticipate that kind of answer, that's okay. definitely. see what you mean.
Maybe before like, yeah, sorry, but even if you think, you know, now you're everybody's gonna have their AI assistant, right? Now, wouldn't it be super frustrating if your AI assistant takes three months to figure out what you like for breakfast? And like, it's experimenting or like, it's just randomly guessing. do you like fish soup for breakfast? Like, How about I prepare you a fish soup for breakfast or like, or I propose you something like that, right?
And so I think again, like this personalization aspect, right? Like again, kind of sticking to, I don't know, personal AI assistance, right? The sooner or the quicker they are able to learn about your preferences, the better that is. And again, you know, we're learning about preferences. Again, I'm gonna refer back to the... you know, time value of many preference learning, like it is just a more complicated version of that. Right.
And so if your latest and greatest AI assistant is able to learn and customize itself to your preferences much more quickly than otherwise, you know, that's a huge win. Right. And I think this is precisely where all these sort of principle data gathering techniques can really shine. Once we figure out, you know, the the sort of the issues that I was talking about, I mean, that makes sense.
Maybe to play us out, I'm curious if you have something like applications in mind, practical applications of bed that you've encountered in your research, particularly in the fields of healthcare or technology that you found particularly impactful. Right. Excellent question. And actually, I was going to mention that aspect as, you know, where you see the future of Bayesian experimental design.
Part of kind of our blind spots, if I may refer to that as sort of blind spots, is that in our research so far, we have very much focused on developing methods, developing computational methods to sort of make some of these based on experimental design pipelines actually feasible to run in practice. Now, we haven't really spent much time working with practitioners, and this is a gap that we're actively trying to sort of close. In that spirit, we have a few applications in mind.
We sort of apply people. particularly in the context of healthcare, as you mentioned. So clinical trials design is a very big one. So again, things like getting to the highest safe dose as quickly as possible, For, again, being personalized to the human, given their context, given their various characteristics. is one area where we're looking to sort of start some collaborations and explore this further.
Our group in Oxford have a new PhD student joining that will be working in collaboration with biologists to actually design experiments for something about cells. I don't know anything about biology, so. I'm not the best person to actually describe that line of work. But hopefully there will be some concrete, exciting applications in the near future. So that's applications in biology.
And finally, you know, there's constantly lots of different bits and pieces like, you know, people from chemistry saying, hey, I have this thing, can we... Can we work on maybe, you know, performing or like setting up a basic experimental design pipeline? I think the problem that we've had so far or I've had so far is just lack of time. There's just so many things to do and so little time.
But I am very much actively trying to find time in my calendar to actually work on a few applied projects because I do think, you know, It's all like developing all these methods is great, right? I mean, it's very interesting math that you do. It's very interesting coding that you do. But at the end of the day, you kind of want these things to make someone life's better, right?
Like a practitioner that will be able to save some time or save some money or, you know, improve their data gathering and therefore, you know, downstream analysis much better. and more efficient thanks to some of this research. So I hope that answered your question in terms of concrete applications. think we'll see more of that. But so far, you know, the two things are, yeah, clinical trial design that we're exploring and some of this biology cell stuff. Yeah, yeah, no, that's worrying.
that's, I mean, definitely looking forward to it. That sounds absolutely fascinating. yeah, if you can make that. happening in important fields like that, that's going to be extremely impactful. Awesome, Desi. I've already... Do you have any sort of applications that you think design, basic experimental design might be suitable? I know you're quite experienced in various modeling aspects based in modeling. So, yeah, do you have anything in mind? Yeah, I mean a lot.
So marketing, know is already using that a lot. Yeah. Clinical trials for sure. Also now that they work in sports analytics, well, definitely sports, you know, you could include that into the training of elite athletes and design some experiments to actually test causal graphs and see if pulling that lever during training is actually something that makes a difference during the professional games that actually count.
yeah, I can definitely see that having a big impact in the sports realm, for sure. Nice. Well, if you're open to collaborations, can do some designing of experiments once you up your sports models. Yeah. Yeah. Yeah. mean, as soon as I work on that, I'll make sure to reach out because that's going to be... It's definitely something I'm work on and dive into. So that's gonna be fascinating to work on that with you for sure. Sounds fun, yeah. Yeah, exactly. Very, very exciting.
well, thanks, Stacey. That was amazing. I think we covered a lot of ground. I'm really happy because Alina had a lot of questions for you. But thanks a lot for keeping your answers very... focused and not getting distracted by all my decorations. Of course, I have to ask you the last two questions I ask every guest at the end of the show. So first one, if you had unlimited time and resources, which problem would you try to solve? it's a really hard one.
Honestly, I because I know you asked those questions. I was like, what am going to say? I honestly don't know. There's so many things. But Again, I think it would be something of high impact for humanity in general, probably so in climate change would be something that I would dedicate my unlimited time and resources. That's good answer. That's definitely a popular one. So you're in great company and I'm sure the team already working on that is going to be very... be happy to welcome you.
No, let's hope. I think we need to speed up the solutions. Like seeing what is happening, right? I think it's rather unfortunate. And second question, if you could have dinner with any great scientific mind, dead, alive or fictional, who would it be? Yeah. So I think it will be Claude Shannon. So, you the godfather of information theory or like actually the father of information theory.
Again, partly because a lot of my research is inspired by information theory principles in Bayesian experimental design, but also outside of Bayesian experimental designs. It sort of underpins a lot of the sort of modern machine learning development, right? And What I think will be really quite cool is that if you were to have dinner with him, if I were to have dinner with him and basically tell him like, hey, look at all these language models that we have today.
Like Claude Shannon was the person that invented language models back in 1948, right? So that's many years ago. And like, literally not even having computers, right? So he would, he calculated things by hand and produced output that actually looks like English, right? In 1948. And so I think, you know, a brilliant mind like him, you know, just seeing kind of the progress that we've made since then. And like, we actually have language models on computers that behave like humans.
I'll be super keen to hear like, what is next from him. And I think he will have some very interesting answers to that. What is the future for information processing and the path to artificial, I guess people call it artificial general intelligence. So what would be the path to AGI from here onwards? Yeah, for sure. That'd be a fascinating dinner. Make sure it comes like that. Awesome. Well. Desi, thank you so much. think we can call it a show. I learned so much.
I'm sure my listeners did too, because as you showed, is a very, this is a topic that's very much on the frontier of science. So thank you so much for all the work you're doing on that. And as usual, I put resources and a link to your website in the show notes for those who want to dig deeper. Thank you again, Desi, for taking the time and being on this show. Thank you so much for having me. It was my pleasure. This has been another episode of Learning Bayesian Statistics.
Be sure to rate, review, and follow the show on your favorite podcatcher, and visit learnbaystats.com for more resources about today's topics, as well as access to more episodes to help you reach true Bayesian state of mind. That's learnbaystats.com. Our theme music is Good Bayesian by Baba Brinkman, fit MC Lass and Meghiraam. Check out his awesome work at bababrinkman.com. I'm your host. Alex and Dora. can follow me on Twitter at Alex underscore and Dora like the country.
You can support the show and unlock exclusive benefits by visiting Patreon.com slash LearnBasedDance. Thank you so much for listening and for your support. You're truly a good Bayesian. Change your predictions after taking information in. And if you're thinking I'll be less than amazing. Let's adjust those expectations. me show you how to be a good Bayesian Change calculations after taking fresh data in Those predictions that your brain is making Let's get them on a solid foundation