Priors represent a crucial part of the Bayesian workflow, and actually a big reason for its power and usefulness. But why is that? How do you choose reasonable priors in your models? What even is a reasonable prior? These are deep questions that today's guest, Sonja Winter, will guide us through.
an assistant professor in the College of Education and Human Development of the University of Missouri, Sonia's research focuses on the development and application of patient approaches to the analysis of educational and developmental psychological data, with a specific emphasis on the role of priors. What a coincidence! In this episode, she shares insights on the selection of priors, prior sensitivity analysis, and the challenges of working with longitudinal data.
She also explores the implications of Bayesian methods for model selection and fit indices in structural equation modeling, as well as the challenges of detecting overfitting in models. When she's not working, you'll find Sonja baking delicious treats, gardening, or watching beautiful birds. This is Learning Bayesian Statistics, episode. Welcome to Learning Bayesian Statistics, a podcast about Bayesian inference, the methods, the projects, and the people who make it possible.
I'm your host, Alex Andorra. You can follow me on Twitter at alex .andorra, like the country. For any info about the show, learnbasedats .com is Laplace to be. Show notes, becoming a corporate sponsor, unlocking Bayesian Merge, supporting the show on Patreon, everything is in there. That's learnbasedats .com. If you're interested in one -on -one mentorship, online courses, or statistical consulting, feel free to reach out and book a call.
at topmate .io slash alex underscore and dora see you around folks and best patient wishes to you all. Hello my dear patients, today I want to thank the fantastic Jonathan Morgan and Francesco Madrisotti for supporting the show on Patreon. Your support is invaluable and literally makes this show possible. can't wait to talk with you guys in the Slack channel.
Second, with my friends and fellow PymC core developers, Ravin Kumar and Tommy Capretto, we've just released our new online course, Advanced Regression with Bambi and PymC. And honestly, after two years of development, it feels really great to get these out into the world, not only because it was, well, long and intense, but mainly because I am so proud of the level of of details, teachings, and exercises that we've packed into this one.
It's basically the course I wish I had once I had gone through the beginner's phase when learning patience tests, that moment when you're like... Okay, I know how to do basic models, but where do I go from here? I remember feeling quite lost, so we wanted to give you a one -stop shop for such intermediate models with the most content possible, as evergreen as it gets. If that sounds interesting, go to intuitivebase .com and check out the full syllabus.
We're enrolling the first cohort as we speak! Of course, you get a 10 % discount if you're a patron of the show. Go to the Patreon page or the Slack channel to get the code. Okay, back to the show now and looking forward to seeing you in the intuitive base discourse. Sonia Winter, welcome to Learning Bayesian Statistics. Thank you. Thanks for having me. I'm really excited to talk to you today. Same, same. That's a treat. I have a lot of questions. I really love.
like everything you're doing in your research. We're going to talk a lot about priors today, folks. So yeah, like get ready. But first, can you provide a brief overview of your research interests and how patient methods play a role in your work? Yeah, sure. So my background is actually in developmental psychology. I did a bachelor or master's degree at Utrecht University. And during that time, I really realized that a lot of work needed to be done on the analysis part of social science research.
And so I switched and got really into structural equation models, which are these big multivariate models that include latent variables. I'm sure we'll talk more about that later. But those models can be hard to estimate and there are all these issues. And so I was introduced to Bayesian statistics. right after my master's degree when I was working with Rens van der Schoot, also at Utrecht University. And he asked me to do this big literature review about it with him.
And that really introduced me. And so now I focus a lot on Bayesian estimation and how it can help us estimate these structural equation models. And then specifically more recently, I've really been focusing on how those priors can really help us. both with estimation and also just with understanding our models a little bit better. So yeah, I'm really excited about all of that. Yeah, I can guess that sounds awesome. So structural equation modeling, we already talked about it on the show.
So today we're going to focus a bit more on priors and how that fits into the SEM framework. So for people who don't know about SEM, I definitely recommend episode 102 with Ed Merkel. And we talked exactly about structural equation modeling and causal inference in psychometrics.
So that will be a. a very good introduction i think to these topics for people and what i'm curious about sonia is you work a lot on priors and things like that but how how did you end up working on that was something that you were always curious about or that something that appeared later later on in your in your phd studies I would say definitely something that started or piqued my interest a little bit later.
I think so after I first got familiarized with Bayesian methods, I was excited mostly by how it could help, like priors could help us estimate, like avoid negative variances and those types of things. But I saw them more as a pragmatic tool to help with that. And I didn't really focus so much on that.
I feel like I also was a little bit afraid at the time of, you know, those researchers who talk a lot about, well, we shouldn't really make our priors informative because that's subjective and that's bad. And so I really typically use like uninformative priors or like software defaults for a lot of my work in the beginning. But then during my PhD studies, I actually. Well, first of all, I worked with another researcher, Sanaa Smith, who was also a PhD student at the time.
And she was really intrigued by something she found that these software defaults can really cause issues when you're, especially when your data is like very small, it can, it can make your results look wild. And so we worked on this paper together and created a shiny app to demonstrate all of that. And that made me realize that maybe uninformative priors. are not always the best way to go. And also a prior that looks informative in one scenario might be relatively uninformative in another.
And so I really started shifting my, my perspective on priors and focusing more on how ignoring them is kind of like ignoring the best part of Bayesian in my opinion, at this point. and so now I really want to look at how, how they can help us and how we can be thoughtful. We don't want to drive our science by priors, right? We want to learn something new from our data, but we find that balance is really what I'm looking for now.
Yeah, well, what a fantastic application of updating your belief, right? From a meta standpoint, you just like updated your priors pretty aggressively and also very rationally. That's really impressive. Well done. Because that's hard to do also. It's not something we like to do. So that's great. Well done on doing that. And actually now that you're on the other side, how do you approach the selection of priors in your research and what advice do you have for people new to Bayesian methods?
Yeah, great question. I think at least within structural equation modeling, we as like applied researchers are helped somewhat because distributions, at least for priors, are sort of clear. Like you don't have to think too much about them. And so you can immediately jump into thinking about, okay, what level of information do I want to convey in those priors?
And I think whenever I'm working with applied researchers, I try to strike a balance with them because I know they are not typically comfortable using like super informative priors that are really narrow. And so I just asked them to think about, well, what would be a reasonable range? Like if we are estimating a linear regression parameter, what would that effect size look like? Right. It might be zero or it might be two, but it's probably not going to be 20. And so we can.
sort of shape our prior to align with those sort of expectations about how probable certain values are versus others. It's a really, I don't know, interactive process between me and the researcher to get this right, especially for those types of parameters that they are really interested in. I think another type of parameter that is more challenging for applied researchers are those that are placed on residual variances, for example.
Like people typically don't... think about the part of the outcome that they can't explain that much. And so that's where I do rely a bit more on sort of, I don't know, industry standard choices that are typically not super informative. But then once we pick our like target priors, I always advise the researcher to follow it up with a sensitivity analysis to see. like how robust their findings are to changes in those priorities, either making them more informative or less informative.
And so yeah, that's really the approach I take. Of course, if someone wants to go full base and full informative and they have this, this wealth of previous research to draw from, then I'm all for going, going that route as well. It's just not as common. Hmm. Hmm. Yeah, I see. in what, what are the... main difficulties you see from people that you advise like that?
Where do you see them having more defenses up or just more difficulties because they have a hard time wrapping their head around a specific concept? I think just all over, I think if anyone has ever tried to do like a power analysis working with researchers, it's sort of a similar concept because It is not, at least in my field or the people I work with are not very typically already thinking about the exact parameter estimates that they are expecting to see, right?
They are just, they just go with the hypothesis. I think these two things are correlated and they might not even go as far as to think, is it positive or negative? So then once you ask them those questions, it really forces them to go much deeper on their theory and really consider like. What is, what am I expecting? What is reasonable based on what I know from, from previous studies or just experience. And that can be kind of challenging.
It's, it's kind of, I think sometimes the researchers might feel like I'm criticizing them for not knowing, but I think that's perfectly normal to not know. Like we already have so many other things to think about. But it definitely. is kind of a hurdle. Also the time commitment, I think, to really consider the priors, especially if you're coming from a frequentist realm where you just say, okay, maximum likelihood go.
Not only do you not have to think about the estimation, but then also your results are almost instant. And so that's always kind of a challenge as well. I see. Yeah. Yeah. Definitely something also I seen, I seen beginners. yeah, it, it really depends on also where they are coming from, as you were saying. Yeah. I did. Your advice will depend a lot on that. yeah. Yeah. And actually you work also a lot on prior sensitivity analysis. can you, can you tell people what that is?
And the importance of it in your, in your modeling workflow and. how you incorporate it into your research. Yeah. So a sensitivity analysis for priors is something that you typically do after you run your main analysis. So you come up with your target set of priors for all your parameters, estimate the model, look at the results, look at the posteriors. And then in the next step, you think about, well, how can I change these priors?
in sort of meaningful ways, either making them more informative, perhaps making them represent some other theory, making them less informative as well. So making the influence of the prior weaker in your results. And then you rerun your analysis for all of those different prior scenarios, and then compare those results to the ones that you actually obtained with your target analysis and your target priors.
And the idea here is to see, how much your results actually depend on those prior beliefs that you came into the analysis with. If you don't find any differences, then you can say, well, my results are mostly influenced by my data, by the new evidence that I obtained. They are robust to changes in prior beliefs, right? It doesn't really matter what beliefs you came into the analysis with. The results are going to be the same, which is great.
In other cases, you might find that your results do change meaningfully. So for example, in effect that was significant with your priors is no longer significant using a frequentist term here, but hopefully people will understand once you change your priors. And that's, of course, is a little bit more difficult to handle because what do you do?
I want to say that the goal is not to use the sensitivity analysis to then go back and change your priors and run the analysis again and report that in your paper. That would be sort of akin to p -hacking. Instead, I think it just contextualizes your findings. It's showing that the knowledge you came into the analysis with is partially driving your results. And that probably means that the evidence in your new data is not super strong.
And so it may indicate some issues with your theory or some issues with your data. And you have to collect more data to figure out which of those it is basically. And so it's, it's kind of helping you also figure out the next steps in your research, I feel, which is helpful. But it can be frustrating, of course, and harder to convince maybe co -authors and reviewers to. move forward with a paper like that. But to me it is very interesting these results from sensitivity analyses.
Yeah, yeah, completely agree in that. That's very interesting to see the, yeah, if the results differ on the priors, and that can also help, you know, settle any argument on the choice of prior. You know, if people are really in disagreement about which priors to choose, well, then you can run the model with both sets of priors, and if the results don't change, it's like, well, let's stop arguing. That's kind of... It's kind of silly. We just lost time. So let's just focus on the results then.
I think it's a very interesting framework. And then there is another. So that is like that entails running the model, running MCMC on the model. But there are some checks that you do before that to ensure the robustness of your patient models. And one of that step is. very crucial and called primary predictive checks. Can you talk about that to beat Sonja? Yeah, so as you said, these checks happen before you do any actual analysis. So you can do them before you collect any data.
In fact, one reason for using them is to figure out whether the priors you came up with results in sensible ranges of possible parameter estimates, right? In some cases, especially with these complex multivariate models, your priors may interact in unexpected ways and then result in predictions that are not in line with what your theory is actually telling you you should expect. And so prior predictive checks basically until you specify your priors for all your parameters.
And then you generate parameter values from those priors by combining it with your model specification. And then those combinations of parameter estimates are used to generate what are called prior predictive samples. So these are samples of some pre -specified size that represent possible observations that align with what your priors are conveying combined with your model. And so ideally, those prior predictive samples look kind of like what you would expect your data to look like.
And sometimes for researchers, it is easier to think about what the data should look like compared to what the parameter estimates can be. And so in that sense, prior predictive checks can be really helpful in checking not just the priors, but checking the researcher and making sure that they actually convey their knowledge to me, for example, correctly. Yeah, did that answer your question?
Yeah, yeah, I think that's a great definition and definitely encourage any Bayesian practitioner to include prior predictive checks in their workflow. Once you have written a model, that should be the first thing you do. Do not run a CMC before doing prior predictive checks.
And recently, I feel like a lot of the software packages for Bayesian methods have... included very simple ways of running these checks, which when I first started looking at them, it was kind of more of a niche step in the workflow. And so it required a few more steps and some more like coding, but now it's as easy as just switching like a toggle to get those prior predictive samples. So that's great. Yeah, yeah, yeah, completely agree.
That's also, yeah, it's definitely something that's, that's been more and more popular in the different classes and courses that I teach, whether it's online courses or live workshops, always show prior predictive checks almost all the time. So yeah, it's becoming way, way more popular and widespread. So that's really good because I can tell you when I work on a real model for clients, always the first thing I do before running MCMC is prior predictive checks.
And actually there is a fantastic way of... you know, doing prior predictive checks, like kind of industrialized and that's called simulation based calibration. Have you heard of that? No, I mean, maybe the term, but I have no idea what it is. So that's just like making prior predictive checks on an industrialized scale.
Basically now instead of just running through the model forward, as you explained, and generate prior predictive samples, what you're doing with SPC, so simulation -based calibration, is generating not only prior predictive samples, but prior samples of the parameters of the model. You stock these parameters in some object. but you don't give them to the model, but you keep them somewhere safe.
And then the prior predictive samples, so the plausible observations generated by the model based on the prior samples that you just kept in the fridge, these prior predictive samples, now you're going to consider them as data. And you're going to tell the model, well, run MCMC on these data. as if we had observed these prior predictive samples in the wild, because that's what prior predictive samples are. It's possible samples we could observe before we know anything about real data.
So you feed that to the model. You make the model run MCMC on that. So that means backward inference. So now the model is going to find out about the plausible parameter values which could have generated this data. And then what you're going to do is compare the posterior distribution that the model inferred for the parameter values to the true parameter values that you kept in the fridge before. You're going to get, so these parameter values are true.
So you just have one of them, because it's just one sample from the prior parameters. And you're going to compare these value, these value to the distribution of posterior parameters that you just got from the model. And based on that, and how far the model is from the true parameter, you can find out if your model is biased or if it's well calibrated. And that's a really great way to be much more certain that the model is able to recover what you want it to recover.
basically playing God, and then you're trying to see if the model is able to recover the parameters that you use to generate the data. And not only will you do that once, but you want to do that many times, many, many times, because, well, the more you do it, then you enter a kind of a frequentist realm, right? Where you're like, you just repeat the experiments a lot. And then that's how you're going to see how calibrated the model is, because then you can do some calibration plots.
there are a lot of metrics around that it's a kind of a developing area of the research but there are a lot of metrics and one of them is basically just plotting the true parameter values and well for instance the mean posterior value from the parameter and then if this mean is most of the time along the the line x equals y well that means you are in pretty good shape you are but I mean it's the mean here you So you have to look at the whole distribution, but that's to give you an idea.
And so the bottleneck is you want to do that a lot of time. So you have to run MCMC a lot of times. Most of the time, if you're just doing a regression, that should be okay. But sometimes it's going to take a lot of time to run MCMC and it can be hard. In these cases, you have new algorithms that can be efficient because there is one called amortized Bayesian inference, a method called amortized Bayesian inference. We just covered that in episode 107 with Marvin Schmidt.
And basically that's exactly a use case for amortized Bayesian inference because the model doesn't change, but the data changes in each iteration of the loop. And so what amortized Bayesian inference is doing is just, well, just is training a deep neural network on the model. as a first step. And then the second step is the inference, but the inference is just instantaneous because you've trained the deep neural network.
And that means you can do, you can get as black, almost as many poster samples as you want. Once you have trained the deep neural network. And so that's why it's almost all inspection inference. And that's a perfect use case for SBC because then like you can just like you get new, a new, new samples for free. And actually, so I definitely encourage people to look at that. It's still developing.
So right now you cannot, for instance, use Baseflow, which is the Python package that Marvin talked about in 1 .07 with PIMC, but it's something we're working on. And the goal is that it's completely compatible. But yeah, like I'll link to the tutorial notebook in the show notes for people. who want to get an idea of what SPC is because even though you're not applying it right now at least you have that in mind and you know what that means and you can work your way to out that.
Yeah that's amazing. I feel like one of the biggest hurdles in the structural equation modeling approach with using Bayesian is just the time commitment. I'm There is one analysis I was running and it takes, I think for one analysis, it takes almost a week to run it because it's a big sample and then it's a complicated model. And so if I would have to rerun that model a thousand times, it would not be fun.
so knowing that there's maybe some options on the horizon to help us speed along that process would be, I think that would change our field for sure. So that's very exciting. Yeah, yeah, yeah. That's really super exciting. And that's why I'm also super enthusiastic about the desalmatized Bayesian infant stuff, because I discovered that in episode 107, so it's not a long time ago. But as soon as I heard about that, I dug into it, because that's super interesting.
Yeah. I'm going to read about it after we finish recording this. Yeah, yeah, for sure. And feel free to send me any questions. And I find it's also a very elegant way to marry the Bayesian framework in the deep neural network methods. So I really love that. It's really elegant and promising, as you were saying. Talking about SCM, so structural equation modeling, do you find that Bayesian methods help?
in for these kind of models and especially when it comes to educational research which is one of your fields? Yes, I think Bayesian methods can sort of help on both ends of the spectrum that we see with educational data which is either we have very small samples and so researchers still have these ambitious theoretical models that they want to test. but it's just not doable with frequentist estimators.
And so based with the priors, it can help a little bit to boost the information that we have, which is really nice. And then on the other side, ever since starting this position and moving into a college of education, I've been given access to many large data sets that have very complicated nesting structures. That's something you see all the time in education.
You have schools and then teachers and students and the students they change teachers because it's also longitudinal so there's a time component and all of these different nested structures can be very hard to model using estimators like nextman likelihood and bayesian methods not necessarily structural equation modeling but maybe more a hierarchical linear model or some other multi -level approach it can be super flexible to handle all of those
structures and still give people results that they can use to inform policy. Because that's something in education that I didn't really see when I was still in the department of psychology before is that a lot of the research here is really directly informing what is actually going to happen in schools. And so it's really neat that these Bayesian methods are allowing them to answer much more complicated research questions and really make use of all of the data that they have.
So that's been really exciting. And actually, I wanted to ask you precisely what the challenges you face with longitudinal data and how do you address these challenges because I know that can be pretty hard. I think with longitudinal data, the biggest challenge actually doesn't have anything to do with the estimator. It is more just inherent in longitudinal data, which is that we will always... unless you have a really special sample, but we will always have missing data.
Participants will always drop out at some point or just skip a measurement. And of course, other estimation methods also have options for accommodating missing data, such as full information maximum likelihood. But I find that the Bayesian approach where you can do imputation while you're estimating, so you're just imputing the data at every posterior sample, is very elegant, efficient. and easy for researchers to wrap their minds around.
And it still allows you just like with other multiple imputation methods to include an sort of auxiliary model explaining the missingness, which helps with the like missing at random, type data that we deal with a lot. And so I feel that that is especially exciting. I honestly started thinking about this more deeply when I started my position here and I met my new colleague.
Dr. Brian Keller, he is working on some software, it's called BLIMP, which I think it stands for Bayesian Latent Interaction Modeling Program, I want to say. So it's actually created for modeling interactions between latent variables, which is a whole other issue. But within that software, they actually also created a really powerful method for dealing with missing data, or not necessarily the method, but just the application of it. And so...
Now that I've met him and he's always talking about it, it makes me think about it more. So that's very exciting. Yeah, for sure. And feel free to add a link to this project to Blimp in the show notes, because I think that's going to be very interesting to listeners. And how, I'm wondering if patient methods improve... the measurement and the evaluation processes in educational settings, because I know it's a challenge.
Is that something that you're working on actively right now, or you've done any projects on that that you want to talk about? Well, I teach measurement to grad students. So it's not necessarily that I get to talk about Bayes a lot in there. But what I'm realizing is that When we talk about measurement from a frequentist standpoint, we typically start with asking students a bunch of questions. Let's say we're trying to measure math ability. So we ask them a bunch of math questions.
Then if we use frequentist estimation, we can use those item responses to generate some sort of probability of those responses giving some underlying level of math ability. So how probable is it that they gave these answers given this level of math? But actually what we want to know is what is the student's math ability, given the patterns of observed responses. And so Bayes theorem gives us a really elegant way of answering exactly that question, right. Instead of the opposite way.
And so I think in a big way, Bayesian methods just align better with how people already think about the research that they're doing or the thing, the questions that they're, they want to answer. I think. This is also a reason why a lot of researchers struggle with getting the interpretation of things like a confidence interval correct, right? It's just not intuitive. Whereas Bayesian methods, they are intuitive.
And so in that sense, I think not so much like estimation wise, but just interpretation wise, Bayesian methods can help a lot in our field. And then in addition to that, I think when we do use Bayesian estimation, those posterior distributions, they can give us so much more information about the parameters of interest that we are interested in. And they can also help us understand what future data would look like given those posteriors, right?
If we move from like prior predictors to posterior predictors, which are these samples generated from the posteriors, that should look like our data should look like that data, right? If our model is doing a good job of representing our data. And so, I think that's an exciting extension of Bayes as well.
It gives us more tools to evaluate our model and to make sure that it's actually doing a good job of representing our data, which is especially important in structural equation modeling, where we rely very heavily on global measures of fit. And so this is a really nice new tool for people to use. I see. Okay. Yeah. I am. I need to know about that in particular. That's... That's very interesting.
Yeah. So I mean, I would have more questions on that, but I want to ask you in particular on a publication you have about under -fitting and over -fitting. And you've looked at the performance of Bayesian model selection in SEM. I find that super interesting. So can you summarize the key findings of this paper and... their application, their implications for researchers using SEM? Yeah, for sure. This is a really fun project for me to work on, kind of an extension of my dissertation.
So it made me feel like, I'm really moving on, creating a program of research. So yeah, thanks for asking about the paper. So yeah, as I already kind of mentioned, within structural equation modeling, Researchers rely really heavily on these model selection and fit indices to make choices about what model they're going to keep in the end. A lot of the times, researchers come in with some idea of what the model would look like, but they are always tinkering a little bit.
They're ready to know that they're wrong and they want to get to a better model. And so the same is true when we use Bayesian estimation and we have sort of a similar set of indices to look at. in terms of the fit of a single model or comparing multiple models and selecting the best one. And so very typically those indices are tested in terms of how well they can identify underfit. And so underfit occurs when you forgot to include a certain parameter.
So your model is too simple for the underlying data generating mechanism. You forgot something. And so all of these indices generally work. pretty well, and that's also what we found in our study in terms of selecting the correct model when there are some alternatives that have fewer parameters or picking up on the correct model fitting well by itself versus models that forget these parameters.
But what we were really interested in is looking at, OK, how well do these indices actually detect overfitting? So that's where you add parameters that you don't really need. So you're making your model overly complex. And when we have models that are too complex, they tend not to generalize to new samples, right? They're optimized for our specific sample and that's not really useful in science.
So we want to make sure that we don't keep going and like adding paths and making our models super complicated. And so surprisingly what we found across like a range of over fitting scenarios is that they do not really do a good job of detecting any of this. Most indices, if anything, just make the model look better and better and better.
Even some of these indices, like model selection indices, will have a penalty term in their formula that's supposed to penalize for having too many parameters, right? For making your model too complex. And even those were just like, yeah, this is fine. Keep going, keep going. And so that's a little bit worrisome. And I think... We really need to think about developing some new ways of detecting when we go too far, right?
Figuring out at what point we need to stop in our model modification, which is something that researchers really love to do, especially in structural equation modeling. I won't speak for any other areas. And so, yeah, I think there's a lot of work to be done. And I was very surprised that these indices that are supposed to help us detect overfitting also didn't really do. a good job. And so I'm excited to work more on this.
I would say in general, if people want an actionable takeaway, it is always helpful when you have multiple models to compare versus just your one model of interest. It will help you tease, sort of figure out better, which one is the correct one versus just is your model good enough? And so that would be my, my advice for researchers. Yeah. Yeah, definitely.
I always like having a very basic and dumb looking linear regression to compare to that and build my way on top of that because you can already do really cool stuff with plain simple linear regression and why making it harder if you cannot prove, you cannot discern a particular effect of... of the new method you're applying. Yeah. And so do you have then from from your dive into these, do you have some fit indices that you recommend? And how do they compare to traditional fit indices?
So I think for model fit of a single model within structural equation modeling. The most popular ones are called comparative fit index, the Tucker Lewis index, and then the root mean square error of approximation. You'll see these in like every single paper published. And so there are Bayesian versions of those indices, but based on all my research using those so far, I would actually not recommend those at all for evaluating the fit of your specific model.
It seems from at least my research that they are very sensitive to your sample size, which means that as you get a larger and larger sample, your model will just keep looking better and better and better and better, even if it's wrong. So something that would be flagged as like a... a misspecified model with a small sample might look perfectly fine with a large sample. And so that's not what you want, right? You want the fit index to reflect the misspecification, not your sample size.
And so I was really excited when these were first introduced, but I think we need a lot more knowledge about how to actually use them before they are really useful. And so my advice for researchers who want to know something about their fit is really to look at the posterior predictive checks.
And within structural equation modeling, I'm not sure how widespread this is for other methods, but we have something called a posterior predictive p -value, where we basically take our observed data and evaluate the fit of that data to our model at each posterior iteration. For example, using a likelihood ratio test or like a chi -square or something. And then we do the same for a posterior predicted sample. using this in within each of those samples as well.
And the idea is that if your model fits your data well, then about half of the predictive samples should fit better and the other half should fit worse, right? Yours should be nicely cozy in the middle. If all of your posterior predictive samples fit worse than your actual data, then it's an indication that you are overfitting, right? Like, the model will never fit as well as it does for your specific data.
And so I think in that sense, that index could potentially give some idea of overfitting, although again, in our study, we didn't really see that happening. But I think it's a more informative method of looking at fit within Bayesian structural equation modeling. And so even though it's kind of old school, I think it's still probably the... the best option for researchers to look at. Okay, yeah, thanks. That's like, I love that. That's very practical.
And I think listeners really appreciate that. I have like, I was wondering about SEMs again, and if you have an example from your research where Bayesian SEM provided significant insights that traditional methods might have missed. Yeah, so some work I'm working on right now is with a group of researchers who are really interested in figuring out how strong the evidence is that there is no effect, right? That some path is zero within a bigger structural model.
And with frequentist analysis, all we can really do is fail to reject the known, right? We have an absence of evidence. but that doesn't mean that there's evidence of absence. And so we can't really quantify like how strong or how convinced we should be that that null is really a null effect. But with Bayesian methods, we have base factors, right? And we can actually explicitly test the evidence in favor of the estimate being zero versus the estimate being not zero, right?
Either smaller or larger than zero. And so that's really... When I talked to the applied researchers, once they came to me with this problem, which started as just like a structural equation modeling problem, but then I was like, well, have you ever considered using Bayesian methods? Because I feel like it could really help you get at that question. Like how strong is that evidence relative to the evidence for an effect, right?
And so we've been working on that right now and it is very interesting to see the results and then also to communicate that with them and see. They get so excited about it. So that's been fun. Yeah, for sure. That's super cool. And you don't have anything to share in the show notes yet, right? Not yet. No, I'll keep you posted. Yeah, for sure. Because maybe by the time of publication, you'll have something for us.
Yes. And now I'd like to talk a bit about your... your teaching because you teach a lot of classes. You've talked a bit about that already at the beginning of the show, but how do you approach teaching Bayesian methods to students in your program, which is the statistics measurement and evaluation and indication program? Yeah, so I got to be honest and say I have never taught an entire class on Bayesian methods yet.
I'm very excited that I just talked with my colleagues and I got the okay to develop it and put it on the schedule. So it's coming. But I did recently join a panel discussion, which was about teaching Bayesian methods. It was organized by the Bayesian Education Research and Practice Section of the ISBA Association. And so the other two panelists, I was really starstruck. to be honest, were E .J. Wagemakers and Joachim van de Kerkoven, which are like, to me, those are really big names.
And so talking to them, I really learned a lot during that panel. I felt like I was more on the panel as a as an audience member, but it was great for me. And and so from that, I think if I do get to teach a class on Bayesian methods, which hopefully will be soon. I think I really want to focus on showing students the entire Bayesian workflow, right? Just as we were talking about, starting with figuring out priors, prior predictive checks, maybe some of that fancy calibration.
And then also doing sensitivity analyses, looking at the fit with the posterior predictive samples, all of that stuff. I think... For me, I wouldn't necessarily combine that with structural equation models because those are already pretty complicated models. And so I think within a class that's really focused on Bayesian methods, I would probably stick to a simple but general model, such as a linear regression model, for example, to illustrate all of those steps.
Yeah, I've been just buying, like I have a whole bookshelf now of books on Bayesian and teaching Bayesian. And so I'm excited to start reading those. developing my class soon yeah that's super exciting well done congrats on that i'm glad to hear that so first eg vagon makers was on the show i don't remember which episode but i will definitely link to it in the show notes and second yeah which books are you are you gonna use well Good question.
So there's one that I kind of like, but it is very broad, which is written by David Kaplan, who's at the University of Wisconsin Madison. And it's called, I think, vision statistics for the social sciences. And so what I like about it is that many of the examples that are used throughout the book are very relevant to the students that I would be teaching. And it also covers a wide range. of models, which would be nice.
But now that I've like philosophically switched more to this workflow perspective, it's actually a little bit difficult to find a textbook that covers all of those. And so I may have to rely a lot on some of the online resources. I know there's some really great posts by, I'm so bad with names. I want to say his name is Michael something. Where he talks about workflow. Yes, probably. Yes, that sounds familiar. His posts are really informative and so I would probably rely on those a lot as well.
Especially because they also use relatively simpler models. I think, yeah, for some of the components of the workflow that they just haven't been covered in textbooks as much yet. So if anyone is writing a book right now, please add some chapters on those lesser known. components, that would be great. Yeah. Yeah, so there is definitely Michael Bedoncourt's blog. And I know Andrew Gelman is writing a book right now about the Bayesian workflow. So the Bayesian workflow paper.
Yeah, that's a good paper. Yeah, I'll put it in the show notes. But basically, he's turning that into a book right now. Amazing. Yeah, so it's gonna be perfect for you. And have you taken a look at his latest book, Active Statistics? Because that's exactly for preparing teachers to teach patient stats. Yes, he has like an I feel like an older book as well where he has these activities, but it's really nice that he came out with this newer, more recent one.
I haven't read it yet, but it's on my on my to buy list. I have to buy these books through the department, so it takes a while. Yeah, well, and you can already listen to episode 106 if you want. He was on the show and talked exactly about these books. Amazing. I'll put it in the show notes. And what did we talk about? There was also Michael Betancourt, E .G. Wagenmarkers, Active statistics, microbed and code, yeah, and the Bayesian workflow paper. Yeah, thanks for reminding me about that paper.
Yeah, it's a really good one. I think it's going to be helpful. I'm not sure they cover SBC already, but that's possible. But SBC, in any case, you'll have it in the Bayes flow tutorial that I already linked to in the show notes. So I'll put out that. And actually, what are future developments in Bayesian stats that excite you the most, especially in the context of educational research? Well, what you just talked about, and this amortized estimation thing is very exciting to me.
I think, as I mentioned, one of the biggest hurdles for people switching to Bayesian methods is just the time commitment, especially with structural equation models. And so knowing that people are working on algorithms that will speed that up, even for a single analysis, it's just really exciting to me. And in addition to that, sort of in a similar vein, I think a lot of smart people are working on software, which is lowering barriers to entry.
People in education, they know a lot about education, right? That's their field, but they don't have time to really dive into. Bayesian statistics. And so for a long time, it was very inaccessible. But now, for example, as you already mentioned, Ed Merkel, he has his package Blavan, which is great for people who are interested in structural equation modeling and Bayesian methods. And sort of similarly, you have that Berkner has that BRMS package.
And then if you want to go even more accessible, there's JASP. which is that point and click sort of alternative to SPSS, which I really enjoy showing people to let them know that they don't need to be afraid that they'll lose access to SPSS at some point in their life. So I think those are all great things. And in a similar vein, there are so many more online resources now.
Then when I first started learning about base, like when people have questions or they want to get started, I have so many links to send them of like papers, online courses, YouTube videos, podcasts like this one. and so I think that's, what's really exciting to me, not so much what we're doing behind the scenes, right? The actual method itself, although that's also very exciting, but for working with people. in education or other applied fields.
I'm glad that we are all working on making it easier. So, yeah. Yeah. So first, thanks a lot for recommending the show to people. I appreciate it. And yeah, completely resonate with what you just told. Happy to hear that the educational efforts are.
useful for sure that's something that's very dear to my heart and I spend a lot of time doing that so my people and yeah as you are saying it's already hard enough to know a lot about educational research but if you have to learn a whole new statistical framework from scratch it's very hard and more than that it's not really valued and incentivized in the academic realm so like why would you even spend time doing that? you'd much rather write a paper.
So that's like, that's for sure that's an issue. So yeah, definitely working together on that is definitely helping. And on that note, I put all the links in the show notes and also Paul Burkner was on the show episode 35. So for people who want to dig deeper about Paul's work, especially BRMS, as you mentioned Sonia.
definitely take a well give a give a listen to that to that episode and also for people who are using Python more than are but really like the formula syntax that BRMS has you can do that in Python you can use a package called BAMI and it's basically BRMS in in Python in the that's built on top of PimC and that's built by a lot of very smart and cool people like my friend Tomica Pretto. He's one of the main core developers.
I just released actually an online course with him about advanced regression in Bambi and Python. So it was a fun course. We've been developing that for the last two years and we released that this week. So I have to say I'm quite relieved. Congratulations. Yeah, that's exciting. Yeah, that was a very fun one. It's just, I mean, it took so much time that because we wanted something that was really comprehensive and as evergreen as it gets.
So we didn't want to do something, you know, quick and then having to do it all over again one year later. So I wanted to take our time and basically take people from normal linear regression and then okay, how do you generalize that? How do you handle? non -normal likelihoods, how do you handle several categories? Because most of the examples out there in the internet are somewhat introductory. How do you do Poisson regression and binomial regression most of the time?
But what about the most complex cases? What happens if you have zero inflated data? What happens if you have data that's very dispersed that a binomial or a Poisson cannot handle? What happens if you have multi -category called data? More than two categories. You cannot use the binomial. You have to use the category called all the multinomial distributions. And these ones are harder to handle. You need another link function that the inverse logit. So it's a lot of stuff.
But the cool thing is that then you can do really powerful models. And if you marry that with hierarchical models, that is really powerful stuff that you can do. So yeah, that's what the whole course is about. I'll have Tommy actually on the show to talk about that with him. So that's going to be a fun one. Yeah, I'm looking forward to hearing more about it. Sounds like something I might recommend to some people that I know. Yeah, yeah. that's exciting. Yeah, yeah, for sure. Happy to. Happy to.
like send you send you the link I put the link in the show notes anyway so that people who are interested can can take a look and of course patrons of the show have a 10 % discount because they are they are the best listeners in the world so you know they deserve a gift yes they are well Sonya I've already taken quite a lot of your time so we're gonna we're gonna start closing up but I'm wondering if you have any advice to give to aspiring researchers who are
interested in incorporating Bayesian methods into their own work and who are working in your field, so educational research? Yeah, I think the first thing I would say is don't be scared, which I say a lot when I talk about statistics. Don't be scared and take your time. I think... A lot of people may come into Bayesian methods after hearing about frequentist methods for years and years and years.
And so it's going to take more than a week or two to learn everything you need to know about Bayes, right? That's normal. We don't expect to familiarize ourselves with a whole new field in a day or a week. And that's fine. Don't feel like a failure. Then. I don't know, I would also try and look for papers in your field, right?
Like if you're studying school climate, go online and search for school climate base and see if anyone else has done any work on your topic of interest using this new estimation method. It's always great to see examples of how other people are using it within a context that you are familiar with, right? You don't have to start reading all these technical papers. You can stay within your realm of knowledge, within your realm of expertise, and then just eke out a little bit.
And then after that, I mean, as we just talked about, there are so many resources available that you can look for, and a lot of them are starting to become super specific as well. So if you are interested in structural equation models, go look for resources about Bayesian structural equation modeling. But if you're interested in some other model, try and find resources specific to those.
And as you're going through this process, a nice little side benefit that's going to happen is that you're going to get really good at Googling because you've got to find all this information. But it's out there and it's there to find. So, yeah, that would really be my advice. Don't be scared. Yeah, it's a good one. That's definitely a good one because then...
Like if you're not scared to be embarrassed or fail, you're gonna ask a lot of questions, you're gonna meet interesting people, you're gonna learn way faster than you thought. So yeah, definitely great advice. Thanks, Sonja. And people in our field, Invasion Methods, they are so nice. I feel like they are just so excited when... I'm so excited when anyone shows any interest in what I do. Yeah, don't be scared to reach out to people either because they're going to be really happy that you did.
True, true. Yeah, very good point. Yeah, I find that community is extremely welcoming, extremely ready to help. And honestly, I still have to find trolls in that community. That's really super value. I feel like it helps that a lot of us came into this area through also kind of like a roundabout way, right? I don't think anyone is born thinking they're going to be a Beijing statistician and so we understand. Yeah, yeah. Yeah, well, I did. I think my first word was prior. So, you know, okay.
Well, you're the exception to the rule. Yeah, yeah. But you know, that's life. I'm used to being the black sheep. That's fine. no, I think I wanted to be a football player or something like that. no, also I wanted to fly planes. I wanted to be a fighter pilot at some point later after I had outgrown football. You're a thrill seeker. I wanted to be a vet or something, but then I had to take my pets to the vet and they were bleeding and I was like, no, I don't want the event anymore.
Well, it depends on the kind of animals you treat, but veterinarian can be a thrill seeking experience too. You know, like if you're specialized in snakes or grizzlies or lions, I'm guessing it's not all the time, you know, super, super easy and tranquil. no. Awesome. Well Sonia, that was really great to have you on the show. Of course, I'm going to ask you the last two questions. Ask every guest at the end of the show.
So if you had unlimited time and resources, which problem would you try to solve? I thought about this a lot because I wanted to solve many problems. So when I give this answer, I'm hoping that other people are taking care of all those other problems. But I think something that I've noticed recently is that a lot of people seem to have lost the ability or the interest in critical thinking and being curious and trying to figure out things by yourself. And so that's something that I would like to.
solve or improve somehow? Don't ask me how, but I think being a critical thinker and being curious are two really important skills to have to succeed in our society right now. I mean, there's so much information being thrown at us that it's really up to you to figure out what to focus on and what to ignore. And for that, you really need this critical thinking skill and... and also the curiosity to actually look for information. And so I think that's, it's also a very educational problem, I feel.
So if it's where I am right now in my career, but yeah, that would be something to solve. Yeah. Completely understand that was actually my answer also. So I like, really? Yeah. Yeah. Yeah. I completely agree with you. Yeah. These are topics I found. I find them. I find super interesting. How do you. do we teach critical thinking, how do we teach the scientific methods, things like that. It's always something I'm super excited to talk about.
Yeah, I also hope it will have some sort of trickle down effect on all the other problems, right? Once the whole world is very skilled at critical thinking, all the other issues will be resolved pretty quickly. Yeah, not only because it's directly solved, but... I would say mainly because then you have maybe less barriers. And so yeah, probably coming from that. And then second question, if you could have dinner with any great scientific mind, dead, alive or fictional food.
So I ended up Choosing Ada Lovelace who's like one of the first or maybe the first woman who ever worked in computer programming area. I think she's very interesting I also recently found out that she passed away when she was only like 36 Which is like I'm I'm getting at that age and she already accomplished all these things By the time she passed away and so now I'm like, okay I gotta I gotta step it up, but I would really love to talk to her about just her experience.
being so unique in that very manly world and in that very manly time in general, I think it would be very interesting to hear the challenges and also maybe some advantages or like benefits that she saw, like why did she go through all this trouble to begin with? Yeah, I think it would be an interesting conversation to have for sure. Yeah, yeah, definitely. Yeah, great choice. I think, I think somebody already had answered. I don't remember who, but yeah, it's not a very common choice.
We can have a dinner party together. Yeah, exactly. That's perfect. Fantastic. Great. Thank you. Thank you so much, Sonja. That was a blast. I learned so much. Me too. You're welcome. And well, as usual, I put resources and a link to a website. in the show notes for those who want to dig deeper. Thank you again, Sonia, for taking the time and being on this show. Yeah, thank you. It was so much fun. This has been another episode of Learning Bayesian Statistics.
Be sure to rate, review, and follow the show on your favorite podcatcher, and visit learnbaystats .com for more resources about today's topics, as well as access to more episodes to help you reach true Bayesian state of mind. That's learnbaystats .com. Our theme music is Good Bayesian by Baba Brinkman, fit MC Lass and Meghiraam. Check out his awesome work at bababrinkman .com. I'm your host. Alex Andorra. You can follow me on Twitter at Alex underscore Andorra, like the country.
You can support the show and unlock exclusive benefits by visiting Patreon .com slash LearnBasedDance. Thank you so much for listening and for your support. You're truly a good Bayesian. Change your predictions after taking information in. And if you're thinking I'll be less than amazing, let's adjust those expectations. Let me show you how to be a good Bayesian Change calculations after taking fresh data in Those predictions that your brain is making Let's get them on a solid foundation