If there is one guest I don't need to introduce, it is Mr. Andrew Gammann. So I won't. I will refer you back to his two previous appearances on the show, though, because learning from Andrew is always a pleasure. So go ahead and listen to episodes 20 and 27. The links are in the show notes. In this episode, Andrew and I discuss his new book, Active Statistics, which focuses on teaching and learning statistics through active student participation.
Like this episode, the book is divided into three parts. One, the ideas of statistics regression and causal inference. Two, the value of storytelling to make statistical concepts more relatable and interesting. And three, the importance of teaching statistics in an active learning environment where students are engaged in problem solving and discussion. And well, Andrew is so active and knowledgeable, that we of course touched on a variety of their topics, but for that, you'll have to listen.
This is Learning Basis Statistics, episode 106, recorded April 2, 2024. Welcome to Learning Bayesian Statistics, a podcast about Bayesian inference, the methods, the projects, and the people who make it possible. I'm your host, Alex Andorra. You can follow me on Twitter at alex .andorra, like the country. For any info about the show, learnbaystats .com is left last to be. Show notes, becoming a corporate sponsor, unlocking Bayesian Merge, supporting the show on Patreon, everything is in there.
That's LearnBasedStats .com. If you're interested in one -on -one mentorship, online courses, or statistical consulting, feel free to reach out and book a call at topmate .io slash alex underscore and dora. See you around, folks, and best patient wishes to you all. on LBS now, so for curious listeners, I definitely recommend episode 20, which was your first one with Andrew Gell -Mann. Yes, you were here.
And with Akive Tali and Jennifer Hale, it was both your previous book, Regression and Other Stories. And then episode 27 with Marilyn Heidemann, where we talked about the 2020 US presidential elections. We talked about the model you folks did for the economists. So definitely recommend checking this one out because I'm guessing this is going to be interesting also for this year's election. Yeah, we're working with them for 2024 as well. So we're trying to improve the model. Perfect.
Yeah. So it seems like you're releasing a book every four year just before the US election. I hope it won't be four years before an Xbook comes out. We're trying to finish our Bayesian workflow book. So we're hoping that will be done by the end of the year. Well, yeah, definitely curious to check this one out. I think I also saw that you're working on an MRP update book. Is that still the case? Yeah, I think Yajuan and some Lauren...
Uh, Kennedy and some other people are organizing this, um, uh, MRP book edited book we're putting together. Yeah. Um, I will definitely check these out. Well, writing books is a lot of fun because you can write whatever you want because you're trying to communicate with the audience. When you write an article, you're trying to communicate with the reviewers who aren't the readers. It's a very weird indirect thing.
It's. I guess similarly, if you're trying to write a TV show, you have to convince the TV network to produce the show, but they're not the people who are watching it and articles are like that too. But a book is so simple. You just write a book and you're just aiming to reach people. It's very pleasant. I recommend it. Yeah. I can see that it's something you really enjoy because you're such a prolific author. Yeah. I am.
Personally, I use MRP quite a lot and often, so I'm definitely super curious to see what's going to be in this book. I'm sure I'm going to learn things personally, and that's also going to help me teach MRP, which I'm doing from time to time. Thanks a lot. We have a research project I'm very excited about now, which is integrating survey weights into MRPs. So people do it now, though.
They'll think they'll run weighted regression or they'll do like in, they'll have the model in stand and use power likelihood, but it's not really quite right. So we have what I think is a better approach, but that's not what you have me here today, right? Here I'm supposed to talk about our active statistics book, my new book with Aki. Yeah, yeah. Yeah, exactly. I would, we can put whatever you want, but yeah, the main focus is going to be your new book. Active Statistics with Akira Etari.
And yeah, so maybe can you give us an idea of the genesis of the book and thanks for showing up the book on the video. So those watching on YouTube. So it's for people learning statistics or teaching statistics. So the story is that everybody says you Want to do active learning so students should be working together class class time should be an active time for students to be thinking about problems discussing problems. I notice so what.
Okay, I teach a class based on regression other stories and it's two semesters and each semester is 13 weeks and each week has two classes. So that's 52 classes. And we cover the book every class is an hour and a half long, or I guess, seventy five minutes long and each class. I have a story a class participation activity, a computer, a computer demonstration, some quick drills for students to work on in class, and then the discussion problem for students to talk about and think more.
I don't always have time in every class to do all of these, but sometimes I do and I can always do most of them. I found when I had been teaching statistics, I told stories a lot, but what happened, it's tricky to tell a story, partly because for other, not every teacher has a lot of experience, so they don't always have a lot of good stories. So, So, okay, so our book, it's okay.
Our book is 52 stories, 52 class participation activities, 52 computer demonstrations, et cetera, one for each class. So, first, these are 52 stories that are pretty good that I've come up with or that Aki and I have encountered in our careers. So, there are high quality stories, but also when you tell a story, when I tell a story in class, sometimes it gets a little disorganized. So, it worked good. It worked well to write the stories down.
And for each story, we very explicitly say how it connects to the week's topic, the week's reading, and also how it connects to the course as a whole. And I felt that had been missing before. It wasn't hard for me to tell an entertaining story with statistical content, but I wasn't always making that connection with what was happening in class. So I feel that if you're a student and you want to learn statistics, you can read these stories and... There are great little stories.
There aren't a lot of sources for statistics stories out there. Textbooks tend to have boring examples. They want to set it up like here's how to turn the crank. Sometimes textbooks tell stories, but they don't tell them well. And I'll give you an example of that in a moment. There isn't really anything like this. And so maybe we should have just had a little book. Our book is, how long is it? It's three and two fifty pages long.
Maybe we should have had just a book that was like 50 or 100 pages long with just the stories, because that already is great. Maybe it should have been several pamphlets rather than one book. Then we have class participation activities. These are things where the class gets involved. They're filling out survey forms or. they're doing an experiment on each other or we do an experiment on them or they're weighing bags of things and trying to get estimates, they're flipping coins. I love these.
Deb Nolan and I had a book a few years ago, Teaching Statistics, A Bag of Tricks, which had a few activities, but this is a million times better. First, we didn't have 52 activities, but also these are lined up with the course. So they go in sequence. So they're not just fun things to do. There are things that line up with particular lessons. And I just love that people tell me they'll say, Oh, I liked your book and I used one of your activities in one of my classes.
And it makes you want to scream and like, you know, throw something at the TV or punch the wall or whatever. I want you to do it in every class, every class should have an activity or at least most of the time. So that was a lot of effort because we had a bunch, but a bunch of them, like we just created from scratch. We need an activity for this. And that's really great. So that could have been its own pamphlet, another 50 pages. Then we have computer demonstrations.
And I find that live demos are great. But if you try to do it from scratch, you get tangled in the code. So it's good to have pre -written live demos. And so that's like to say you should have a demo. And it's surprisingly hard. You create even something simple, simulate fake data and run a regression. You have to have good values of the parameters or else you're not really demonstrating the point you want to make. If it has some curvature, how much to have.
So we tested them out and did them in class. And so that way when I teach, I can always have a live demo, which is everybody's favorite part of class and so forth with the others. And then we have some homework assignments and we have some chapters at the beginning where we talk about how to set up the class and how to learn better. It's not really just for teachers, as I said, should be for. students. So that's what's in it. Yeah, well, thanks a lot, Andrew.
I already have a lot of follow -up questions for you. But something also you've told me in preparing the episode is that you have thought about the book in three distinct parts. All right, so first one being the idea of statistics, regression and causal inference. Then another pillar would be like using stories to explain statistics. And the third pillar would be the method of teaching with active student participation.
So why did you choose these three different pillars and how do you think they are helping an active learning of statistics, which is one of the goals of your book? So. Teaching or learning is like a vector. It has a magnitude and a direction. And the magnitude is how hard you work to figure stuff out. And the direction is what you're learning. So yeah, I think applied regression and causal inference is super important.
This typical audience for this book would be students who took one statistics class. Maybe they already took statistics in high school or at university. took that one class where they learned about sampling and experimenting and estimation, intervals, normal distribution, stuff like this. This is all about using it, about going beyond that. So, yeah, I think applied statistics is great.
I want to teach regression about, like, the most important thing is understanding the model and being able to use it. Not so much the mathematical theorem about… least squares estimation. That's important too. There's other places to learn that. So yeah, the direction is that it's applied statistics. I think the magnitude is about how to make that work, how to get people to learn.
And so most of the learning is not done in class, but at least if students are doing these activities, in class that the hour and a half or the three hours a week they're spending in class, they are already heavily thinking about it. Which, and you know, I just like, it's kind of horrible for the students because you really make them work. It's like teaching a foreign language class, right?
If you go and take a usual class in college, you sit in the back and you zone out and you're like, oh, this is pleasant. It's like watching a movie, maybe. But if you're in a foreign language class, you're working all the time, right? The teacher's always making you talk and listen. If you lose focus for a second, it's...
Difficult statistics is a foreign language and you can learn by speaking it and practicing it So I think it's important in class to be able to do that or if you're studying at home to have these activities and stories That there isn't I mean, it's and of course the computer I'll say like my computer code is pretty bad. So that's good, right? Because that's like student code. It's all crappy code.
So it's realistic I know it's not the world's cleanest always I would say, but it runs, but maybe it doesn't all run either. It ran when I wrote it. But it's supposed to be, when I do code demos in class, what I like to do is actually type in the code, not copy and paste it. So that's modeling how someone might do it. So we try to keep them short enough that you can do that. Yeah, thanks a lot.
I see what you're doing and I really appreciate it because that's also helping me in my own teaching philosophy because I do have the same experience where the students who end up learning the most are usually the most active ones. but then the main question is, okay, how do I make them all active? Or at least give them the opportunity to all be active. And that's really one of the things. Yeah, when I teach, I make them talk.
Like even it could be a class with 50 or more students, but I'll tell the story and then I'll pause and then say, well, what do you think? Talk to your neighbor about this. And I look and I make sure they're talking. And if they're not talking, then I walk over and say, you know, I go like this to them. and if their computer is out by look and if they're on their social media, I ask them to close their computer and if their phone is out, I ask them to close their phone and so forth.
The funny thing is as a teacher, that's hard, it's easier as a teacher to just talk and talk and talk and talk, like I'm talking now, I'm just talking. It's easy to talk and you have complete control over it. So that's why I really needed to structure this in this way. That was my original motivation for all of this.
was that many years ago I was teaching a class and I couldn't make it because I had my co -teacher, another faculty member in the department was teaching the same level class, teach my class, and then I went and taught hers and she said, oh, your students were just dead. And then I talked to her class and they were so lively and I realized not that she was lucky, but that they had been in that habit of participating in class. She's just a natural great teacher. I'm naturally not a good teacher.
And so I... do this stick in order to get them involved. And then I just wanted to do it well. I want to tell stories, but I want to be able to make the point, to help them learn it. Yeah, that's interesting because me, when the teachers were doing that to me, it's because I was talking too much. That happened quite a lot. Maybe that's why I have a podcast now. Apart from these philosophical considerations. Yeah, that's very interesting. I'm going to try that in my own classes.
The thing is I personally teach a lot of online courses and so I cannot beender and see the screens. So that's pretty hard. Yeah, it's tough.
I remember when I was doing the class over Zoom and you could try to put them in a little room so they work in pairs, but yet if you can't see them doing it, I think there is some online... conferencing software where you can actually see the pairs and then then or the small groups, but I don't I don't know the full story with that, but I could get so I gave you an example. There's something one of the things it's difficult. I don't know.
There's any answer about this about the stories is that if they're too if they're too simple, that's boring. But if they're too complicated, then you know, that's not good either. One thing I like to say like I I want to send the message that. Statistics, how did I put it in the book? I had a slogan that statistics is hard. It should not feel tricky. So I don't like those. I don't like this.
I like statistics stories with a twist, but I don't like the kind of stories where the messages, this is just hard like this, like at Monte Hall problem. I hate that because it's just so confusing to people. Like, what's the lesson that you're teaching? Right? Like, this is really, really confusing. I don't want to teach that. But here's an example. And this is a very standard example used in United States statistics classes where we put another twist on it based on the recent literature.
So this was a survey that was done in 1936 by a magazine called the Literary Digest. And they did a very famous in statistics books example. They did a survey for the presidential election and it was the presidential election was Franklin Roosevelt running for reelection against somebody who wasn't Franklin Roosevelt. So you kind of know who won that election. But in the their poll, actually, Franklin Roosevelt was going to get destroyed.
They did a poll with they they surveyed 10 million people and two and a half million of those responded. And out of that, it looked like Roosevelt was completely getting smoked. Well, there were two things happening. One is the two and a half million respondents were not random sample of the 10 million people. Second, the 10 million people were themselves not representative of Americans because it was from lists of people who own cars and things like richer people.
So it wasn't a representative sample and usually it just stops there. But that's not a good place to stop for a couple of reasons. One of which is what lesson are you telling people? If you don't have a random sample, your survey is no good. Well, unfortunately, no surveys are random samples. I mean, no surveys of humans, no political polls are. So the message would be, oh, you can't ever trust any political poll.
Well, that would be a mistake because political polls, even when they're off, they tend only to be off by a couple of percentage points. So what goes on with political? Well, so let's OK, so let's look at this survey. The first thing is that the same magazine had done this survey in previous elections and it had worked well. So they had some track record. It wasn't as dumb as it sounds.
Second thing, and this is something that two statisticians recently looked into, I was able to take advantage of their work. So Sharon Lore and Michael Brick had written a paper on this 1936 Literary Digest Survey where they realized that, or the data from the survey are actually somewhere, like they're available. The, um, and one of the quest, the survey asked people who they would vote for, but it also asked who they voted for in the previous election.
So you can adjust for that because you know, the election outcome, the previous election outcome. Well, it's not perfect. It's not everybody voted in the previous election. And, but it's pretty good. And when you do that adjustment, you get, well, you find that Roosevelt was supposed to win. Well, it's not a perfect adjustment. It's still quite a bit off. It's. Even after doing this adjustment, it's still not a representative sample.
But now we've changed the lesson from, hey, it's not a random sample, you fool, blah, blah, blah, to, hey, this sample is not a representative sample, but statistics can be used to adjust it. Look at this. But the adjustment is imperfect. So it's a more subtle message. Well, it's trickier to teach. That's one reason why I like having the story written as a story very clearly in the book, because then the student or the teacher can read through the whole thing.
If you're a student, you can read it through. And if you're a teacher, you can first read it before trying to teach it. And there it is. It's on page 36 and 37 of our book. There's a copy of the survey form. And. It takes it. It's it's literally like the takes up the description takes up one one page of of the book. Almost almost all of it is a quote from Lauren Brick because they're the ones who did it and then a little discussion of how it relates to the class.
But everything is like these stories are all like that. Like they're all you have to balance it. And it's it's it's tricky like they almost should be another. booklet of the really simple stories that we've been including because they're too boring for me, but maybe still interesting for the students. I don't know. We went back and forth. It's structured from beginning to end of the course.
So each sec, there's 20, well, there's a couple of introductory chapters and then there's 13 sections for the first semester and then 13 sections for the second semester. So most of the book is, is 13 straight, is 26 sections. And in each one we have a story and the participation activity. And we went back and forth about whether to do it that way or whether to put all the stories in one place and all the activities in one place. And I don't know. Now I'm thinking I wish we had done it that way.
But Aki and I went around and around on this a million times. There's no, you don't need to hear about this. I wanted it to look right. The thing is, if you opened up at random, you might get a page of homework assignments and then it might look like a textbook. So it's like, that's the... it all kind of looks the same. So maybe if we had separately done the different things, it would have then there'd be a whole section of stories.
But when you're teaching, it's convenient that's in order because you just go to the week of your class and then you can see what to do that week. So that's, I used it to teach. Yeah, and I mean, I really love also your focus on the stories, right? I see it's definitely a theme of your work recently, and I really love that because I think it also puts an emphasis on the fact that statistics is not done in a vacuum, right? And it's also done by humans.
with their biases and also their motivations and so on. And I found that way more interesting, way more realistic. And also that captures more the imagination of the students rather than teaching them theorems and formula, which often is quite intimidating to a lot of them. So yeah, I hope to admit the stories are like all things that I can personally relate to. Like either there are things that I was, I was either it's research I was involved in or it's something close enough to what I do.
Like I'm interested in the question being asked. Um, it's yeah, there were, there were, and the same with the same with the activities. The activities have a lot of simulated data. I'm a big fan of. Yeah, you are. Uh, in a, in a lot of your books, you, you took up with that. Um, do you want to, do you want to talk about. bit more about that or you think we've covered already the idea of simulated data in the traditional data?
Well, I'll just say briefly that I think we are, as statisticians or computer scientists or whatever, we're used to the idea of here is a data set, let's see what we can learn. But science, I mean, sometimes we proceed that way in learning. We want to understand the world, you're curious about something, someone gets a bunch of data from Basketball or whatever, and then you play around and see what you can get. So that happens, but often things are more directly motivated.
Like, yes, in a public opinion poll, you're really starting with the question. When in demonstrating a method encoding examples, it's super great to have simulation. partly because it's like it's the dual problem, right? If I can, I simulate the data, then I fit the model. I can check, I can see if the parameter estimates are similar to the true value, but also just the active simulation is the time reversal of the active inference. So it makes sense to show the forward process too.
And I think it's kind of a bit of a power thing. It's a student, like I can state, I can simulate data. I can make fake data myself, right? That's. That's something that can be done. Traditionally, we do simulation when we're teaching probability, like you'll teach the central limit theorem by simulating draws. But just a lot of examples come up. It's very simulation is a kind of it's like a universal solvent.
Like, for example, I think one of our discussion problems in classes, I show them data from some regression, which is based on real data. And I don't remember the example, but something where there's some treatment effect. which you maybe expect is positive. Maybe the estimate is, let's say the estimate is 0 .3 and the standard error is 0 .2. And so then I say, and it's based on 100 data points. So then I, so it's estimates, estimate is 0 .3, the standard error is 0 .2.
So I'd say how large a sample would you need to get a result that's two standard errors away from zero? That's statistically significant, a term that I don't like to use, but of course they need to know how it gets used. So you'd say, oh well, the standard error is 2, but really the standard error would have to be 1 and 1 half for it to be 2 standard errors away from 0. So the sample size would have to increase by a factor of 2 divided by 1 .5 squared.
So you take 2 over 1 .5 squared, and that's, you know, so you can do that, you know, and you say here, 2 over 1 .5 squared times 100, and that's 177. So you'd say, well, you need a sample size of 177 to get your estimate to be true. So work that out. That's wrong. That's not the correct answer. Because if you redo a study with 177 people, there's no reason to think the point estimate will be the same.
In fact, Like the whole point of saying that the estimate is less than two standard errors away from zero and you don't know whether to believe it, somehow the whole point from a Bayesian point of view, the point is that it's likely to be closer to zero. From a classical point of view, the idea is that you can't rule out zero as an explanation and zero is like typically a privileged value there.
So if you're replicating a study or even doing it longer, you would have to, the answer depends on the true treatment effect, not on the coefficient estimate. And well, that's harder, right? But the point is you can show that with a simulation. If it's based on real data, it's trickier to show because what are you doing? But if I then do a simulation and then I say, well, look, let me try simulating 100. with this true treatment effect and then I see what I get.
I say, well, shoot, I didn't get a treatment effect of 0 .3. I was supposed to have to keep doing it. And then you realize you're selecting just some. So to me, it brings it to life. The applied point gets demonstrated in a way that's harder to do with just one data set. Yeah. Yeah, yeah, yeah. I really love that. I agree. And that's also something I tend to use. On a lot of questions people have on, you know, A, B tests, settings, things like that.
There's a lot of questions about these, the sample size, the iteration, things like that. And I find personally, I have to do the simulated data studies to answer these kinds of questions. Like I, I'm bad at like remembering, you know, all those rules are awesome. Like, like let's do that kind of studies with simulated data and that gives me a way better idea. So in a completely unrelated topic, I can tell you about our two truths and a lie example. That's a demonstration we do.
I'm mentioning that partly because writing a book is like writing a hundred articles. So at one point I thought, well, maybe I should publish these as a hundred articles because each story could be, well, that just takes a lot of work and maybe more people will read it in book form. So I didn't do that, but. I did one of them. I did one or maybe I did one or two. It takes a while to publish an article.
And for the bad reason that it's just formatted in a different way, for the moderately good reason that you need to explain more if it's in an article rather than a book because you need the context, for the pretty good reason that you're forced to that, that like you have an opportunity to expand because you have more space in the book. I can't take up too much. I can't have each thing take too long.
And for the probably the biggest thing is you get useful reviewer comments and people point out problems anyway. So the one of the the activities I did write up as an article was two truths and a lie. And I gave a link to the article version, which is longer than what's in the book. But I love the story. OK, is the story how it came out is that there's this game which did not exist when I was a child. But I don't know if they do it in Europe. It's a big it was it's popular in. in the U .S.
as the kids do it as an icebreaker in class, you'll have a group of people and one person is the storyteller and this person tells three things about themselves. Two of them have to be true and one has to be a lie and then the other people discuss and try to figure out which is the truth or which is the lie. So it's such a fun activity. I like to use it as an icebreaker in my statistics class. But it has no statistics content.
I mean, it is because there's uncertainty, but what do you do with it? So I thought about and thought about and well, I decided to put it in the second semester. I was ready for a good icebreaker and the second semester started with logistic regression. Okay, I can make it logistic regression problem because you can say, what's the probability you get it right? What's the probability you guess correct? But then you need some predictor. So, oh, predictor.
Well, you can have when you guess, you also have to give a certainty score, some number between zero and 10 representing how certain you are that you're correct. Then it has to be done in groups. So I figured it out. Each, you divide the class into groups of four. Usually we do pairs, but this one, four. Each group, you have one student is the storyteller, tells the three statements. The other three discuss together.
And then, come up with a guess of which they think is true, which of them that they think is a lie, and a certainty score. So write the certainty score down in a sheet of paper, then find out whether your guess was correct and write that down too. So they find out. Then there's four of you in the group, so you rotate. Then the next person does it. So as a result, as a group, each group has four certainty scores and four. correct or incorrect answers.
So they have four numbers, they have eight numbers, first four numbers between zero and 10, and then the four numbers which are zeros and ones. And so, by the way, when you do this, I have a slide prepared, or I write it on the board, the exact instructions. You need to give in, you can't just tell it, people aren't paying attention for one second. I'm just doing this for you in that thing, but actually we have the instructions there. Then did this thing I discovered a couple of years ago.
It's putting things on Google Forms. So live in class, I create a Google Form, I open Google, type it in right there. So this is also it's a power thing for them. Look at this. I didn't have to prepare this. I type the Google Form, I put question one, certainty score, make it a response from zero to 10. Question two, yes or no, did you get it? Was your guess correct? So with each group, I want you to go, oh, and then we use tiny URL to get a URL.
And then for each group, I say, pull out your phone or your computer, and one person from the group, enter your four data points. So we set it up with four. So there's actually eight responses, the first one, the first one. Then we get the data, it takes them a minute to type it in. Then I have it all prepared. I've done it before, right? So I have the code ready. I. So I go to the Google page, I download it, I put it on the desktops.
It's not even my laptop, it's just a computer that's in the classroom. Then I go, I open R, I read it in, and I have the code prepared so I can do it. And then we can make graphs. So we fit a legit, so, but then I did something I always like to do. I set it all up. Okay, we have the data. I type in the code for logistic regression. Again, I have a pause. I say, well, write the code with your neighbor what the logistic regression code would look like.
So, yeah, and then I do it and then I type it and I said, then I do display, you know, of the fitted regression. And before hitting carriage return, I said, this is what it's going to look like. There's going to be coefficient estimate, standard error. What are they going to be? You and your neighbor have to figure out, try to guess what the estimate and the standard error are gonna be. Well, the standard error is tricky, like that's hard.
So I said, just figure out, guess what the estimate will be. And so then I have them do it, I go around the room, I make sure they're all drawing the curve, and then I have someone go on the board and draw what they had done. And then I ask people, do you think this is reasonable? Do you think this slope is reasonable? Now what do you think the standard error will be? Do you think the slope will be more than two standard errors away from zero? Then you fit it.
and you have the scatter plot and they can see and they've thought about that committed to it. So that's logistic regression. But when I wrote up the article, the people in the journal said, well, what about other classes? And then I realized you can use this to teach measurement. You can use it to teach experimentation, like all sorts of things. You could do a lot with that. But I felt so satisfied because just I felt like it was just created out of nothing.
I wanted to true Snellai activity and now there is one. So that was just felt so it felt so good to have created. Now I want everyone to do it because now that I created this this beautiful thing out of nothing, it did not exist. Anyway, just I'm very happy about that. Yeah, I love that. I definitely tried that in my own. My own classes seems like a good thing to do on the first or second class, isn't it? Right, exactly. Now the point is that you're killing two birds there. Yeah, yeah.
No, that's super cool. Definitely going to try that for sure. So, and it's like, I have a commencement device now. I have officially publicly committed to do that. So I have to do it and then. Come back to you, Andrew, to tell you how it went.
The other thing you can do is there are certain fun psychology experiments from the literature that can be done in class, because things that have very large effects, like some of the classic Tversky, Kahneman experiments of cognitive illusions, we have one of those examples too. You can do it live in class. Yeah, that sounds also super cool. I also saw in preparing the episode that you have a flipped classroom, like you emphasize a flipped classroom environment.
I don't think I've ever heard you talk about that. Could you explain what this approach is and how you think that enhances the learning of client progression and calls on inference? I think to me the flipped classroom is pretty much the same as traditional high school classes, high school math class. So if you take math in high school, you have a book you're supposed to read and there's homework assignments. Usually you read just enough of the book to allow you to do the homework assignments.
Then in class, the teacher does a couple things in the board and most of the time in class you spend working on problems in pairs or small groups and then people go up to the board and share their answers. That's kind of what I think should be. So that's the model of so it's very traditional. The flipping part is, you know, I don't have videos. I guess I could, but I don't. Akki has videos for his glasses that I have. But the flip part is the reading.
Right. So they I'm not lecturing because they're supposed to have read the book. Now, what happens, you know, it works only if you have a book that you can can lean on. But I think that's very important. This semester, I'm teaching in a statistics class teaching some multi -level modeling and some other things. My book with Aki and Jennifer on advanced regression and multi -level modeling doesn't exist yet. It's supposed to be the updated version of my book with Jennifer.
I couldn't quite bring myself to teach out of my book with Jennifer just because the code is old, but then I don't have a new book. And so as a result, the class I'm teaching this semester, It's fun.
I think the students are enjoying it, but I'm not it's not going as perfectly as it could because I can't really do the flip thing because I keep I end up spending a lot of time in class like my computer demos typically end up being me doing the homeworks, working them out the homeworks that were just do which is fine, but it's it's not they're a little bit more elaborate than. Ideally, I think computer demos would be shorter.
They don't have enough to read before, so I end up spending a lot of time lecturing. I think I spend most of today's class just talking. I felt a little bad about that. I don't know. I think it's still fine. It's still a breath of fresh air compared to other classes they're taking. I'm sure if all the classes were like mine, then that would be horrible. But an occasional class that's like mine can be good. I think in general, students like more organization. A book is better.
Even my My when I teach ever regression other stories that's super organized, but it's not always what students want because they want to set up methods and formulas and theorems and so forth. So I'm not always giving people what they want. Anyway, I think that they again, I think they're really looking for very clear. I don't I have this thing, the goal is to be fluent in the foreign language, but I don't think people usually think of it that way. I think that they're looking for.
something different. But what that means is that it puts a special burden on me to be super organized because if I'm not super organized, then I think students will not see the point. So my class this semester, it doesn't use the book. It's not as flipped as it could be. I still have them talking with each other in class, but not having the flipped classroom makes it a little more of a passive experience for them.
And then when I do have them talking, they're often just talking to each other saying, oh, I have no idea what's going on here. It's like, oh, good that I know that, I guess. That's true. Yeah. And I mean, I do relate to this idea of the, you know, getting fluent in a foreign language. That's actually also a metaphor I use quite a lot to people who are curious about what the... work of a statistical modeler is.
And that's funny because there's that weird human brain bias of just thinking that someone who is doing something that looks hard to you, or they must have been good at it since the beginning. And at least for me, it couldn't be further from the truth. It comes from a lot. As you were saying, I think you were saying learning is a Vector is magnitude and direction, right? So definitely magnitude is very important for me each time I learn something.
And often I'm saying, yeah, well, it looks hard because you have to learn kind of two languages, the language of stats and the language, like the actual programming language that you need to do the stats. But it's just as any other language, you need to... talk to people in that language and with time you'll see your brain just getting there. So it does go through to people, but at the same time they need to see some results along the way because otherwise the motivation is gonna fall down.
So it's always that needle that's a bit hard to thread in my experience. Yeah, well, I like this book. See, I seriously think this book is just fun to read. Although, as I said, I kind of I kind of wish I had separated it out in a different way because I do feel when people when you open at random, you end up you might see some code or you might see a homework assignment or you might like it's not always clear what like you're not necessarily opening into a middle of a story.
And so like homework assignments don't look like fun and code doesn't look like fun.
So I'm. Don't think I realized you don't see the book until it's a book before that's this PDF on the screen and it has it has a different experience that way and and Akki's gonna kill me that I say this because we went back and forth and and but like now I think we really should have of I really think we made a mistake by not doing it the other way because I think it would look a lot more fun that way if If like all the stories were in one place and all the activities were in another place
I'm really feeling bad about that. I still love it. It's just, we just have so many fun things. Oh, then we have, for the final exam, we made, it's multiple choice. So what I do is I have four or more questions per chapter. It's like, it's, it's, The exam has so there's 12 chapters for the fall and 12 for the spring. So each chapter, I have four or more questions. What I do is I randomly sample one per chapter and give that to the students as their practice exam.
Then I randomly sample two per chapter and give that and make that the final exam. So therefore, by construction, the practice exam is representative of the final exam because they're two random samples from the same population. So I think that's that that's great to be able to do that now. Of course, all the problems are now in the book, although without the answers. So you'd have to figure out which it is. But in theory, someone could read through all of those.
But of course, the usual story is if someone really goes to the trouble of reading through all of them and figuring them all out, that's probably good anyway. So I don't mind if they didn't do well on the exam. But it took a lot of effort to write. These multiple choice questions are hard to write, but I think they're easier to grade. And I think they're testing something that's a bit more focused. It's very easy to write open -ended questions and not know what you're testing.
True. Yeah. Yeah. It's a bit more like astrology, where you always find something you're satisfied about. Yeah, yeah, exactly. And it also encourages a certain behavior among students to just keep writing and trying to like touch all the bases. True. Yeah, yeah. As a pure product of the French educational system, I can tell you open ended questions are like my bread and butter. I've been trained at that a lot. So if someone have to answer, like I have a weird feeling of familiarity and that...
At the same time, I like it and I dread it. So that's what... Many years ago, I taught a class in France and the students are supposed to do projects and it just happened. Yeah, everybody's busy. So one of the groups did, they did nothing. They turned something in, which was pretty much they had just like, it wasn't plagiarized, but they had just copied stuff from the internet. Like, you know, they just literally copied some images and it was essentially nothing. So I talked to the...
The head instructor of the class, I said, well, I want to give him a two out of 20 on this. Like, I guess, you know, I, I, maybe I don't give them zero because they wrote out sentence or two, but like, can I, can I give them a two out of 20? He said, well, yeah, you're giving the grade. I said, in the U S if you want to give someone a low grade, you have to ask for permission because you're afraid they might sue you or complain or something.
And, but he said, no, in France, you can give people, you know, two out of 20. They might even think it's a good grade. So it is a different... French system is a little more rough in how the grading goes. I don't remember that. Yeah. I mean, it depends. I don't know at what level you're teaching, but if you're teaching in the... especially in the class préparatoire, you know, so that weird stuff we have in between high school and universities. These were graduate students.
Yeah. So you can definitely do that. I know I was like my first philosophy... dissertations when I was in the class, were absolutely a disaster. Um, it was, that was, I think I got four out of 20, something like that. And that was not even the worst grades. You know how like in gymnastics, like it's like 9 .8, 9 .9, 9 .93, like that, like the grading system did that. But statistics is, it's really hard. Like I think real world problems, I wouldn't give myself.
a 20 out of 20 in my analysis, because if you're doing an experiment in political science or psychology or economics or an observational study, everybody knows about identification being difficulty, but there's a lot of other difficulties. So usually if you're doing a causal study, you wanna have between person comparisons, or in political science or economics, it would be called panel study. You wanna have... Ideally, you do the treatment and the control on each person.
But if you can't do that, you want to make comparisons. That's super important, partly for statistical efficiency and for balance. And it's also kind of a measurement issue because measurements can be biased and biases can actually like the treatment effect. The treatment can affect the measurement bias and you can even have treatments that affect the measurement bias without affecting the outcome. Like, it's so naive view that if you just.
give randomly assigned treatment and control that you have a kosher estimate, the causal effect, that's not really right in general, because that assumes that the measurement bias doesn't vary with the treatment, and that's often a mistake. So you really want to have panel structure or repeated measurements with in -person designs. That means you want to start setting multilevel models.
So if you don't have a lot of observations or a lot of groups, then your inferences can depend on the prior, which it really does. You can't, you could act really tough and say, oh, I'm really tough. I'm not using a prior, but then it just means your inference is really noisy. And that's, that's not good either. It means you can get bad things. And then what predictors to include in theory, everything should be interacted with everything because otherwise that can induce bias.
But in practice, if you do that, you have a lot of the coefficients running around. So even the simplest problems are like, like there's no right way of doing it. which gives me a lot of sympathy for researchers. And I know here we're not talking about like the crisis in science, but I'll say that like sometimes people will say that you should pre -register your design and analysis.
And I think that's great, but it's not gonna solve a lot of problems because if I don't know the right analysis to do, I don't know what I'm supposed to be pre -registering. It's really difficult. It's not, we can't just do better science by just like. Like there's this phrase, questionable research practices. Like it's not like you can just stop doing questionable research practices and everything will be okay. It's not clear. Doing it right is not just the absence of making mistakes.
It's very difficult. And so when we're teaching or when you're learning, I'll say, cause I really would like our book to be read by people who are not necessarily teaching a class, but just want to learn the stuff that when you're learning, there is this. weird thing where you have to learn the skills and at the same time realize the limitations. And it is, it's hard to teach in that way.
It's not like, it's easier to teach something like physics or chemistry where you say, here's what we're doing. And then later on, we're going to tell you why these ideas aren't correct. And we're going to do something more elaborate in statistics. It's hard to reach that like plateau where you say, well, here's the basics, learn the basics. Once you're learning the basics, you keep seeing all the problems at the same time. So it makes it very fun to learn, but also challenging. Yeah, true.
Yeah. And actually that makes me wonder, how do you think, so for people who are going to use your book for teaching, so instructors, how can they adapt the materials for different educational settings like... such as introductory course or more advanced courses. So it's set up for this class on applied regression and causal inference. So if you're teaching out of regression and other stories, it's very easy. It just gives you a whole template for a two semester class.
I've also taught a one semester version where I just do one activity and each week I have two of everything. So instead I just pick one story, one activity and so forth. That's what actually I did. Last semester, if it's a more advanced class, and I would say, or or more basic, if it's a more basic class, I think it's still pretty much works. You just have to simplify the code demonstrations are going to be way too complicated for more basic class.
But I think the stories work and the activities work. You just maybe have to change it a little. So. In two truths and a lie, you wouldn't do logistic regression, but for example, you could still make a scatter plot and you could still compare the probability, the proportion of correct guesses for people's certainty scores higher than five or lower than five. You can adapt it. I think a lot of the activities are like that in the stories.
For more advanced class, I think again, it works in the other direction that this can be a starting point. You give the story and... And also people have their own stories. So reading my story might help you as a teacher, think of your own story and tell it in the same way. Yeah. Okay. Yeah, I see what you mean. I'm thinking randomly. It sounds like you would be interested in Andrew at some point in writing some fictional stats -based stories.
something like, I think Carl Sagan, right, did write some science fiction. Would you be like, do you see yourself doing that at some point so that you are forced to maybe not use any modeling or things like that in the book and you have to completely only tell stats through the stories and all? Well, well, fake data for sure. I did have an idea.
I was thinking about having a book where it's all like it's learning statistics through fake data simulation where everything is just you just start with some very simple things like everything that's like the gimmick right the gimmick is here all the principles of probability and statistics and you're only you're not allowed to use any real data you're only allowed to do fake data simulation and you can cover a lot like all sorts of things the the
attenuation of the of the code the treatment effect when you have measurement error in your predictor and Like anyway, all sorts of things you might want to cover. You could do that way. So I thought that would be fun. Maybe a fun future book. I mean, fiction, you know, Jessica and I wrote a play, Jessica Holman and I wrote a play recursion, which is fiction. It has computer science theme. It was performed at a computer science conference recently. So, so I guess, yeah, we have written fiction.
It didn't really have, it had some statistical principles in there. There were, there were some, it had some. Like we, yeah, I think we had some line where one of the characters talked about their code being beautiful, and then somebody else said, code that runs is beautiful. And then somebody else says, code that runs and you know it runs is beautiful. So that's like some workflow principle. So we were able to put in some of our thoughts about statistical workflow in fiction.
So yeah, it's possible. I knew it. I knew it. Yeah. I love to hear that. I love to hear that. Read that book. And I was saying here, because I think, and you could even record the audio version yourself. I think that'd be awesome. Yeah. Well, that performance apparently went well, but they didn't video it. So we want to get it performed somewhere else. Yeah. Well, let's try that. If there is... One day if I manage to do a live LBS dinner, that should definitely be performed at that dinner.
That's a must. Now I'd like to ask you something about, I know a topic that's dear to your heart is visualization and it's time to understanding. Because... the focus on visualization is a key aspect of your book, Active Statistics. It's also a key aspect of almost all your work. So I'd like to hear your thought about that. How do you think visualization aids in the comprehension of statistics and cost of models? Well, so I'll talk about two things.
First, visualization in teaching and second, visualization in statistical. Like applied statistics. So with teaching, I think like I think the deterministic part is usually the more important part of the model. So I want people to be able to visualize what is the line? Why goes a plus BX? What what does it look like if I have an interaction? What would the two lines look like? What is logistic curve look like?
I I don't I think it's a mistake when statistics books start with things like a histogram. Histogram is not fundamental. Actually, it's very confusing. I used to do this assignment where I would say to students, gather between 30 and 50 data points on anything and make a histogram of it. And about half the students would do it. Like they might gather data on 30 countries or 50 states, or they might take 30 observations of something and make a histogram. The other half would.
make a bar chart showing their 30 observations in time order. So it would be like, basically it was a time series except it would just be displayed in bars because it was a histogram. And so like you see the problem is that a histogram is supposed to convey a distribution, but what people are getting out of it is it looks like a bunch of bars and half the students didn't get the point. The concept of a distribution is very abstract because...
The height of the bar represents the number of cases or the proportion of cases. It's not like a scatter plot. I think it's actually more intuitive. But I noticed that statistics classes were always focusing on that because, oh, histogram is one dimensional. What could be more simple than that? I think a time series is really much more basic. So when it comes to plotting data, I think we really have to get a little closer to what we care about. Um, a lot of just stupid stuff, like box plots.
I hate that. I hate that stuff. And it's like, I don't see it. It's just like, people just do things that are conventional and I think are absolutely horrible. But anyway, all this focus on distributions, I think the linear, the deterministic part of the model is more important. And so that's what I try to convey. I do. One thing I noticed is that students will learn stuff if it's on the homework and on the exam. They won't learn it just because it's on the blackboard in class or in your slides.
So I found that when I did my work, I often make sketches of graphs. And so I require like I have homework assignments where you have to make a sketch, sketch what you think it's going to look like, then fit the model. Because if you don't ask people to do that, they won't. So teaching has to be. Like you want people to actually practice that kind of workflow. So that's then I had something else to say, but I won't. We can say it another time about statistical graphics.
It's already kind of going on a little bit. So if we ever talk about statistical graphics again, just ask me to tell you what I think is this really super important aspect of statistical graphics within statistical inference. And I'll tell you about that. Okay, perfect. Well, definitely. Definitely tell you. Do you still have time for one or two questions or should we? Yeah, sure. I have time for one or two questions, sure. Okay, awesome. Let's continue. I'm curious about that.
How do you handle the distinction and or the transition from regression analysis to causal inference? How do you navigate these two topics in the classroom setting? to ensure that students grasp both concepts effectively. So I overlap. So I start talking about causal inference at the very beginning, partly because they can't avoid it. So we'll have a regression. Maybe you fit one of the examples we use in regression, other stories is predicting from some survey, predicting earnings from height.
Taller people make a little bit more money than... shorter people and then you can also you can throw sex into the model and men make more money than women taller men. So you can say how do you interpret the coefficient of height? Well if you're one, you know for every inch taller you make this much more money. So that's not right.
You have to say comparing two people of the same sex one of whom is one inch taller than the other under the model on average the taller person will be making this much more money. So what are the things you need to say? You have to say comparing, because it's all comparative. There's no causal language. You have to say, on average, you have to say according to the model. And you have to say not controlling for blah, blah, but comparing to people who are the same in these other predictors.
You're not holding everything else constant. You're doing this comparison. So I do this, I have a drilling class where they have to do it. I can then they laugh. It's like a joke as I say, here's a regression, explain each coefficient of words. And they say, like, what's the coefficient of the intercept of this model? It's like something I'm predicting something as a function of time. So this says in the year Jesus was born, this is well, that's the intercept right at year zero.
So is that interpretable? Well, maybe it's interpretable. If you have a time series going from 1900 to 2000, maybe we're not particularly interested in what happened when the year Jesus was born. That's a bit of an extrapolation that implies. So, but same with the coefficient. So it's like a joke in class. It's a fun inside joke we have in class that I'll ask them to explain the regression coefficient and they have to say it without using the wrong language.
And it's like, It's like the game you play as a kid where like you're not like you say like you're not allowed to say the word no. If you say the word no, you lose. You have to figure out a way to decline. Will you give me your cake? I choose not to give you your cake. You know, like I choose to do something else or whatever. So similarly, you're not allowed to use this word. And so right away, we're introducing the idea that causation is important.
And. Then when we get the causal inference, well, we have regression already. So we use that not for controlling for things, but for adjusting for things. So we've already done non -causal examples, like the survey example, where we adjust for differences in order to post stratify. So then it fits in. So there's a lot of specific things about causal inference, but we first half is we don't cheat at the beginning. We don't pretend to be causal when we're not.
Then when we get to causal inference, We make use of what we've already done rather than treating it as an entirely new topic. My little particular pet thing is that the usual way causal inference is taught is there's an outcome and a treatment. And some people get the treatment, some get the control. I say the basic is there's pre -test measurement, a treatment, and an outcome, and that's in time order. So it introduces time. You don't have to have a pre -test, but you should.
And so it's good practice, but also it... It puts you into the regression framework already, which is helpful. So sometimes things that are too simple are harder to understand. A little context can help. Yeah, I found so the... The Dirichlet graphs do help quite a lot in teaching the causal inference concepts, especially because you can then... marry that with the graphical representation of the Bayesian model that you can come up with. And then you use simulated data.
You can come up with the model, then write the model, and then just simulate data and see what the model tells you. And if it's able to recover the true parameters, I find these fit pretty well together in the workflow. Good. Yeah, I think there's a lot of different ways of teaching these things and using these. There are different frameworks that can work well. And I think that's good that that's the case. There's more than one way of explaining things and understanding things. Yeah, true.
Actually, I'm curious, based on the methodologies and... Also, the philosophies that present in active statistics, how do you see the future of statistical education evolving, particularly with the advent of new technologies? And how do you see that play out in the coming years? I don't know. I mean, I'm still unhappy with how statistics is usually taught. So introductory statistics, it's really been...
Like the textbooks now are almost all pretty much the same as the textbooks from 40 years ago. I mean, they look different, but it's based on this thing where they teach, like there is this, they teach these distributions and it, so it starts by focusing on variation, which I think is not even really quite right. And then, it's not really focusing on the questions that are being asked, it's really focused on the error term.
And then there's all this stuff about the sampling distribution of the sample mean, which is just kind of weird. Nobody cares about the sample mean and or rarely do. It becomes very abstract and hard to follow. And then there are these like confidence intervals, like a huge amount of work to create these little summaries that you don't really want to be using along with a bunch of messages. If you don't have random assignment, you're screwed. If you don't have random sampling, you're screwed.
Then at the end, there's some stuff like regression and Chi -squared tests and things that people do. And it's just kind of a disaster. I really, I really hate it. And I, I would like things to be much more focused on the questions being asked. It's hard for me to think exactly how to construct the introductory class to do this. But for the second class in statistics, like the one that we teach on applied regression and causal inference, I do like how we do it in regression and other stories.
I feel like we developed through the models. in a way that makes sense. I try to do that in active statistics. But really, the most important part of teaching are the most basic classes. And there, we're still working on how to do that. So I don't really know what the future is. There's a lot of statistics and machine learning methods out there, but a lot of... basic concepts, of course, are still coming up no matter how you do it, like issues of adjustment and bias and variation.
So it's hard, it is hard to get it all like feel like it's all in one place. It's frustrating. Yeah. Yeah. Yeah. Now I agree with that. I'm also asking the question because I'm pretty curious about it because I'm also personally a bit lost when I start thinking about these things. It's so cute. And, uh, Like for now, I don't have a clear organization in my head, you know.
Maybe one last question for you, Andrew, before I let you go, because you've already been extremely generous with your time and you know me, I could really interview you for like three hours, no problem. I have so many questions. But maybe what's next for you? What are your coming projects in maybe in the, in this coming year? Well, we're trying to finish. Well, Aki and I are trying to finish our Bayesian workflow book, and we'd like to do our advanced regression and multilevel models book.
It would be fun to get recursion performed somewhere by some university theater group somewhere. Doing this research on combining, you know, multilevel regression and post -traffication and with sampling weights, which I think is really important. And I think also this could be useful for causal inference too, because people use weighting there. So that's probably the one project I'm most excited about from that direction. And then we're trying to write. I have a list.
I have on my web page, I have a list of published, unpublished, and unwritten research articles. So the unwritten is a list of like, things that I want to do or write up. So there's a long list of that. I'm collaborating with an economist. We're trying to create a unified framework for causal inference for panel data, which really includes things like before -after studies and regression discontinuities and difference and difference and just regular regression, time series. I have a...
Like just as a simple example, if you're doing linear regression, like you have a pretest, you regress, you condition on the pretest, you adjust for that, really. But if you have a, usually things in Econ, like things are measured with error. And so you won't really want to regress on the pretest. What you really want to do is regress on the latent value that the pretest is a measurement of. Well, you can do that in Stan now.
So now in Stan, you can write these models and do Bayesian models with latent variables and. I think there's some theoretical results to be done to show how or see how these things reduce to other things in special cases. It's a little related to my chickens paper that I did a couple of years ago, which I really enjoyed. That's another story. The chicken story is not in the Act of Statistics book. I don't think it's like there's more stories. There's room for another 52 stories, I'm sure.
in the future. Yeah, for sure. And the, yeah, we should link to your chicken paper, actually, in the show notes. I like the chicken paper. It's not the world's most readable. I mean, it's technical, but I like it. It's Bayesian. It's good. Yeah. Is it, are you referencing the one from 2021? Or is that... Yeah, yeah. Slamming the sham. A Bayesian model for adaptive adjustment with noisy control data. Yeah, it's published in Statistics in Medicine, which like a journal, nobody reads.
But what can you do? I guess nobody reads any journal anymore. So that's fine, perhaps. Nobody reads anything. Nobody reads anything. They're too busy reading stuff. Yeah, I mean, definitely that's why it's very good that you come on the show. And also that you write these books. I think it's extremely important because definitely the general public doesn't read paper. I know I do read paper, but it's mainly because I have to for my job.
I almost never read a paper by pleasure because it's just like, yeah, the way it's written is just like so dry, you know, and I really love a story, as you were saying. That's also why I really love your writings in your books, in your blog, because it's always wrapped. in a story and in a context and the papers are mainly just, okay, this is the result, this is what we're doing, but it's just too drawing to me and so I'm not reading that when I'm trying to just read for fun, you know.
But yeah, awesome, well thanks a lot Andrew. I will, that being said, I will link to this chicken paper in the show notes for people who want to dig deeper. Thank you so much Andrew for... again, taking the time and being on this show. Two patrons will have the chance of receiving for free a hard copy of your book, thanks to your editor. So thank you so much, Cambridge University Press. And in the show notes, you will have the links also to buy the book on the Cambridge University Press website.
So... Go ahead and do that. You have a 20 % discount active until July 15, 2024. The code is in the show notes of these episodes, so definitely go there. And By Andrew's book. This one is really fun and you can read it on the beach this summer, you know, and then you'll have a lot of cool stories to tell your children or at the bar at night, so definitely do that. Thanks again, Andrew, and of course, welcome back on the show anytime you finish your 15 upcoming books.
Merci encore pour l 'opportunité de parler avec toi. Perfect, as you can hear, Andrew speaks very good French. This has been another episode of Learning Bayesian Statistics. Be sure to rate, review, and follow the show on your favorite podcatcher, and visit learnbaystats .com for more resources about today's topics, as well as access to more episodes to help you reach true Bayesian state of mind. That's learnbaystats .com.
Our theme music is Good Bayesian by Baba Brinkman, fit MC Lass and Meghiraam. Check out his awesome work at bababrinkman .com. I'm your host, Alex and Dora. You can follow me on Twitter at Alex underscore and Dora like the country. You can support the show and unlock exclusive benefits by visiting Patreon .com slash LearnBasedDance. Thank you so much for listening and for your support.
You're truly a good Bayesian change your predictions after taking information and if you're thinking I'll be less than amazing. Let's adjust those expectations. Let me show you how to be a good Bayesian Change calculations after taking fresh data in Those predictions that your brain is making Let's get them on a solid foundation