Welcome Tim Johnson to Novus Sanity Check. Thank you. It's a pleasure to be here. I'm so glad you had the time to do this and maybe you could tell me a little bit or the listeners a little bit about yourself to start with. Sure, sure.
Well, where to begin? I I grew up in the western part of New York State in the US. The closest landmark would be Niagara Falls. We were very close to Canada, which makes me a big ice hockey fan and I have a number of scars from my teenage hockey career that remind me of it on a, on a daily basis almost. I, I heard my doctorate at the University of Kentucky in, in Lexington, KY in sociology.
And while I was there, I, I, I had a, a nice job coordinating research at a survey Research Center that the university had opened. And this was during the heyday of random digit dialing. It had it really just been established as as the go to methodology for probability
telephone survey sampling. I, I remember arriving at the, at, at the survey center and they had this amazing library of telephone directories, paper telephone directories that were all organized and with some overlap would cover the entire, the entire state of Kentucky, which is where we, we were focusing most of our, our data collection.
And I remember bringing in a, a, a very talented undergraduate programmer to develop our first RDD program where we would feed in the, the area codes and exchanges to generate random samples and, and then to improve those using the Matoski Woksburg sample designs. That was really the heyday of RDD surveys. We routinely exceeded 70% response rates and we're taught to look down at any, any surveys that had response rates of less than than 70%. So that, that was, that was a long time ago.
That was the early 1980s. But I, I ultimately graduated and moved up to Chicago. It was hired by Dick Warneke, a cancer control epidemiologist who directed the survey research laboratory at the University of Illinois at Chicago. And I had done a lot of health survey research as, as as part of my own studies and my dissertation while while in Kentucky. So we, we hit it off right away.
And so I, I was blessed to collaborate for more than 20 years with, with Dick Wernicke and also with Seymour Sudman, who was our deputy director down on the Urbana Champaign campus and also a a quite a renowned methodologist himself. So I, I, I thought I had hit the jackpot, an opportunity to study and and work with both Wernicky and and Sundman.
I ended up Dick stepped down after a few years and I I became director and directed the survey lab for maybe 2/3 of of my academic career at the University of Illinois and ultimately retired a few years ago. But I'm still active engaged in, in my own personal research and collaborations and also in in leadership for professional associations concerned with public opinion and and survey research. So that's a long winded
introduction. Yeah. And and you also been president of both a per and wafer among other things. And I looked up your your bio. So, so you you basically let them two of the most prestigious research organizations in the world, according to my book at least, Yeah. And which is which is really, really something, I think. Well, well, it's, it's a willingness to help out and sometimes they have a hard time finding candidates. So a little bit of each I think.
Yeah, OK, you're like me. You'll forget to back down when someone asks, asks you, and then suddenly you're standing there alone or. Well, you know, yeah, some of us were like just a kind of a knee jerk reaction to help when asked to help with a project or a problem. And then only later do you think more carefully about what you've gotten yourself into. Yeah. But, but as you said also that there's been a lot of changes in
the methodology. As you said, telephone was, was the the gold standard for a very long time. And also the response rate, you said that it was about 70% or or over. And now in the US and, and, and several other places, it's, it's close to 7% and 70 I think, which really, really makes it hard to, to, to look at the response rate as a good indicator of, of a properly conducted survey. And, and maybe a bit an
interesting detail. But, but one thing that we as researchers always know is probably the first 100 interviews, you know more or less where the results will end up. So you get an indication, although there's only 10% of the total respondents who have answers, if you have 1000 respondents into you. And still we talk about the, the, the dropout rate as being the biggest thing because the theoretical dropout rate there
is 90% higher than it is later. And so, so, which is something I've been thinking about for quite a long time actually. And I'm, I'm, I'm, we've contacted some experiments here and realised, OK, but if it is OK in the 1st 100 interviews, more or less giving a good indication, then the entire data collection is actually just going more granular in, in the, in the subgroups. And also, of course, the margin of error will get smaller with a hard higher number of
interviews. But, but it's, it also gives an indication that the, the dropout rate is not the, the indication that it's often being talked about like, oh, if it's under this level, it doesn't work. So, but, but, but I also tend to, to, to see the challenge here because now a lot of people say, OK, the response rate is so low, so you can't rely on the pulse anymore. And I think both of you and I know that that's not really that simple, is it? Not even close. Yes, no, it is really
challenging. I what I've always tried to tell students is that it it, it's really variable specific. There, there, there are some variables that you can estimate pretty reliably with those very low response rates and there are others for which other variables for which the selection effects of a low response rate render it not, not terribly useful.
But those tend to be behavioral questions, attitudinal questions is my in my experience and my readings aren't as vulnerable to those those types of selection effects as as our behavioral measures. So when we're doing public opinion polling, it's it's seems to be a little bit more robust to these the these low response rates that that we're going to be living with for the foreseeable future.
Yeah, yeah, we are, aren't we? And and, but, but that is also part of the magic with the probability sample is that you the response rate tend to be random as well. The the the non response. I mean, sorry, because as you said it, it's the the the attitudinal, the attitudes are seem are quite reliable still, but but the behavioural and may maybe just to try to explain it to to the not so nerdy person. I if if someone is not deep into this swamp as we are.
I, I, I tried to explain it that OK, if, if you want to find someone helping you with their plumbing, it's really easy to find the guys who don't have a job, but they might not be the best ones. And, and from that perspective, also in the research perspective, if you have very niche, if you want to find a very niche competence there, you might have a really big problem with the non response because
you will not reach those. But in attitudes, what you're going to vote for in the next election, sort of, for instance, that that is such a broad opinion, which it doesn't matter if you don't find the perfect or the greatest plumber there who'd really are going to fix your, your emergency in your, in your bathroom. So, so that that's me trying to dumb it down. Maybe I'm completely wrong, but
what's your take on this? Well, it's, it's you're right, it's hard to dig even deeper into into it off the top of my head, but but there is a strong correlation between the types of questions and and the the effects of the of the low response rates with the behavioral questions just just be more difficult. Yeah, yeah, yeah.
And and difficult actually means cost more to to to for often because you can still do it, but it costs a lot more money because you have to work harder to get the the correct respondents or reach the correct respondents. Right, right. I'm, I'm sorry and, and, and maybe it's, it's easier for us to to, to use some external methods to validate those behaviours. That, that it is, it's how do you, how do you validate somebody's attitude can be done but but indirectly, but with
with behaviors. If we're talking about your, your visits to the doctor and, and and and so forth. There are external external records that can be used to in, you know, the, you know, it takes a lot of effort to do it. But to, you know, to, to validate what, what, what people are telling you. And for a lot of other those
types of variables. You know, some of these wonderful registries that you have in in in Europe are very, very useful for those those purposes that that you you can validate health conditions as as as well as health services receipt that that you can't do with opinions. The the only the only external way to validate self-reports on
public opinion. You know, the the common one that we're familiar with and that we get in a lot of trouble arguing about at the professional conferences is a horse race questions in the polls who's you know, because we can there is an external way to validate that and we get in a lot of trouble when the snapshot polls are are not successful in in incorrectly validating or correctly, you know, predicting I guess is a better term to use the.
Outcome, yeah, that's the challenge as well because it's actually not predictor Tory methods. So it's as you said, it is a snapshot. But then we say yeah and enough. And if nothing changes, this will be the outcome because. That's a big caveat you've got there, exactly. Exactly. And that's what actually, sure, you can add, you can, you can work with predictatorial models, but pollsters, the, the public opinion pollsters actually don't do that. They, they, they have a snapshot of today.
This is the last day that we measured before the lecture that this would be the outcome if it were the election today. But then it's being talked as being that the a prognosis which is at per to sign isn't so. So there you also have this challenge and, and, and just just as an example, in Sweden in the last two national elections, 4% of the voters changed their mind after we as an industry did the last questions for for the election and the election. Interesting.
So we know now that the voter mobility is so much higher than we used to because of things like Internet, because people that they're that and and that, that the drama in in media often tend to create some small, small catastrophe for some candidate the last couple of days, which really move the voters because you, you don't, you're not really if in the US you have two parties and it's more polarized by by the sign maybe. But but in Sweden, we have eight parties.
But for instance, the days where you were born, once a Social Democrats, always a Social Democrats is over long ago. Now we can move the entire spectrum depending on where you think of the present and you think of the future. And then the parties are really triangulating this a lot. So the the polls is actually not a good indication of how good. No, the sorry, the election is not a good indication of how the
polls are going. But thankfully, although 4% changed their mind, there's still a pretty good indication. But if 8% changed their mind, it would be worse. Yes, well, and, and especially in, in, in in most Western style democracies anymore the the outcomes of the election, there's no more landslides. It's it's always the United States being a a terrible example where it's very close even when it shouldn't be. It it turns out to be very
close. And so people are going to continue to use the election outcomes because that's all we've got. That's the only Yeah, yeah, but but the old sick available. Exactly. But at the same time, we have an entire industry in in, in the US spending billions of dollars getting the voters to change their minds, which we don't have in much other issues.
So, so, so the pollsters are trying to measure something that the, that there's $1 million at stake or maybe billions of dollars at stake, getting the voters to change their mind in the until the last second, which is a fascinating weird. It's it's like someone is actually doping the horse in the horse race. It's interesting analogy.
There are a lot of moving parts. You're right in the election and there, there are a lot of actors and, and I, you know, they're not all good actors that, that are, are attempting to influence the outcome. You know, the political parties, of course, that's their job is to try to sway voters. But there are a lot of external groups that, that are also sending messages and more and more we see these disinformation campaigns and so forth. And it's also challenging to to
measure their impact. Yeah, yeah, it is. And and I'm maybe maybe back then to, to, to the the only joystick we have for for opinions being the horse racing. But at the same time, proper polls is still a very good indicator both if there's something bad going on about real voter temping tampering, exit polls are often the only proof you have or an indicator of of something not going properly in in an international election.
But but same time you can also see when the voters change speak of of of Canada, there was a radical shift in opinions going going like AUS effect, but going going the opposite way. We don't. Have to talk more. That was very interesting. Yeah, some of my Canadian friends have actually thanked us, you know, because by, by electing your current president, our current president, we saved them from from a similar fate. Yeah. Yeah, which is, which is really tragic and fascinating at the
same time. But you have all this that's probably an unintended consequence for, for those who thought they were going to actually drive it home in Canada. But but at the same time, the voters actually react to what's happening in the world. But the polls showed the shift of opinions during this change. And, and so if we move from, but that's, that maybe that also the stories a little bit of the argument.
Because if we, if we say that the, the, the election is not a good indicator of how good the polls are because people change their mind. And until the last second, then you have no way of proving that the polls work on measuring opinions. But the same time we know that polls is a good indication of explaining when and how the opinions changed. But you can't prove it if they changed their mind after you stop measuring.
Because at some sense me, me being in the private sector, I also, I, I, I really feel the responsibility because if I present the poll at the borning of the Election Day, I might change the voter turn out. Because we, we have science showing that for, for Brexit, for instance, when, when, when, and also the US and, and Denmark.
There's a bunch of examples that when someone calls the election based on polls saying this is over, it actually affects the turn out rate of the election because the people who support the winning candidates say oh good, I don't have to vote. Exactly that. I'm certain that's what happened in 2016 in the in the US. Yeah, yeah, yeah, exactly. And, and, and, and the Brexit, there's several indicators as well, because the voter turn out was lower in the remain areas
than in the leave areas. And there were also a lot of maybe anecdotal reports afterwards saying that hang on, I voted for Brexit because I didn't think it would matter. I just wanted to protest because everyone said it would be remain. So I just want to put in my my own. So yeah, they have to get get their act together, but I I still didn't really want to leave. So it's, it's, it's really
challenging. You're right that the public has this assumption that the polls are correct within half of a percentage point. And and it's just not realistic given our, our, our models to, to expect that that kind of precision, especially as you said earlier, when we stopped polling a few days before the actual election takes place. You cannot capture all of that churning and and last minute movement in the electorate that's out there.
In the old days we thought we could understand voter turn out by by the weather conditions on the day of the election. That certainly doesn't work, but but there is something that drives turn out that all of our likely voter models cannot cannot adequately capture, especially with the the fairly rudimentary questions that that are used, you know, they're they're all realistic, but but I just I don't think that we have have a good way of of adequately modeling, you know, the the turn
out. No. And I, I also think because it also has to do with, with, because you said something, I'm just going to air that idea as well because it's always a 5050 race. And, and I'm starting to think that we're, we're getting so close to game theory in the political consultant realm that you've, you're creating this 5050 choices for everyone because you're trying to try also using really reliable polls, really expensive polls to dig out.
Where is the key issue that I have to get the people to tip? But, but I think that they're too professional here in getting this. So it's like a, a, a coin toss for the voter to to much that the people actually care because there's something weird here. How come it's always a 5050 race? It was the same in Poland the other day. Two, one or two percentage point over for, for, for one of them. There's something here that's so weird. It hasn't to do with polls.
It's it's actual, actual election outcome. Sorry. Yeah. Right. No, that's OK. That's, it's polarization is, is what I, you know, yeah, see discussed in the literature and in the media at, at a regular basis. And they're right. There is just a heavy dose of, of polarization in, in most of our Western societies these days. And it's produced all kinds of negative outcomes, including what we see in, in these elections.
These outcomes in these elections are probably just a a by product of of of those broader social divisions that that are being discussed. Yeah. But but I also think that it's it's because you focus on these polarizing issues because so, so because the communication is so heavily, you research it so deeply to making sure that you get this polarization, these false choices, basically black or white choices. Yeah. They refer to them as wedge issues.
Oh yeah, yeah, yeah. And, and, and, and then when you have a normal distribution curve on anything, because that's what you have, you're being actually forced to, you're at the tip tip of the edge all the time and just tipping over for one or others issue. There's something there that hasn't to do much with poles except that poles are actually being used to to find these issues. But it's also how you use this information that is the
challenge here. And when the grayscale disappears and everything is black and white, yeah, you're, you're either black or white. So, so there's something there. But I realise now that we're completely off the topic but but it's OK. But still interesting.
Yeah, Yeah, it is. But I also, but because I said something that I, I think that listeners might react to because I said if, if the, if the elections is not a good indicator of reliable polls, because if the voter mobility increase, then it will be a worse and worse indicator. But that actually doesn't mean that the, because you're, you're talked about this a little bit as well.
That does actually doesn't mean that you cannot validate other polls because you can do it on some other stuff. There are some, some behavioural issues, as you talked about or, or topics that that you could, you could look at. But I think also one thing that is getting forgotten is that actually the theories that's being used has been working for almost over 70 years now. And the theory includes also the margin of error and and that's it.
We're 95% confident that it's within the margin of error. OK, that's a that it's, it seems a little bit dodgy to say that, but it's also you're 95% confident it's within the margin of error, which is quite good in most issues and the theories are still there, yes. Now that's largely sampling error, right? Yeah, that we're talking about. Yeah. You know, sometimes I, I, I use the rule of thumb that you, you double that margin of error to
account for non sampling errors. If if, if it's really, really important to you to understand you know what the odds of one candidate or the other winning the election is, it's it's probably safer to double that that margin of error. Yeah, but but but. It's a rule of. Thumb yeah, yeah, that, that is a good point because if you're stuck at the the 1-2 percentage point or whatever, it's like, OK, if it's really important, why not double it?
That that's a good, that's a good point, especially when it's a 5050 because the margin is actually bigger than on the middle than they saw on there and and but it also needs to be a random sample or probably. Absolutely. And that's the thing that's getting forgotten because there's a lot of arguments saying, OK, the response rate is so low, you cannot do probably probabilistic research anymore. It's no point. And, and I see no proof for
that, no actual proof. It's, it's just like an accusation being thrown around all the time. But but I see it all the time. Although there are no real proof that the the response rate low responses, it is affecting the opinions as as we talked about, but it's still being OK. It's no point doing random samples. We have to go to alternative methods. But then you'd completely lose the margin of error.
Totally. And you know some of these studies, these these poll aggregators, our, our colleague Claire Durand in Montreal does a lot of work looking at the accuracy of of final polls based upon sampling methods. And she finds consistently that the probability based sample frames are, are more accurate, not not tremendously more accurate, but more accurate consistently. Yeah. And and there you have a, a, a complicating factor as well because no one knows really what
happened in this last election. But but the polls were so close, it was basically mathematically impossible that they were independent of each other. So there was something where some say it's hurting, I don't know because I'm not there, but there's something there because if you don't know, if it is a really independent poll from the others, you also you have that variable here as well so that no one knows what's happening because the, this is being done in commercial actors.
But there's also I, I often joke I could make a perfect poll in Sweden without asking any questions because there are so many good polls to look at. And, and if you're, if you're at in close, the close to the other ones, no one would rise and raise an eyebrow. And you will save a lot of money doing that, but you will of course be lying.
But if it's your reputation at stake, which is what what we're talking about as as a poster and you have to show you that you're actually are good enough to be trusted for the commercial work because you don't make money on on on the horse race as as a poster for the media, you actually lose money.
You make that. You make money working for the parties, but and they're they're they, I talked to a lot of of those and they say, yeah, we never know, use anything else but proper probability samples, random samples, because they can't afford to be wrong and they also get paid to be right. So that's true. Yeah, that's true. You know, I, I don't think that there's a lot of hurting, you
know, I, I understand. And the concern about that I, yeah, I, I've seen so many polls that that seem to be outliers and the, the pollster was kind of embarrassed about it. Yeah, but this is what the data showed. I, those would be the ideal candidates for a little hurting. My I maybe they tweak and this is not hurting, tweak the variables in their models, their weights and sample weights and so forth. But that's well. Yeah, Yeah, yeah. I get to what you mean.
And as I think there's different layer layers of this as well, because OK, in Sweden, I've seen so many obvious indicators of this because some non probability polls were so way off from from us, Ipsos and Kantar.
We were in the same space all the entire time during because in general, we we publish a poll every month during the election year or even with between elections of all the time actually, but but close to the two or three latest polls before the election, the non probability base subtly close up to us, which is really impossible to explain because and that has happened with so many providers for non probability based panels in Sweden.
And because and the argument for them has always been we're better because we don't have the problem. We don't response. We are using the water methods, whatever, whatever. And if and that's how they explain to be different between the election from us, but it doesn't explain how we continue to to that the probability based are to continue to be close to each other the entire time.
Of course, there's some noise there because and it should be, but it's giving the same view of the opinion and the non probability being completely different until exactly in before the election. So that that is that I've seen this for at least three elections because I'm so now I've followed this for such a long time. So this cannot be a coincidence, but I can never prove it.
But it's still annoying me because either you're completely different and you get it right, but then you should be completely different even in at the Election Day, because then the traditional methods doesn't work and they have a new perfect method. Yeah, boy, you know, you, I, I almost, I, I don't have an answer to that either.
No, no one has. Yeah. But, but I think it, it, it's going to require some forensic analysis of those non probability polls because when we talk about non probability, what's the definition? It, it, it's, it's what it's not, it's not a probability poll, but it could be anything. Yes, there's such a variety. They're they're not all the same animal, these non probability polls. So I I don't know that. That is true and and exactly and it's it's like yeah, yeah.
And that is part of the problem as well as as as you know as because you know, you've left the scientific field to be just to be very broad. OK, we have a definition of what probability based sample is and that's the the theory we have for 70 + 8. OK, there's details. There's so there's so many things there that also makes that a, a, a sliding scale to some to, to a lot of sense.
But it's actually at least there's some sort of theory bit that that I usually say that you try to replicate it to the best way possible because we all know back to the response rate. You cannot do a perfect probability samples probably anywhere anymore. But you can try to replicate it to the best way possible. And and maybe in the simplest form, you as a poster have a big bucket when you randomly sample people. And then we ask you, but not the
opposite around. You cannot stand in line saying I want to give my opinion to you because that that's the simplest difference. But but, but as you said that there is no definition of non probability based samples. There, there are so many different things. And maybe that's also why why you can't say a method works that's not based on probability based because it's like, OK, it's just not what we know. What's the underlying theory? Yeah, exactly. Because I'm, I'm still waiting
for that. I would love to see another method that actually works, but it has to be some theory that can be confirmed. Now at at least quota sampling, you understood the rationale there there was, there was a reason for what they were doing. Yeah, that's true, but.
A lot of these other approaches it's it's it's not so clear and and the problem well, the general problem would be inaccuracy in in a lot of these election polls leads to a lot of public suspicion regarding the quality of the the polls. They don't, they don't believe the polls, especially you think about the motivated reasoning theories that they don't like the polls when they don't show the results that they would like
to see. And if you if, if in a heavily polarized society, there's always going to be a significant fraction of the population that does not like what, what the polls are, are, are telling us. And and that they first thing they do is well, it's fake polls and some of our candidates are capitalized on that. Yeah. And that and yeah, that's a good point. And it's also, it's hitting the, the integratives of, of, of science in general as well. Yes, it is.
Because polls is polls and, and you have spent the entire career not political polls, but doing, using polls as part of your research. And and, but, but no one knows the difference between these horse race polls that no one pays for because media just gets them and the proper research that's being based on on, on on the same theories, which which is a quite big dilemma because there also the census bureaus use the same theories to a large extent on the same methods.
You know, you're absolutely right about that and, and it's an interesting point. A lot of my work is back to my collaborations with Richard Wernke is, is related to cancer control and there are dozens of validation studies that did you get a mammogram And then can we have your permission to look at your medical records to confirm the dates? And a lot of it is is not whether or not they received the, the the exam, but the, the, the, is it in the proper time
interval? Do they recall was it in the last five years or whatever the the Cancer Society recommendations are for, for speeding those studies? Yeah, we've got a pretty good idea what the, the concordance rates between the reports and what the medical records tell us are across all of these cancer screening medical procedures today. Every single one of those studies is based on probability sampling it if If they came in and they were non probability sampling I I, I, I'm not sure
they'd even get published. Yeah, yeah. But but when the polls in quotation marks are wrong in in in in an election, I think it also hurts the willingness to finance these kinds of standards because the general idea is that OK, polls don't work, the response rates this to low, everyone knows it doesn't work. Look at the last election. Why should we spend a bunch of money on these non working
methods? So I, I see indicators of that for, for several places because it's so hard, because the politicians are the one who, who gives grants and also they have the voters who the most vocal voters. And as we say now, if it's a poll you don't like, yeah, it's the polls wrong. They they are biased. Can't be right. Yeah, exactly.
Because everyone I know thinks like me so and, and the poll is saying something different, which I have heard a lot of times because I work for politics, political issues. But that's how it works. Because you are in your own bubble. And of course you don't meet the opposition, no, no matter where where you are to that extent because you don't talk about those issues with people who are probably on the complete opposite. But but. Partially, that's true.
Yeah, which is which is fine as well because you don't want to fight all the time. To be honest. It could be it's it's OK to live your life. But if you start extrapolating that and also say, OK, the post don't work. So I don't trust the science, I don't trust the statistics from the census bureaus. Suddenly it's like quickly unraveling worldview where everything is contested and that that's basically the core.
Why why I'm I'm constantly nagging about this is about this part because the knowledge society and a fact based society is so much based on what the core of what we're actually doing. Although I'm commercial and and you, you come from the university and, and and the real science part, it's the same. Overlap. We're all part of the same statistical knowledge infrastructure. Yeah, yeah, yeah, you're right in in the US right now that that that infrastructure is being
taken down. The the the NISTA of a national survey of drug use and health. They fired the entire staff. Yeah, the federal government did. Yeah, it's tragic. And you, you start politicizing knowledge and, and we've seen this before in other countries in, in Russia or the Soviet Union, the, the science was heavily politicized. You, you only were allowed to come up with the, with the findings that was in line with
the party. So, so we, we know what happens and we don't know where we're going. But but just to stay out the politics because it's it's really it's it's really too sad to talk about even people who don't think like you need to trust facts like gravity works. It's a good thing. We need a common, a Common Core of knowledge. Yeah, we did. It's uncontested. Exactly.
And and, and sure. And, and we also have to trust the spokesperson who stands behind science because politicizing that I, I, I know John Krosnick did a lot of studies on this. When, when, when in climate, in climate studies, when he did experiments, a scientist saying something about climate change, but then he did the same, but adding a political hat on the same person.
The trust was degrading rapidly, of course, because suddenly you weren't this independent scientist, you were a political actor using science to get your point through it. That's how it was the the person was saying the same thing, but it was being perceived as trying to getting political about the
issue. So, so there's something there that that really needs it's, it's, it's a really, really important issue just to move away from trying to push your view into explaining this is the way it is. So yeah, I don't know what the fuck to make of that, sorry. No, that's OK. We're we're we're down the rabbit hole now. Yeah. Really, I so don't mention the war. Oh, OK, it's all right. Like I, I always do that. I'm, I'm sorry, but oh. No, that's OK.
But but it but it's really interesting to but we also have a challenge because we shouldn't contest facts because it in the long time it benefits no one how you use the facts. That's a different thing That could be a political thing, but facts should still be there. How many people living in a country whatever all these data or or or also the views, the opinions of people is often really important. The population estimation of how how people think is often a a
vital part. Even between the election of of so many things, how we behaved. We had a pandemic, but the opinion of of the people and and the behavioural that that that the opinions resulted in was vital to handle the to do not panic, for instance, the simplest part and and we need to know that and we need to trust that those opinions are representative for the population. Well, it, it, it, it, it really is a challenge. I you, you mentioned the COVID and, and just the, the ongoing
controversy. It's, it's almost becoming like the Kennedy assassination. You know, what is the origin of the COVID virus? Is it, is it from that, that laboratory and that game of function research, or is it a, an, a terrible, terrible accident that, that took place in the, in the, in that open wet market? I don't know that we'll ever know. No, that that exactly. It also has to do with with the trust between people. Yeah. Agreed that the the inter human trust is is the key in in a
democracy. We can trust our neighbors and then realise that we have laws and rules and regulation that we follow them because they are the consensus of the culture of, of, of, of a democracy. Yeah. And, and there's an incredible amount of trust associated with the scientific enterprise that that what you are reporting is
actually what you did. And, you know, I, I, I think in the US and the transparency initiative and other efforts around the world, you know, to reveal as much information as possible, not not only so that the results can be replicated, but also so that that other scientists will accept as credible. Yeah, the the research that that, that you're reporting and. Yeah, yeah, that, that that's
true. And and also the the the complicating part in actually explaining complicated details in a trustworthy way without having the audience to read all the footnotes. But they should be there. But it's yeah, so so that. That that information should be there for those who are interested in primarily other other scientists and in in understanding those details.
Yeah. And maybe that's the last topic because I realized we have completely way off this, but actually you, you presented a, a paper with, with one, one of your colleagues at the latest. Peter Miller. Peter Miller Yes, at the latest April conference about what happens when you, when you present more information about what's the transparency. Can you just talk a little bit about that? Oh, sure, yeah, sure.
So the, the E POR transfer initiative was actually Peter's brainchild when he was president of the association back about 15
years ago. And it, it was a heavy lift on the part of the whole association to to develop a, a workable model for a, a, a program that would encourage survey organizations, commercial government, academic survey organizations to make a pledge, A voluntary pledge to report on a routine basis for all studies that they publicly report all of the mythological details that are in the official code of conduct, professional code of conduct for the association.
And there are over 100 members of the association of, of the Transparency Initiative today that that have taken that pledge and, and that report that information, it's never been evaluated. And, you know, and it became obvious sometime back Peter and I had discussions that, well, the, the methodological disclosure of those details that we were talking about, you know, it's scientists talking to other
scientists. Nobody else understands or cares or wants to read all that, that nerd speak or that Godly go. So we were wondering, what are we going about this wrong? We're not really reaching the audience that, you know, has declining trust in, in, in public opinion research and science in general. So what can we do about that?
We took advantage of an opportunity and, and, and the people at the University of Southern California's Understanding America study, what we're kind enough to give us some time on, on one of their nationwide panel survey questionnaires to, to conduct an experiment where we, we took a fairly neutral study, news media report, I think it was about weight loss drugs. And then the latest findings from a public opinion poll about the use of, of, of weight loss
drugs. And we developed three different types of mythological disclosures to, to, you know, for that news story and then randomly assigned each respondent to one of these three disclosures. 1 is what we call the minimal disclosure. And it was about two sentences. The survey was done in this month and it was a sample of X number of Americans, very basic minimal disclosure.
The second condition was what we call the informational disclosure, which is basically what the Transparency Initiative requires. There's a much, much longer list of, of data elements that need to be disclosed to conform with the the Transparency Initiative's requirements. And then there's something that Peter developed called the explanatory disclosure. And rather than just throwing all of these numbers at you, the response rate was this, well,
what does that mean? If if you're you're that talking to a scientist, what exactly does that mean? What, what Peter did, he very carefully crafted an explanatory description of all of this data so that people would understand why decision, why methodological decisions were being made and and what how this information should be interpreted. It was a much longer piece, as you, you might imagine.
So the experiment ranged from 2 sentences to kind of, it's kind of like Goldilocks with three different levels of, of, of obial. But we, we randomly assigned a large sample. We had 3000 respondents and they each received one of these, these disclosure texts, the minimal to the maximal, the explanatory.
And then we, we measured their, we had this very nice scale 4 out of scale that had great reliability and, and construct validity that measured their perceptions of the credibility of the pole that they were, you know, that was the focus of the of, of the experiment. And we did not find tremendous differences in terms of the effects of this methodological
disclosure on respondents. We found a little bit that that the explanatory text, despite the fact that it was a much longer block of information written probably at a slightly higher reading level, nonetheless elicited higher perceptions of of credibility relative to the minimal. We interviewed these many people On this date and we, we had a series of, we looked at a series of, of potential moderators of the, of that relationship and, and, and found virtually no
effects there. And we also learned that not a lot of people know about the transparency initiative or it just just mentioned the transparency initiative as a Good Housekeeping seal has no effect on, on people. So that, that that's the word and, and the final piece of of interesting information was we randomized the presentation of this material such that half we just embedded it the methodological details in the study.
For the other half, we said For more information about how this study was done, click here on this hyperlink. And so they, they had to actually go out of their way to to access the information. And what we found was just under 20%, about one in five respondents had any interest in even going to look at at the methodological information that we got. That tells me we've got a long way to go. Yeah, yeah. Sorry. That was a long winded no, no,
no but. I think it's, it's, it's good because I, I, yeah, I found it really interesting as well. And, and also the challenge here. But of course, there's so many more challenges because most of this polls or, or research result is presented in, in news media and, and, and the trust in media is, will influence the trust in the poll if you're unlucky as well.
So there's so many layers. But I, I thought it was really interesting to just look at this because at the same time, it is good to have some sort of disclosure of what's what, how, how the, the poll is constructed because. Yeah, just going back to the probability based, non probability based, that's one step. But also OK, there is some science behind this and we have a method that we we actually know it is possible to reproduce the method and and in that case also a result.
So I thought it was really interesting and, and that there are, as you said, there's so much more to do. But but I, but I think we all have to figure out how to work more on increasing the trust in the science behind this and and also, of course, protecting the neutrality of the science, because that's not facts or facts. And we should treat it like that. Then you might not like it, but just it's still probably the good to know it. That's basically my my take on facts. I'm with you.
We have to protect the neutrality of the facts. They're not. Yeah. And then? Partisan. Yeah, yeah, exactly. Because then every, every fact will be a partisan. That's that's an entire different challenge. But but yeah, we, we, we still have to acknowledge that there are science that actually works and that we know how to do it. And and that in turn creates a lot of vital facts from the, from in the Western world that we need in our daily life. It affects the economy.
It affects everything actually what we're doing. And we tend to forget that that's, but I'm really glad you took the time because I realized we have run out of time right now. Oh. Have we? Yeah. You know, it's been a pleasure. It's been a pleasure talking with you. Yeah. These are important problems that aren't going away. We'll have to continue our dialogue on these topics. I totally agree. I'm so glad it took the time and
now my day is ending. I think your day is starting, but it's amazing that we can talk across the world like this. Thank you. So much a great invention, yeah. It is. I hope you have a great day. Thank you. Thanks so much.