Y'all keep sending me your Instagram polls and saying who likes candy corn? I love candy corn. Do you like candy corn? No? What it's like? Wax? What are you talking about? Candy corn has such a great flavor. I also saw a pole on Twitter where somebody was saying what's the most trash candy? And everybody was saying, you gotta get Milky Way out of there. I like Milky Way. Something's wrong with everybody. Milky Way is a very good candy. My favorite candy Three Musketeers. That's good too. I like
a good nugut? Is that? What's in there? Nowgat you throw those things in a freezer for just a little bit. Ooh, I've never tried that. Hey, stuff you gave up? I'm going to I'm t T and I'm Zakiah and from Spotify Studios. This is dope laps. This happens every year around Halloween. As soon as October hits in that Halloween candy hits the market, everybody becomes the great debater about the best Halloween candy. Full size, family size, funny size,
all these things like that? Are y'all giving out full size candy bars? On Halloween. We're not not in my house. I saw a thing on Twitter where somebody has given out rotessery chickens. Ha ha ha. First of all, you're bawling. I don't care what grocery store you're going to to give out rotessity chickens. That's crazy. I'm having three guests
and that's it. Three tricker treaders and good night. Right, But it's so interesting to me that people will do these polls and you know, your little ten followers will answer. I'm only saying that because I got about seventeen, but your ten followers will answer. And then people are like, it's official, Milkie Way is out of here, and I'm like, based off of ten people, your ten friends is that. That's not how polls work. That's not how it works.
You need more people. We don't believe. We don't believe you. You need more people. And that leads us right into today's topic, polling. There's a lot of polls out there, and we see them all the time. Sometimes you're not even aware that you are looking at poll data and it's right in front of your face. A lot of y'all are sharing memes. You got bar charts where you should have pie charts and scatterplots when you need a line plot. And I want to say, hey, that's not right.
But today we want to bring you the facts. We're going to tell you everything we know about polling. We're going to ask all of our questions about polling, because I mean, the election season is upon us. Bernie Sanders got like twenty five million in donations, Elizabeth Warren got like twenty one million in donations. They're taking that money
and they're gearing up. And what they're gonna start doing is they're gonna start using that money to not only put out like campaign ads and things like that, but they're also buying results from survey firms. And they're going to start to look at this data about all of us and put it out there and they're going to say this, many percent of Americans do this, and this is why you should vote for me. So I just
saw the latest Cricket Media poll. Have you seen this when it has Bernie Sanders and Elizabeth Warren as the top candidates and it says here are the top five candidates after the first debate. But if you zoom in, pinch to zoom, that is so tiny. Yeah, it says a poll of Democratic voters in Iowa, New Hampshire, and South Carolina. And so sometimes people share these polls and it's like, this is what Americans think, and I'm like, this is from three states, and it's the subgroup of
a subgroup. You even ask folks from the biggest states or the most populated states. Nobody in my family lives in any of these states. And so what we're trying to do with this episode is give you all the necessary information that you need to be able to look at some of this stuff and say, oh, that's not what that pole meant and find that the real dat of yourself. Yeah, And if you're like me, you're wondering
who are those people getting surveyed? We also have a tip about how you can make sure your voice is hurt. So let's get into the rescitation. I have questions about that because I see a lot of poles being shared on Instagram and on Twitter without any of that qualifying information. Where did you get these people from? How many people
did you survey? When did you survey them? And even if the information is there, you see how tiny it is, and when you just scroll past it and you're like, Okay, Sanders A. Warren, I thought it was a shadow box, right, I thot just a line. I thought it was a line that separate this from the next paragraph divider. So I think, like, I have so many questions, like can just anybody issue a poll and put this information out?
Are there certain places we should go when we're looking for polling down or like when I want to know what's the latest trend? You know, where do you go for reliable information? Yeah? And then if I'm seeing information, like if I'm seeing a bar graph and it says like, Okay, these are the top candidates, blah blah blah, how do I make sense of that? Like? What else should I look for to say that this is reliable information? What am my cue? Right? Which polls should I take with
a grain of salt? And which polls should I drink with salt water? Okay, So now that we know what we want to know, let's get into the dissection. This week, we are jumping into polls, as you know, and we ask doctor Rachel Bidekoffer to help us out.
I'm the assistant director of the Watson Center for Public Policy at Christopher Newport University and a senior research fellow at then a Scannon Center in DC.
Whenever we bring up polls, everybody jumps right to thinking about elections. But we know that polls can be about anything.
Not all polling is about elections. For sure. You get polling about public policy issues. You can get polling that about market research issues. I mean, companies will use polling to determine product placement or you know, the types of advertising they want to use for their products. Polling is used in cross every sector.
We're constantly polling each other every day. Like anytime somebody puts out anything, people are I don't like that, like I love it, you know. So it's like when you look in the comments section of like a post on the Shade Room or TMZ or anything like that. I sometimes I'll just scroll through them and like, Ooh, this person don't like it. That's one. This person like it,
that's one. Then going back and forth and seeing like a running pole in my mind to see like where people are falling on it, you know, another pole that just popped into my head. It's always in these toothpaste commercials where it's like nine out of ten dentists choose Colgate and I'm like, compared to what else? Right, there's always an asterisk and then it says like standard toothpaste. I'm like, I don't know what standard toothpaste is? And
did y'all really survey all the dentists? How many dentists? Is that there's a lot of different types of polls. I think the one that we are most used to seeing as an election poll, and that's what doctor Bittercofford calls a horse race.
Horse race is just an election poll. So a poll in which you're trying to determine who's up and who's down, or whether or not an election is going to be competitive. I wouldn't say there's necessary categories, but different types of questions and approaches that election polling might cover, depending on the goal of the survey.
I always see polls that show like people think this, or candidates are trending this way, they're doing well with mothers, they're doing well with veterans, Like huh, how do you get down to the actual sample or people that you want to pull?
So the way that sampling works in a population is that you're taking a random sample of a population and through a process called waiting, it does tell you within a certain window what the preferences of that broader population is.
So, if I want to say everybody that eats candy in America, Okay, we know that that's two million people. I know that's not right. But like, we know that's two million people, So how many people do I need to survey? Right? That becomes your survey sample size in order for it to be an accurate representation of that entire population of candy eaters, right, Because you're not going
to get all two million people to participate, right. And even if you say, okay, I need to ask a thousand people in order to understand what two million people think, you're not going to get all of those one thousand people to participate. So what she's talking about with waiting is creating a poll that has the same demographic of the folks that you are trying to represent.
There are some survey firms that are still using what we call random digit dialing for voter surveys, and I would argue that that's not an ideal way because voters, as it turns out, when you call people and ask them are you registered to vote? They will lie to you and tell you yes. Because it's socially desirable to be registered to vote, and they don't want to admit that they're not registered to vote.
And so when I think about that, you know, you say candy corn is great, And when we think about the population of people eating candy, we might say, there are people who eat candy who don't like chocolate, people who eat candy who love chocolate, people who eat candy that are allergic to nuts, right, and all those things. We will want to capture that in our We wouldn't want to only get people that are allergic to nuts, because then that would make that kind of that group
of people. It would keewe the results because then you would only get from their perspective, and so sneakers will be out of here then exactly because they can't them exactly. So basically, what we're trying to get you to see is you have to do your research on your population to make sure you're finding ways to ensure the credibility
of your sample. So you'll see the phrase likely voters in a lot of polls, and more people claim to vote than actually go out and vote when the time comes, And so part of the job of the pollsters is to try and find those people who will actually go out and vote, as opposed to the people who will just say they're voting but stay at home. Oh. So this also kind of ties into the last episode of
social cognition, like how do people perceive you? Exactly? So if somebody's polling you, you're like, what are they going to think of me? And it may change the response that you get. Yeah, so you'll say, oh, I'll definitely vote with which is a con MM. And that's why I don't like exit polls. Oh, because I'm like, everybody's out here line they're like, oh, well, the exit poll because even on the day of the election, they're basing
a lot of stuff off of exit polls. So people will go in and they'll cast their ballot, and they'll come out and they say, who did you vote for A or B? And somebody might say A because they feel like, oh, well, in my county, everybody's voting for A, and I don't want to come out and say I voted for B. That might be embarrassing. Somebody may hear me, they may judge me, and so then they'll change the answer.
Another con So, now that kind of captures the essence of deciding who you will pull and getting your sample, But what about what you'll ask them? That's very important. The first thing you can do when designing a survey is to make sure that your questions aren't leading.
You really want to have questions that are worded in a way in which you're getting responses that are a product of the questions.
So, for example, if you say, do y'all love my outfit? Yes or no? That's leading. You're leading it with we should love the outfit. Are y'all going to vote for this idiot? Yes or no? Right, that's not a good survey question. So when you create questions like this, you're basically generating what's called a push pull. This is something that pushes the respondent to answer in one way as opposed to the other. So her next suggestion is to
not provide an easy out answer. So that's something that's like if you have a bunch of different you have a question, you have a bunch of different answers, and it ranges on a scale from agreed to disagree. Having something in the very center that says neither agreed nor disagree. People are more likely to choose that because they don't want to do anything that's, like I guess, controversial in their mind. So they just want to ride that middle line.
That's crazy to me. I love to be at the extremes. We know my answers always strongly agree or strongly disagree. I hate it, loved it. This is the best thing I've ever seen. I was just having this conversation with my friend and I was like, I thought you said you hated that. She said, when I say I hate something, I don't really hate it or I don't really love it. And I said, huh, that's you. You do that too,
I don't really hate or really love. So you'll say I hate this and then be like, oh, I guess it's not that bad. That's just uninformed. I don't know any I can't think of a single example where I said I hated something and then I said, I guess it's not that bad. I can think of situations where you've loved stuff and then hated it. Wait, are these people people? I just did not love? That a point?
Now love to hate? Yes, that's an easy progression. I start everyone to get from love to you are a ghost. Sorry to that man, Sorry to this man who who a third tip that doctor Bitterkoffer gave us when it comes to designing good surveys, to rotate your answers so each person taking the survey will see the answers in a different order.
Because, believe it or not, people will go with the first response disproportionately. So if you have, say you have a horse race question, and you have the one candidate offered always first, you might not get an accurate estimation on the race because people will choose the first candidate more often just because it's mentioned first.
That reminds me of like in the SAT when they were like, oh, always choose C. Can remember scantrons and stuff, Gantron's everything, and people with default to C because they were like, oh, when they're making these exams, you're more likely to get the right answer if you choose C because of whatever reason. I don't know if that's how the computer would do it or whatever. And so people got hip and then they started rotating the answers. Another
reason to get these standardized tests out of here. Yeah. The last tip we got is to make sure your survey is short and to the point. I don't want to read this paragraph right, So if it's long, I'm probably won't even participate, and so the shorter the survey the better. So those are our top pointers for designing a survey. You may be asking why is designing good questions so important? All of these factors contribute to a
higher response rate. Response rate refers to the percentage of people that you are trying to survey that actually complete the survey.
And over the last few decades, especially as people have become more technologically diverse, response rates have been declined.
That's a good point TT because if you don't have if you have long questions, if people don't want to participate, that affects your response rate. And so all that work you did trying to figure out what's the right sample size and number of people, you won't get enough respondents. And that's why you know now when you call your cell phone provider and they want to do a quick survey, at the end of the call, they're like, it's just one question, like hanging there. They're starting to catch on.
It's like, I'm not going to talk on the phone to a computer for you know, thirty minutes answering all these questions, like on a scale of one to ten, what was your experience? No ask me one question was the experience good or not? So now we know how to get the right population for a survey and how to design good questions to ask them. When we come back, we're going to talk about making sense of the results.
We're back, and we're going to get into how to we through all of these polls and find the right information. Right there's so many polls out there. Every week, I feel like there's a new article telling me about the latest updates or who's ahead, who's dropping out, who's face. I don't know how to make sense of it. And as someone who watches academic polls, Twitter polls, political polls, and all of y'all's ig polls, how do I know whether or not they're any good?
When a poll comes across the TV and you know, the journalist is like, oh, forty five percent of Americans love lamas, you know, not to just take that as a face value, but to go and look at the survey and say, okay, how is the sample taken?
I love lamas? You do? You should not go of Americans. That seems a little low. That seems a little low to me. But I'm not gonna dwell on that too much. Let's let's just keep going.
So the very first thing that I look at when a survey comes out is that end size really need a baseline of at least five hundred completes for a survey to be reliable statistically.
So in size is the number of people that complete the survey. Right. In size is important for all surveys. The larger en size, the more reliable the outcome of the survey is.
And there are many surveys that have gotten you know, primo attention on MSNBC and CNN this cycle that are nowhere close to five hundred. Technically speaking, of course, you can get a statistical response off of a lower end size, but in terms of horse race pulling, there's this statistical problem called the margin of era that's exasperated a lot by a small in size study.
You really have issues with margin of error when you have a small sample size. In the field that I work in, we call that uncertainty. And so it's basically saying that when you take a measurement, so let's say it's something you're trying to weigh ten pounds, and if you have uncertainty that's plus or minus five pounds. That's
a large range. That means that when you're weighing this, it can either be five pounds or or anywhere between five pounds and fifteen pounds, so you'reself subtracting five or
adding five on both exactly. So that means that you have don't have a high confidence in the measurement that you're that you're taking, but you can also have a smaller margin of era though, right, Yeah, So if the measurement that you're taking, you're really you have a high confidence in it to a higher confidence level, your margin of error could be really really small, like a pound. So you can say this thing is ten pounds plus minus one pounds, so that means it could be nine
pounds per eleven pounds. And that's really cool to think about it, like the way it works in your field, because that's really similar to how it kind of works for polling. Right, you can say thirty percent of moms choose gift, and you can say, with a five percent error, twenty five anywhere from twenty five to thirty five percent of moms choose gift, and you can get that, right. You can get to that type of resolution with a
small sample size. But if you want to be able to say thirty percent of moms choose Gift, and you mean twenty nine to thirty one percent of moms choose You're gonna have to sample way more, way more moms, yeah, to get that type of accuracy. And so you're saying that, so margin of RAA is just how uncertain are you about this statistic? And then the other part of that is your confidence level. And so I think confidence level is really cool because that also has to do with
the number of people that you survey. But I don't know what they might call this in your field, but confidence level is like thirty percent of moms choose JIFF, And if I'm ninety five percent confident, then that means that ninety five out of one hundred times that I do this survey, thirty percent of moms will choose JIFT. Or if I have a ninety nine percent confidence level, then that means that ninety nine times if I do this survey one hundred times, ninety nine of those times,
thirty percent of moms are gonna choose JIFT. Jeff, is really you? I thought you were allergic allergic to peede us. I told you it's a mild allergy. I still eat it. It's not safe. I don't recommend that for anybody that has mild or extreme peanut allergy. Go to your doctor. So, as consumers of poll data, we have to go and ask some real questions about how the data was collected. One of the things is did the survey cast a
wide net like that survey I showed you earlier. Yeah, like the survey we were talking that we were talking about. They only surveyed people in Indiana, South Carolina. Yeah, three states. Yeah, and they're making very like sweeping claims, yeah, that we're all consuming. It's like, oh my gosh, Okay, so Bernie Sanders and Lizbond are tied at nineteen percent and that might not necessarily be true everywhere. And then the other
question is how did they even get the information? Are people taking these surveys on the phone, you know those people that are standing outside with clipboards at different places. Are they sprint past them? Are they doing the surveys like that like sorry, I have diarrhea, like I have to go? Or are they doing them online?
If it's online, have they done due diligence to try to make their panels randomized as much as possible. Are they being transparent about their methodology and the design effect.
Anytime you see a poll, there should be something at the bottom of it that tells you where you can go to find more information about that poll, or tell you a little bit more about it. Yeah, show me the link and click the link like this is important. If you're reading a poll and you're feeling swayed by it, you should want to know more information about it.
If you're a smart connoisseur, you can look at the data and understand. Okay, you know, I might not be able to take this to the bank. I probably can't say for certain candidate A has forty five percent of the vote. But I can say that candidate A is somewhere near forty five percent of the vote. And if candidate B is at forty and candidate A is at forty five and the margin of era is three points, I can say with some statistical confidence that candidate A is indeed leading candidate B.
So now that we know all that we should consider when looking at poll data, let's take this information and apply it retroactively. Right, what happened in twenty sixteen? Yes, because all of those poles are wrong, I looked at a lot of polls, and all those polls told me one thing, and it didn't turn out that way. So what happened.
What people usually are talking about when they talk about the polling being off in twenty sixteen is they're talking about the aggregated forecasting models, like the stuff that came out from five point thirty eight that said Clinton was going to win you know, seventy thirty, right, And those are distinct from polls because forecasting models produce probabilities of you know, one candidate winning over the other.
Right. So forecast is basically like, feed me all your polling data, and I'm going to predict what might happen based on multiple different poles. But if the polling data is no good, then the forecast is no good because that's what it relies on.
So one of the reasons that the forecasting models was so off is that there was a clear signal coming from the polling data that everybody missed, and that was that we had a very high number of voters double sometimes triple the amount that we're saying they were undecided or don't know in their vote choice running up to the election. Even a week before the election, that is double the amount of normal voters for a presidential election.
And what the narrative should have been in terms of the polling was given this amount of uncertainty and this unsettleness in the polling data, we really have no idea what's going to happen on election day.
All right, But see now that everybody's listened to the episode, you already know the threat to the soundness of your poll when you have that non committal option there. So when you have a large number of respondents saying neither here nor there, unsure, you probably can't trust that information.
In twenty twenty, Democratic voters, people who want to see Trump lose, are never going to believe any of the polling. So you know, if twenty sixteen was about comfort, twenty twenty is going to be about distrust of that polling data. And that's actually to the country's advantage because it's going to make people show up no matter what the data says.
But yeah, I mean, we're just living at a time period where people are deeply distrustful of everything, and you know that distrust is starting to hamper our ability to function as a nation.
I think I'm ready. I was just about to say that girl, I was like, I am red d Okay. I'm not going to just read these headlines and then run with it. I'm going to be doing a lot deeper dives into all of these surveys because they are about to be coming at us like rapid fire. Yes, And I'm already telling you. If I follow you and you post the results of a pole, I'm gonna ask some questions. I'm gonna say, send a link. How many
people did you survey? I'm saying, even if you're posting thirty percent of people said I should where does outfit? Is that three out of ten or is that thirty out of one hundred? Even on your Instagram surveys. So if you're posting a pole on Instagram and you're like, oh, what shoes should I buy? This red shoe or this black shoe? And you buy the red shoe because seventy five percent of people said that you should buy the
red shoe. But then we find out that only four people participated in your Instagram pole, you probably shouldn't have bought either shoe. Go return it. I hope you kept your received all right. So this week we are finally going to get to the bottom of it. Dope Labs Podcast wants to know candy Corn in or out? Candy Corn is so in? Nope, you are? I know there. Hey, hey, y'all, I know my friends out that it's trying to carry it on. Hey, I got three bags already. Oh. I
feel sorry for the kids in your neighborhood. These kids, if they're smart, love candy corn. Smart people love candy corn. I'm starting a campaign. Oh my god, We're gonna post a poll on Instagram. So y'all go and vote candy corn yes or yes no, don't forget to check out our website for the cheat sheet on today's episode. You can find it and sign up for our newsletter at Dope Labs podcast dot com. Also, we love hearing from you. What do you think about today's lab? What are your
ideas for future labs? Our number is two zero two five six seven seven zero two eight. You can also find us on Twitter and Instagram at Dope Labs Podcast. T T is on Twitter at d R Underscore t s h O, and you can find Zakia at z Said So follow us on Spotify or wherever you listen to podcasts. Special thanks to today's guest doctor Rachel Bittercoffer.
You can find her on Twitter at Rachel Bittercoffer. That's b I t E c O f e R. Doctor Bittercoffer also suggested some links for anyone who's interested in learning more about survey methodology and polling, and you can find those in our show notes. Our producer is Jenny Radlet Mass of Wave Runner Studios. Mixing and sound design by Hannis Brown and special thanks to Tyler Adams. Original theme music is by Taka Yasuzawa and Alex Sugi Eurra,
with additional music by Elijah Alex Harvey. Dope Labs is a production of Spotify Studios and Mega Own Media Group, and its executive produced by us T. T. Shadia and Zakiah Wattley. Doctor Bittercoffer said that a lot of these surveys they're calling landline because you're not allowed legally to robocall cell phones. First of all, who has a landline? Well,
in twenty nineteen, no one. Do your parents have a landline? Yeah, but they're not answering it mine either, And when they do, it sounds like they're answering it from the depths of the earth, beyond the beyond yeah, I'm like hello,