The Election Pollster’s Song

Biden has a serious lead. I don't know about you, but I don't pay much attention to the polls until I do, and I've entered that phase of the election where I can't quite help myself, like a guilty pleasure or maybe a non pleasure. I keep stealing back to the websites to see what the polls say. Can we trust the polls that are out there? What have posters learned from recent experiences, including the twenty sixteen election, and what methods do they use to get the results that

we see when we look online. Here to discuss these questions with us is Anthony Salvanto. He is the Elections and Surveys Director at CBS News, which puts him in charge of one of the most significant operations doing polling and calling elections in the country. He's also the author of the book Where did You get This Number? A polsters Guide to making sense of the world. Anthony, thank

you for being here. I sometimes feel like the rest of us are being cruel to posters because three and three quarters years we're interested in what you're saying, we're intrigued with what you're saying, and then for a tiny run up to a election, we suddenly blitz you, demand that you perfectly predict the future and predictively complain if you get it even a little bit wrong. So a collective apology in advance of my doing exactly those things

in our conversation. You know what, It's all good. The history of polling is one such that elections were used as a benchmark. It was never really intended to be a tool to just quote unquote predict elections. They were used as a benchmark by posters in the earlier part of the twentieth century to basically show people, yes, this works, Yes we can do a scientific example of the population and then get something external to validate that. People tend

to conflate the ideas of predicting with understanding. It kind of went from there, and that just somewhat comes with a territory. But it's not at all cruel if it brings attention to what we do, because I think the large or point of what we do is to try to understand the public mind and to try to understand people through the lens of looking at aggregate data to

what they do as mass behavior. And if that brings us attention, and then if people say, hey, you know what, these polls are kind of interesting, maybe I will pay attention to them in twenty twenty one or twenty twenty two and see what people think of public policy and other important issues, then it's all good. Anthony, you said something kind of profound right out of the box, which

is impressive at eight in the morning. You were just saying that we tend to conflate our ability to predict something convincingly with our ability to understand that same thing. And when you put it that way, you're making the point that actually prediction isn't the same thing as understanding. You know, understanding. We actually want to know why people do things, whereas prediction in theory could be blind to the question of why as long as we know what

people are going to do. And I guess I want to ask you more about this idea that you're seeking to under stand people. Is it do you see when you wake up every morning your goal to be I want to make the best predictions that I can, or do you see it as I want to understand why people are going to do the things that you're going to do, including voting the way they're going to vote.

It's the latter, It's entirely the latter. And I've told people you should judge me on whether I help you understand what is going on in the world around you. Here's an example. You always see around this time of year, people saying, well, I can't talk to friends and family because they're for the other side. You know, one person's a Democrat, one person is a Republican. Let's provide for Trump.

We don't get wrong anymore. We don't know what they're thinking. Well, suppose you had a tool that could help you understand a little bit about what the other side is thinking, then maybe that could facilitate a conversation. Maybe basic level. You just have a better time at dinner. You have that ability if you read and understand a good pull.

We posters often try to avoid the name posters, and we prefer to be called survey researchers because we talk to people and that idea can be carried through to the user as well. So I like to think all the way back to the start of your question, if we can deliver that understanding, then that's really the goal,

that's really the job. Let me ask you a few questions about key concepts that you and all other survey researchers use but that at least for me, aren't as clear as I would like them to be, and I might be not alone in this. Let's start with likely voter,

which is obviously hugely important concept. It's great to get a sense of how the average person would vote, but the average person isn't the person necessarily who was going to vote, because we have turnout levels in the United States that are much lower than they are in some other countries, and so a lot rests on your prediction of whether someone is likely to vote, and then you have to weigh that person's opinion in your overall analysis

of your research data, balanced by the probability that you assigned the likely of their voting. So how do you know what are the measures that you consider most reliable when you're asking if someone is a likely voter. So the likely voter model, which we used in a survey, basically says, how can we combine things you the respondent, tell us you're going to do, and whether or not we believe, for lack of a better word, that you're going to do what you say you're going to do.

This is the behavioral component of a poll. We know the attitudinal component. I think the economy is good. I like the president and only the president. This is the behavioral stuff, which frankly is a lot harder. Well. On one hand, we can take everyone who in a previous poll told us they were going to vote, and we can go call them back after the election and see

how many of them tell us that they actually did. Now, there may be some overreporting in that, but for the most part, this is a pretty good measure, meaning that people don't necessarily tell you the truth. If they did vote, they're happy to tell you, But if they didn't vote, they might be ashamed to say, yeah, I didn't actually turn out. That's right, And we know that there is some overreporting of that. It is not large by the way, Just as a quick aside, I don't mix that with

people lying. This is usually what happens. They say they're going to vote, they intend to vote, and then things get in the way. Absolutely, we do that, and we also know from lots of other economic data that we do all kinds of predictions about ourselves which overstate our ability to stick to our word exactly. So we've got some rough measure, and we can go back and look at that from past polls, and let's say it is the case about ninety percent of people tell us they

vote actually vote. Well, then we've got other measures, like from the social science literature, from political science literature. If people feel like they're part of a community, they are increasingly likely to vote. So someone who is a homeowner who's lived in a place for a long time is

more likely to vote. Now, how do you put that into a pull In our case, you build an aggression model that would have worked in the past, or you can back test your data on and you assign everybody in the poll a likelihood of voting based on their characteristics such as they're known in the aggregate. If we know that ninety five percent of people who have voted

in every single election, vote in the next one. So and then so we have a respondent who's voted in every single election, give them a point nine five probability score of voting in this one. And then the final technical side of that poll is that you take everybody with a weight assigned to them, a likely voter score, and you sum those up to get the overall likely voter estimate. As one way of doing a likely voter model.

There are others, but it is taking what you see in the aggregate and behavior in the aggregate and trying to apply it to an individual, which for anybody, whether you're trying to gauge voting or whether somebody will buy your product, can be a tricky business. There are factors that are particular to any given election which are much

harder to incorporate. So in a given election, somebody might be really motivated to vote against or for an incumbent, or there might be closed polling places and it might be harder to vote, or it might be that you've recently moved and don't know where you're going to vote. All of those things are variables that are much harder to gauge. I want to ask you and drill down on those elections specific features because they seem to, at

least in my generalist perception. So I've had a big impact on the twenty sixteen election, and one imagines they

might on the twenty twenty election. So are you and our other posters trying now to make strong predictive answers to the following question, our Biden voters going to be more likely to really want to turn out to vote against Trump, or our Trump voters are going to really be likely to turn out in high percentages to defend the president as opposed to what people do and let's say the immedian election, or as opposed to what they

did in twenty sixteen. I mean, that seems to be certainly from the standpoint of the way the campaigns talk maybe the whole ball of wax here right that if you know, if one side can really motivate its base, it's going to have a huge advantage in winning. So I guess what I'm wondering is, are you and our other polls actually trying to call that feature in? So tell us how you bake it in. Yeah, what we're doing is, first we ask attitudinal questions, are you motivated

to vote? Are you motivated more than you were? I have questions like how long would you stand in line? And the majority of people tell us they'll stand in line quote as long as it takes. Is that the case? Probably the case for most of them, but things get in the way. But I also see people who say they'll only stand in line for half an hour. That's useful information we have and turnout model in every poll

number that you see from US. Joe Biden's fortunes right now, frankly, including in all the polls that I put out, is that he is heavily dependent on people who tell us that they're going to vote for the first time, so they're included in the poll. But does past behavior predict future behavior? That's harder to know. And if they don't show up, this election will be much tighter. How much of a discount factor are you applying? I mean, just

to make it as practical as possible. If the first time TI voter on average did not turn up, how bad is that for Biden? Whereas if the first time voter who says, yes, I'm voting, but I've never done it before does turn out and vote, how good is that for Biden. Oh, it varies state by state, but if the first time voter turns out, then what you see in the polling will will manifest itself, meaning Biden's

big lead will be carried out. He'll win by a lot. Yeah, I mean I shy away from saying anybody will win, but it would greatly advantage him. However, if those folks don't show up, that's a much much tighter race. And look, I will add this because it really applies here. The uncertainty in this election, in my mind, is something that I have a great deal of difficulty quantifying, and that is the mechanism of voting. It is the balloting process.

We talk a lot about turnout as though it is all based on motivation, because in much money part parts it is, there's a motivation, etc. But in this case, you have people transitioning from old habits to new ones.

They stood in line at their local polling place, their local school, which is two blocks away for twenty five years, and now that polling place has been closed because counties are consolidating due to COVID, and now they have to drive across the county to a massive voting center where they need to find parking and where they need to stand in longer line, and where they need to know where it is in the first place. Or they're requesting a mail ballot, Well, okay, this big envelope comes in

the mail, and it's not that difficult. But now you've opened up these new forms and you have to sign it in the right place, and in some places you have to stick it inside another envelope. All of those changes and how they affect people are really unknown to us, and I think that's a great deal of uncertainty in

this election. And I suspect if turnout patterns vary a lot, or by more than we think that they are, that that's going to be one explanation why that person in the example I just used, Well, they decided they couldn't find parking at Dodgers Stadium, which they turned into a big, giant polling place. Maybe no surprise there, Or they didn't quite navigate the mail ballot process correctly and their ballot didn't get there or was rejected or what have you.

So all of that is incredibly difficult to quantify. I suspect we'll get some help in that in the aggregate as we get closer to the election, because we we'll see is the return rate on a lot of mail votes, and we'll also have the total early vote from all these early voting locations, so we'll start to have a sense of who's done that, how much of it has sort of been cannibalized from what would have been election day vote, and then we'll narrow down to what remains

the people who haven't voted yet when we wake up on the morning of November third. That'll be a help, but from a polling standpoint, that is all very, very hard to quantify. We'll be right back. You mentioned aggregation, and that's also something that I'm totally fascinated by, and it's something that in recent years has become more and more salient to anybody who reads polls in the newspapers or watches them on television. I'm specifically thinking of the

aggregation of lots of polls. There are lots of different polls. They have different methodologies. Some are local, some are national. But there are sites and experts, and you're one of them who say, look, here's what my polling data shows, but here's what my aggregation of all everybody's polling data shows.

And there's some implication there, a kind of wisdom of crowds idea that it's better to have as many different polls, even if they different methodology, some of which might not be your favorite methodology, than just to rely on one poll.

What's your sense of that. Are you one of those people who believes that will have better information by aggregating lots of polls even if some of them have a methodology that you wouldn't be crazy about, or are you somebody who thinks, no, we're better off relying on carefully done, well done polls and only those polls. The second, I'm not a fan of aggregation. I understand why it's done, and in general it's not unreasonable. The comparison I make is do you want a beat you can dance to

or do you want to hear the song? If you go to a nightclub and the DJ is combining a lot of different songs together to give you a really good beat, or blending one after another, that's really fun and that's really useful because you just want to dance and have a good time. If you really want to appreciate music and you want to dive into what an artist put out as their song, and you listen to a song, and that to me is a little closer

to what the polster is trying to deliver. Any good poll is trying to tell you what people think and explain why people think it. And they've offered you a model of the electorate. They've implicitly or directly offered you a model of behavior because of the questions that they asked, because of the things they tried to test or examine by asking those questions. Is it about the candidates personalities, is it about public policies, etc. So they're offering that

and you can take it for what it's worth. But if it's done well, it should offer you a good study of the electorate, and you can compare it. If you have the time or the inclination, you can compare that study to another one that's a really comprehensive and I would dare say a little more sophisticated view of how to approach getting information. But again it depends on what you want, how deep you want to go. So

the aggregate is for perhaps a shorthand who's winning? I can see it from this, Okay, God, move on with my life. Do I want to understand what's going on. I don't think you get that with aggregation at all. I love the analogy that what you do is you're a musician. You know, you're a soloist. You're singing your song or you're playing your piece of music, and we get a certain kind of depth and appreciation out of that.

And what the aggregators are doing, you know, on five thirty eight or you know what the New York Times aggregation is doing is they're DJs. You know, they're they're playing it all. They're mixing and matching and there's a beat that emerges, and you know that can be as you say, that can be great too. It sounds like it depends on what you want to listen to in a given moment or circumstance. I think that's a brilliant metaphor. So, but let me ask you, what song are you playing

right now? What's your feeling at a musical interpretive level of the information you're gathering. That's a great question for many many people, it is a referendum on the president. Now. In some sense that's not unusual because there's an incumbent on the ballot, but it is very particularly the case now and for Democrats They say that they are voting as much, if not more, to vote against the president than for Joe Biden. They say that they are motivated

to vote. At the same time, you have the president who has as solid and a core base of supporters as we've ever seen, and that kind of allegiance is something unlike we've seen. We did a study right before the Republican Convention where we ask does Donald Trump deserve your loyalty? And many Republicans said yes. In fact, I even tried this. We said what's more important to you being a Republican or being a Trump supporter? And Moore said it was being a Trump supporter. That kind of

connection is strong. So you have deeply held, very emotional connections to their positions on each side. And it's one of the reasons that you can explain in the aggregate why you've seen the president's approval ratings hold so historically incredibly steady throughout his presidency, where the economy has gone up or whether it's gone down, no matter what he said,

no matter what he's done. And it's also one reason why you find remarkable stability in the polls this year that Joe Biden edge has been about what it's been and it's not like twenty sixteen at all, where we saw a large fluctuations in the polling. That is on the personal level. On the individual level, part of the explanation for what you see in the aggregate that is the song of twenty twenty. It is those deeply held

convictions about what people are seeing. And the final maybe a coda if you will, on this is that there are very different views of what kind of shape the country is in, not just differences on how to solve the problems. The Republicans feel that the coronavirus outbreak is not as bad as the medical folks say that it is. They believe that the death toll is overreported. They are less concerned about the virus themselves. Democrats, by contrasts, believe

that the death toll is underreported. They are themselves much more deeply concerned about the virus, and there's some personal connection and experience in that, and that they tell us it's affecting their communities more. The Democrats live in cities, etc. That may be changing as they outbreak unfortunately spreads to

other places. So there's a very different view. And that's just one example of what is happening in the country that we don't even agree on what the facts are, let alone the more traditional or conventional arguments around politics of what do we do to solve this? And that is something supposed We're still kind of wrestling with what happens in that environment. That remains a question for me.

Exit pulling. How do you do exit pulling in this weird COVID year where many people have mailed in ballots? How does that game change and how will you do the exit pulling this time around? Is it easier because of the mail in votes or is it harder because of the disparities in the different ways that votes are

being cast in different places. Well, the mail in votes will have to be interviewed by phone, so that'll either be a random digit dial phone looking for somebody voted by mail, or where possible, you take the voter rolls, you know the folks who returned mail ballots, and you attach a phone number to that you call them. So that's a change. We've always been able to reach out and we have reached out to people who voted by mail by telephone. That's always been a component of the

exit polls. However, this year obviously is going to be a much larger one in the early vote. What we're doing this year, and this is changed or it's rather enlarged or expanded, is positioning exit poll interviewers at the early voting sites, so they're out there now and they'll be interviewing people who are lined up for early voting the same way they would have been on election day

as people are leaving the polling place. And then the final part of that, the actual election day exit poll is going to have to deal with the fact that polling places are consolidated, there may be more people at those polling places, and there will be COVID protections in place, so the interviewers will be wearing a mask, they'll be hand sanitizer at the table. All of those are sort of added this year. As far as the actual sampling is concerned, people sometimes think I sort of take every

chance I can to address this. People sometimes think that the exit poll goes to a set of Bellweather precincts. That is not the case. It is a randomly sampled set of precincts, and so that sampling will have to be based on where they open up polling places, and then people will be deployed out to a sample of them.

In the more conventional way, decision desk. You sit on the CBS decision desk, And these decision desks are fascinating institutions to me because I would say, and my day job as constitutional law professor, that the decision desks, though unmentioned in our written constitution, have come to be crucial elements in how we in fact do the constitutional practice

of deciding who won elections. We don't have in the United States, but a lot of countries have a central electoral Commission for the whole country that says, here are the results. We have fifty states which each have to do their processing on their own, and we never wait for all of those to count all of the votes before we quote unquote call the election. You call the election. And I don't just mean UCBS, I mean you, the person sitting in charge of the decision desk with your team.

So how does it work and how different do you think it's going to be in this strange year. It's going to be different. People may need to be patient. However, I don't think if folks pay attention and they listen to us and they watch, I'll do a plug here if they watch CBS News, but really, if they watch a good network broadcast, they won't be confused because and this goes to a question on how does it work. What we do is we report what we see unfolding

from an event, which frankly has already happened. Pretty unusual from a pulling perspective, right for once in your lives, you're not predicting a future event. You're sort of predicting a past event. That's exactly right, that's exactly right. We are like a puzzle being revealed piece by piece. There's votes in a bank somewhere, there's votes at a voting center that are being reported, and the puzzle pieces you could think of them as sort of county by county.

As that's revealed, we start to see a picture, and what we're doing our best to do is report what that picture looks like, perhaps a little ahead of anybody else that could see it that doesn't necessarily have the tools that we have at our disposal, and then telling you this is what we believe has been revealed. So that's the big picture of how it works. What we do, specifically is we combine a bunch of different kinds of data that we're collecting. One of them is the exit polls.

So we've got a sense from talking to voters who've left their polling place. Now, sometimes in a state that's completely lopsided, that's enough to make a projection about what's happened. Well, we've heard from voters and so and so has a fifty point lead. Okay, that person has one after the polls have closed. But in most cases, and in certainly in battleground tight states, that's not enough. So now you start to get county vote coming in, and what we do with that is we look for patterns in it.

You can look for patterns and you can model data in a bunch of different ways. So one would be to compare it to pass vote. Let's suppose in county after county you see Joe Biden running five points better than any Democrat has than the last time, typical Democrats

or whatever. Well, if you get twenty counties and in every single one he's in sagurated example, every single one he's five points better than past Democrats, well you can make a pretty good inference about what he'll do in the remaining counties, maybe five points better than what if the Democrat did there, and you can extrapolate that out to what he might get statewide, and off you go

to get a statewide estimate and maybe a projection. But where it gets more difficult is, let's suppose you get county by county and there's a lot of variants in that. So Joe Biden in some places is doing five points better, someplaces fifteen points better, and someplaces ten points worse and some places twenty points worse. Well, there's no clear pattern in that. So what do you do with it? Well,

not much. You wait before you make a projection. What I would add after all that is if you see a network say oh we're waiting for vote someplace or it's too close to call, that's not always the case. What I will tell you, when I try to tell you is the patterns here are uncertain. We can't get a read on this just yet. It's not that we don't have any information. It's that that information looks like

a big cloud instead of a straight line. And that I think, I hope is helpful to people as we are transparently trying to describe for you what we see being revealed as it's revealed. I don't think of myself as filling a constitutional duty, and I remind people we do not as networks seat anyone in office. We don't certify any votes. We are reporting, I would dare say, merely reporting what it is that the voters have done.

And that's not a sort of false humility. It is, in fact the case, as much as the bright lights and TV cameras make it seem like it's very, very important well, and the fact is that the candidates also act on that basis of information. At least back in the day, when there were concessions and declarations of victory that were credible, they were often based on precisely the data that you were aggregating. Thank you for taking the time, Anthony,

really thank you. I really appreciate it. No, thank you, and I hope we get to talk more soon. Talking to Anthony Salvanto led to a few powerful takeaways for me. Most significantly, Anthony made the point that a lot of the reliability of the existing polls depends on people who say they are first time voters and that they will vote for Joe Biden. The lead that Biden has in so many polls is, he says, in important ways, a product of believing those voters what they do will have

a large impact on the outcome of the election. Another their takeaway is just how different this time around may be from previous times. People will be voting in different places than usual, and that means that exit polling also has to follow a different approach. What's more, there will be mail in votes, potentially in very large numbers, and

those two need to be incorporated into changed models. Last, but by no means least, there's Anthony's idea that individual polls, like the kind he conducts, give you the music of the election, as opposed to the aggregated beat that comes from adding lots of polls together, and for him, the

music of the election. The song of this election is Democratic voters very eager to get rid of Donald Trump on the one side, and deeply loyal Trump voters, for whom being a Trump supporter is even more important than being a Republican on the other. In other words, the future of our republic is genuinely at stake. Remember to tune into our special series Deep Bench this Saturday. Until the next time I speak to you, be careful, be safe, and be well. Deep Background is brought to you by

Pushkin Industries. Our producer is Lydia Gencott, our engineer is Martin Gonzalez, and our showrunner is Sophie Crane mckibbon. Theme music by Luis Guerra at Pushkin. Thanks to Mia Lobell, Julia Barton, Heather Faine, and Carlie mcliori, Mackie Taylor, Eric Sandler, and Jacob Weisberg. You can find me on Twitter at Noah R. Feldman. I also write a column for Bloomberg Opinion, which you can find at Bloomberg dot com slash Feldman.

To discover Bloomberg's original slate of podcasts, go to Bloomberg dot com slash podcasts, and if you liked what you heard today, please write a review or telefrat. This is deep background