Did the national polls get it so very wrong?
Yes, polls. Again. Why am I still talking about them?
Well, I see a lot of statements on the subject, here and elsewhere, that are just plain incorrect. That’s not new, either; it’s been going on as long as I remember. And not just concerning the recent election (the one I’ll mainly be discussing here); about past ones, too. So I think it’s good to point out a few salient facts—and then everyone can keep on arguing about the value of polls, the methodology of polls, and the intentions of pollsters.
To begin with, it’s necessary to understand that polling is inherently difficult to get right. That makes it flawed as a predictor (as I often say) but not totally flawed by any means. And besides, it’s the best predictor we’ve got. Crowds don’t matter, nor do yard signs (as I explained here, and as Romney discovered in 2012).
But if one pollster keeps getting it wrong when all the other pollsters get it right, that pollster isn’t going to last long, or if they last for a while they won’t have a good reputation. Pollsters’ most financially productive money-making activities—internal polling for candidates, and market research for businesses—depend on maintaining their reputation for accuracy, so they are interested in getting it right. There are also organizations that rank pollsters for accuracy over time, so among people who hire pollsters (for example, for market research, which is lucrative), the pollsters’ reputations are known.
But what if most of the pollsters get it wrong? Or did they?
I have yet to understand why so many people seem to think that the polls were so very, outlandishly wrong this year (except at the state level in a couple of states, particularly Wisconsin; but state polls are known to be much more difficult to do in terms of sampling, and are often somewhat outdated by the time the election rolls around). Why do I say the national polls were not so far off? Take a look at what the polls were actually saying the day before the election, rather than what you think you remember about what they were saying.
Remember also, when you look at this article (written one day before the election), that Hillary Clinton (so far, anyway) has won the popular vote by 1.27%. So any poll that showed her winning the popular vote by around that figure would have been remarkably accurate:
With just one day to go until the 2016 presidential election, Hillary Clinton’s support is at 46.8% while Donald Trump’s is at 44.3%.
When analysts refer to the Democrat’s 2.5 percentage point lead, they are talking about the difference between those two figures in polling averages.
It’s an important number but it’s also probably an inaccurate one. That’s because polling averages do not capture fully a few factors that will affect the final outcome.
So, the average of polls right before the election was only off from the actual result by 1.23%, and they got the popular vote winner (which is what national polls measure) right. That’s really pretty close.
The article goes on to state some of those factors that are not reflected in the polls but which affect the final outcome—early voters, turnout, margin of error, and vote distribution and the Electoral College (this turned out to be a very large factor which determined Trump as the winner). The article also mentions an unusual amount of uncertainty for this particular election. It doesn’t mention the undecideds, but we have learned since the election that they broke disproportionately for Trump, and this may indeed have made the difference between his winning and losing.
So I actually don’t see that the polls were especially inaccurate.
Take a look at what Nate Silver was saying two days before the election (Silver isn’t a pollster, but he analyzes polls and makes predictions based on them, and he has a pretty good track record):
In an appearance on ABC’s “This Week,” the FiveThirtyEight chief claimed Clinton is a “2-to-1 favorite,” but noted that recent polls show Republican presidential nominee Donald Trump with a slight edge in electoral college-heavy states like Florida, Ohio, and North Carolina.
“The electoral college math is actually less solid for Clinton than it was for Obama four years ago, where four years ago we had Obama ahead 320-some electoral votes. Clinton has about 270,” Silver said.
“So she’s one state away from potentially losing the electoral college. You’d rather be in her shoes than Donald Trump’s, but it’s not a terribly safe position.”
Finally, take a look at Nate Silver’s list of polls taken right before the election, from an article written two days before the election. Fourteen polls are listed. Two show Trump leading (by +1, and by +8). One shows a tie. Four more show Clinton leading by either 1% or 2%—and since it turns out she actually won the popular vote by 1.27% (at the time of this writing), I’d say those particular polls were highly accurate, the most accurate of all. That means that seven out of fourteen polls either got the winner right (Trump) or got the actual vote right or extremely close to right. That sounds accurate to me for six of the polls, which is approximately half (one of those that got the winner right was very inaccurate, so I’m not adding that to the “accurate” list, although it was right about the Trump win).
What’s more, three more of the polls measured Clinton at +3 in the popular vote. That turned out to be only 1.73% off. Not too shabby, either. That leaves two polls that said Clinton +4, and two that said Clinton +5. So the majority (9 polls) got it right or came quite close. And the LA Times poll, one of the two that got the winner right, was also the most inaccurate poll, because it showed Trump ahead by +8, which was actually 9.27 points off, whereas many of the polls that got the winner wrong actually were quite accurate about the popular vote.
Polling is flawed, as I’ve said almost every time I write about it. It’s just a guide. In races that aren’t close, it can be a very good guide. In close races (and this was definitely a close race) polls are iffy. But, contradictory to many people’s recollections, most of the polls this year right before the election predicted a close race. And if most also predicted a Clinton victory in the popular vote—well, they turned out to have been correct about that, too.
Here’s what Silver wrote two days before the election, and remember again that in the actual election Clinton seems to have won by about 1.27% (and that figure could actually rise over time, and has been steadily doing so):
On average across the 14 polls, Clinton’s lead is 1.9 percentage points…Or you can take the median instead of the average, which is also 2.5 points. Use only polls rated A-minus or higher? Her lead is 2.6 points. Use live-telephone polls only? It’s 2.3 points. Use only polls used to determine eligibility for the presidential debates? That gets you to 3.5 points. Use only the very recent surveys, which conducted all of their interviews in November? Her lead averages 2.3 points in those.
Not so very far off. And after reading the following from that same November 6th article by Silver, you may agree with me that Silver (who works with the poll figures as a whole, analyzing and commenting on them) was surprisingly accurate:
How about polls of swing states in particular? Right now, the tipping-point state in our forecast ”” the state that would provide the decisive 270th electoral vote if the polls got things exactly right ”” is New Hampshire. There, Clinton leads by only 1.7 percentage points in our adjusted polling average, as several recent polls show Trump tied or slightly ahead, along with others that still give Clinton the lead. Thus, Clinton’s doing a little bit worse in the tipping-point state than she is overall ”” a sign that she might win the popular vote but lose the Electoral College.
In case you’re unaware of what happened in New Hampshire, Clinton won—47.62% to Trump’s 47.25%, a margin of 2687 votes out of a total of 694,307 cast. So the polls in New Hampshire seem to have been pretty good; they predicted both a very close race (true) and the eventual winner in that state.
I was very impressed with Nate Silver’s predictions in 2012, and I remain impressed. And the polls weren’t so shabby, either.
In a prior lifetime I did a lot of pretty high level statistical analysis, which is why I don’t think the earth is warming and why people don’t understand polls.
Neo is right that for a close election (and this one was close) it is the individual state results are. These are expensive to do compared to national polls and more difficult to compile on a national basis. Had a pollster decided early on to concentrate on swing states and not spend effort at all on all states or national polls they might have done better. There is a non-randomness in small changes in key races that is not easy to model (and probably not meaningful in politics).
Trump won simply because of the late day disclosures on Clintons email etc. Kellyanne Conway somehow kept Donalds mouth shut during that time. Hillary simply got the bottom of the deck in the last days.
Right now Donald is looking better (except for a few wild tweets) because it’s quite and people are hoping for normal people selected for cabinet positions. Soon he will have to speak on many issues as a manager not a candidate. Watch the polls.
By the day of the election, the national averages were within spitting distance of being “accurate”, though I still question using error bars that aren’t recalculated based on larger sample sizes. But did the electorate really change their minds in the last two weeks that much? I doubt it- you had an A+ poll swing from Clinton +12 to tied in a week’s period before swinging back to Clinton +5.
And the state polls in the what were identified as swing states did indeed underestimate Trump’s chances across the board. Also, state polling isn’t harder to do than national polling- that is a canard. It no more difficult to call up 3000 people nationwide and get a response level than it is to call up 3000 people in Wisconsin and get a response level.
Yancey Ward:
I don’t have time to look it up now, but there are many reasons state polling is harder. Some have to do with sampling, if I recall correctly, but one of the biggees is that there are many states and it’s too expensive to keep taking polls often enough.
And yes, in this election, people changed their minds a lot. And many didn’t make up their minds till the end. Neither candidate ever polled anywhere near 50%. A lot of wiggle room.
Neo,
The only objection that makes sense is the one about cost, but that is only true if you try to do national and state polls together. One could have jettisoned all the national polls taken every single week and invested all the funds in polling the following states that one or both camps spent significant time in:
FL, NC, IA, OH, MI, NH, NV, and PA.
Do that polling well, and you could have predicted the winner easily. It is a mystery to me why national polling gets the lion’s share of presidential polling dollars- as this election and the one in 2000 shows, it is the electoral election that matters all of the time.
“Trump won simply because of the late day disclosures on Clintons email etc. Kellyanne Conway somehow kept Donalds mouth shut during that time. Hillary simply got the bottom of the deck in the last days.”
Would add obamacare – renewal notices just coming out with extraordinarily large increases.
That probably hits at many of the voters in the rust belt that swung trump’s way.
The absence of discussion of this issue, in general, in the MSM, is remarkable.
Humbug.
I keep seeing this argument from the intelligentsia.
Here is the rub Neo, do that same analysis with the polls one week earlier, or one month earlier.
Think those showing Clinton up 11 a week or two before the election were accurate? They were breathlessly reported as such.
My Boss predicted this, and I lost $50. He said that the week of the election all the polls would suddenly get “right”, because that last poll is the only one by which they will be judged. He nailed it, and made me look like an idiot.
What would the final vote tallies have looked like had all the polls except the LA Times, been as accurate as their last?
The media are not just biased in their coverage, and the polls are a large part of why people are reacting negatively to the media. It is something they have seen over time and thru the years.
However, if even neo refuses to call it out, and gives them a pass, then they will continue their battle space prep in future elections.
What kinds of polls do you think the campaign’s spend money on? They spend all of it on state level polls, not national ones- they just don’t publicize the results.
And it is a lesson I keep needing to relearn myself- I questioned Trump’s focus on Michigan and Pennsylvania in preference to Virginia, but they clearly had much better data than any public polls of those three states, even though Trump also ran closer in VA than any public poll showed, but he lost by enough of a margin that I can’t question his lack of focus on it now, at least not too much.
What Jim said.
Even if the final week’s national polling average was fairly close to the popular vote result, the same cannot be said for a good number of the national polls that were published earlier, which tended to show a comfortable (if not large) lead for Clinton throughout the months of September and October. In particular, I remember the ABC tracking poll moving from something like Clinton+10 to Trump+1 within the space of a single week shortly before election day, which is too large a swing to be credible. So, even if the national polling was, on average, somewhat accurate on the eve of election day, the same cannot be said for much of what was published before that time, raising the question of pollster shenanigans.
Whoops. I posted under “Neo” instead of “Meh.” My apologies for that.
“Even if the final week’s national polling average was fairly close to the popular vote result, the same cannot be said for a good number of the national polls that were published earlier, which tended to show a comfortable (if not large) lead for Clinton” – not Neo and Jim
So, y’all don’t think the Comey announcement and the obamacare notices in the final week or so had ANY impact whatsoever?
With millions fewer voting (as it turns out), historically high negatives for presidential candidates (#1 and #2 worst), and historically high undecideds all along, there was bound to be large swings.
Maybe a percentage point or two, which is significant but not an earth-shattering 11-point swing significant. The large majority of people have made up their mind by the final week.
Polling CAN NOT work. Statistical analysis is only valid for relatively stable systems with many actual outcomes to validate the model.
Nate Silver made his bones predicting baseball games. You have 30 teams playing the same 13 players in the same ballparks and they play 2,340 games a year. Lots of outcomes to test the predictions. Even then there is an inherent 6% error rate according to the 538 website.
Elections are a whole ‘nother animal. Each election has a different electorate – every year 2.6 million eligible voters die and 2.6 million come of voting age. The candidates are different. The issues are different. The world is different. And you only have one result.
The polls are a SWAG – a Scientific Wild Ass Guess.
Big Maq, 5:31 pm — “So, y’all don’t think the Comey announcement and the obamacare notices in the final week or so had ANY impact whatsoever?”
Meh, 5:47 pm — “Maybe a percentage point or two, which is significant but not an earth-shattering 11-point swing significant. The large majority of people have made up their mind by the final week.”
And to confound the analysis, my understanding is that an unusually large number of people voted early this year, maybe in the neighborhood of 20 percent (?), which introduces something of a dent into any Comey late announcement effect.
The real bias inherent in national polling is the fact there are 50 elections for President. National polls just set up the electorate for disappointment when the popular vote doesn’t line up with the results of the 50 state elections.
Jim Doherty; Meh; M J R:
It’s never good to try to reconstruct these things without looking them up and being aware of the total picture, rather than picking and choosing your memory of a certain poll or two.
First of all—yes, there were swings this year, in part because neither candidate was liked and there was a fairly large undecided group compared to previous years. So the news du jour could have quite an effect.
And yet the swings were not nearly as big, on average, as what you might think you remember. Here’s Nate Silver, writing three weeks before the election:
So, it wasn’t such a large swing after all, measured overall, from that high point until right before the election.
More from Silver, 3 weeks prior to the election [emphasis mine]:
So it had been a close race for a long time, and then it crept up, and about three weeks before the election it reached what turned out to be the highwater mark for Clinton, a 6-7 point lead. Then it started to close again pretty quickly, until right before the election it had reached the levels I wrote about in the post.
That Nate Silver piece, three weeks before the election, was written on Oct. 16. The Billy Bush tapes that caused a backlash against Trump had been released on Oct. 8. By voting day, the polls had closed again to a very small margin between the two candidates. I see no reason to believe that the swings that were measured in the polls were not accurate. They seem very reasonable and quite accurate to me.
Jim Doherty:
Take a good long look at my comment right above this one.
But in addition, I’d like to say the following to you—just because the “intelligentsia” says something, it doesn’t mean it’s not true.
Also, you write:
I assume that if you have read my comment above this one, you are aware of why I “refuse” to “call it out.” But let me assure you—although I have no idea why I would even need to assure you of this—I have no particular interest in defending pollsters. I am not a pollster. None of my relatives or friends are pollsters. I don’t even play a pollster on TV. I look at the data, study the arguments pro and con, and come to the best conclusions I can, based on that. I have no particular agenda. I call them (or call them out) as I see them.
Just before the election, I thought Mr. Trump was more likely than not to win, but I had to be realistic in assuming it could have gone the other way. It’s nice to be able to watch TV now without getting bombarded by political ads, since I live near a swing state.
Voter turn-out for 2016 will probably have been around 55% of eligible voters, even with the first woman and the first non-politician candidates. And while Mrs. Clinton got more popular votes, such a result is not as controversial as it was in 2000, with the margin so very close in that one state of Florida.
The state-by-state results are very interesting, as it is harder to see how a more conventional Republican candidate could have won the same states that were the key to Trump’s electoral victory, without the same appeal from his personality and his policies.
One also wonders if Trump could have won the states of Minnesota, Nevada, and New Hampshire, if there had not been the same third-party candidates (and if Trump had less personal flaws). Johnson and Stein both received many more votes than in 2012.
Again from the state results, Mr. Trump had the broad appeal that was needed to win (with 30 states and over 300 electoral votes). He also had multiple avenues to success, even with all the votes still uncounted, which was why Mrs. Clinton conceded early on.
Maine even awarded Mr. Trump one electoral vote, from its second district, which went 52% to 41% for Trump (nearly the opposite of the first district, which went 54% to 39% for Clinton; there really are two different Maines). The results in Massachusetts were pathetic, 60.8% to 33.5% for Clinton, which leads one to wonder:
Just who did Neo-Neocon vote for, anyway?
Neo,
In reality, Clinton never had a 6/7 point lead in October–Nate and the pollsters were wrong. Think about it: if that were true, then Clinton should’ve won the popular vote by something like 4/5 points, as Trump cleaned up among the 15% that were still undecided in October by 10-15 points (doubtless the FBI controversy was a factor here). So, the math suggests that the race was always tighter than the October polls were (on average) suggesting.
The final RCP average was Clinton +3.2, as of October 17 the average was Clinton +7, which amounts to nearly a 4 point shift in the national polling average within the space of the final two weeks of the campaign when less than 15% were still undecided (per Nate). That is too big a shift at the very end of the campaign to be credible, as it would translate to Trump winning undecideds by more than 25 points! Trump did well among late-breaking undecideds but not that well.
Nate Silver simply assigns a percent chance of winning to the established favorite. He rarely if ever takes a contrarian viewpoint. Look at his NFL picks from this week. Every single one was the Vegas favorite, and the one slight upset predicted (the LA Rams) inevitably lost. His sure-fire lock of the week, with a robust 84% chance of winning? The KC Chiefs, who also lost.
There were only three surprises in this presidential election–Pennsylvania, Michigan, and Wisconsin. Silver (and the rest of the pollsters) whiffed on every one of these states and thus predicted Hillary as the winner. If he called just one of those states for Trump, he would have stood out from the crowd of pollsters who said exactly the same thing as him.
In 2008, all Silver did was predict that every state that had a chance to vote blue would. You had a popular candidate, the assurance of a white guilt vote, and an uninspiring old moron as opposition in McCain. It was like shooting fish in a barrel. But the Left is naturally suspicious, they project their own voter fraud schemes onto the Right, they worried about some “vast right-wing conspiracy” to steal the election. So when one of their own (Silver) had the “courage” to predict the election landslide that all of us on the right also precisely saw coming, he was hailed as a “genius”. He has since gone on to make no predictions of any significance, save for Obama beating Romney, and the two times the pre-tournament favorite went on to win the NCAAs in basketball (UNC in 2009, Louisville in 2013).
It’s a tough job, but polling just comes down to people reassuring the public how comfortably the expected winner is going to win by, backed up by math or no.
@Meh – Ya wanna make it as something that is only based on your “feeling” rather than getting back to the measurable whys of your argument.
You linked to RCP’s national average. Fine, even though when one digs into it, we really ought to be looking at state polls as more relevant to the ecv win. But please walk us through how you get to your conclusion from there.
Also, examine the “margin of error”, as one factor. You haven’t shown why that doesn’t explain it.
Big Maq,
This isn’t about “feelings,” I’m just teasing out the implications of how the 4 point shift seen in the national polling average from mid-October to just before election day is not consistent with how undecideds actually broke for Trump during that time. My conclusion is that even if it is granted that the final national polling average as of November was roughly accurate, the same cannot be said for October.
However, I remain unconvinced that even the final week’s national polling average was accurate. If Greg Phillips is correct in saying that over three million non-citizens participated in the 2016 election, then Trump would have almost certainly won the popular vote if only eligible citizens had voted, and the final national polling average would’ve been off by roughly 3 points (assuming the illegal vote goes overwhelmingly for Clinton).
If what you guys are saying, neo included, were true, then sometimes those polls would show the republican in the lead and he would under perform on election day. But its such a coinky dink that its always dem’s who are shocked. I grant we went thru a spell when we bitched about the weighting etc.
Still. Its always the dems who are over sampled, remember how the exit polls showed bush losing, and the shock when he won reelection?
I am not a conspiracy crank. But how many times do we have to see the media pretend to be subtle, when they are not?
Jim Doherty:
As I’ve said before, there are reasons there are trends in polls, and those reasons need have nothing to do with purposeful bias. For example, basing turnout on a previous election is a typical one.
Plus, there were times when Trump was ahead in the polls, as you would see if you actually look at timelines of the changes in the polls. What’s more, various surveys (LA Times in particular) almost always showed Trump in front. That was because they used a different polling method.
As I’ve also said, purposely and falsely putting Clinton ahead could have backfired on her just as easily as it could have helped her. So it makes no sense even conceptually.
Meh:
Silver’s late average was considerably lower than that, as this article makes clear. If I recall correctly, the RCP average averages in too many earlier polls from that Billy Bush week when Clinton was uncharacteristically higher, and gives a false picture of what’s happening towards the end.
And let’s have some actual figures for how those “undecideds” broke:
That’s a lot of people breaking for Trump, and in close states it turned the tide.
One more thing—for your 10:30 AM comment to be correct, one would have to assume that only the previously self-labeled “undecideds” can change their minds. That is a false premise.
Neo,
Right. Nationwide Trump wins the 15% who were undecided in October by about 15 points or so, which comes to a 2 point national shift in his favor, with the margin of victory varying from state to state. However, if the polls in October were right and Clinton had a 6/7 point lead coming into the final leg of the campaign, then she should’ve won the popular vote by about 4/5 points. This is why so many people were confident that Clinton would win.
Sorry, but the others were pretty well locked in. Apart from Clinton having a seizure during a campaign event, or being publicly incarcerated by the FBI/NYPD, the people who were committed to voting for her in late October were not about to change their mind.
Meh:
They were right 3 weeks prior to the election. Then the polls changed as the weeks passed and the election got closer, and they changed not just because the undecideds changed their minds or made up their minds. There were still plenty of undecideds right before the election, and the polls had already changed.
Other people, who felt they had decided, changed their minds, and kept changing their minds. I know such people. We don’t know how many there were.
Nor were the polls perfect. But they were pretty darn good.
Neo,
No they weren’t! As of October 18, the RCP national polling average was Clinton +7.1. If that was right, then she should’ve won unless Trump wins undecideds by something like 30 points nationwide, which wasn’t going to happen.
Look, we can also get into the specifics of how certain polls were rather drastically “re-weighted” from D +9 to more reasonable samples as election day got closer. It’s much easier to say that the polls were off in October, but were more reasonable in November (though still too favorable to Clinton).
Meh:
You keep ignoring my main point, which I’ve made over and over, about a fallacy in your reasoning about which voters might be changing their minds.
Hint, hint: it doesn’t have to just be people who call themselves “undecided.”
It’s a waste of time to talk to you further about it. Go back and study what I’ve already written.
Polls are designed to manipulate public opinion, not reflect it. Once people start trying to parse out if it is true or false, the trap activates.
@Neo – agree it is worthless carrying on.
Once people have in their mind that everything is essentially “rigged”, then they will always have a story for why that is so.
Might as well be arguing over how a thermometer is biased because the manufacturer, the weatherman, … everyone! …, is aligned to deceive us into believing in CAGW.
There is no basis to believe in any objective observation, especially when it is based on the black arts of statistical analysis.