Tired of polls?
Well, I’m not.
Not quite, anyway. Here’s a good catch by Datechguy. He notes that Morning Joe, etc. “are doing their victory dance” based on some new Quinnipiac polls showing Obama way ahead in both Florida and Ohio.
He next alerts us to this Hugh Hewitt interview from August of 2012 with Peter Brown, assistant director of Quinnipiac. It explains a lot. You may or may not believe this is the way polls are conducted, and you may or may not believe Peter Brown. But from my earlier research I think this is what’s going on with at least some of the pollsters, and I actually think Quinnipiac is probably among them.
Hewitt begins by asking the reasoning behind Quinnipiac’s models that show large Democratic percentages such as 8 or 9 per cent:
PB: Well, what is important to understand is that the way Quinnipiac and most other major polls do their sampling is we do not wait for party ID. We ask voters, or the people we interview, do they consider themselves a Democrat, a Republican, an independent or a member of a minor party. And that’s different than asking them what their party registration is. What you’re comparing it to is party registration. In other words, when someone starts as a voter, they have the opportunity of, in most states, of being a Republican, a Democrat, or a member of a minor party or unaffiliated…
HH: Now what I don’t understand this, so educate me on it, if Democrats only had a three point advantage in Florida in the final turnout measurement in 2008, but in your poll they have a nine point turnout advantage, why is that not a source of skepticism for people?
PB: Well, I mean, clearly there will be some people who are skeptics. This is how we’ve always done our polls. Our record is very good in terms of accuracy. Again, remember, we’re asking people what they consider themselves at the time we call them.
HH: But I don’t know how that goes to the issue, Peter, so help me. I’m not being argumentative, I really want to know. Why would guys run a poll with nine percent more Democrats than Republicans when that percentage advantage, I mean, if you’re trying to tell people how the state is going to go, I don’t think this is particularly helpful, because you’ve oversampled Democrats, right?
PB: But we didn’t set out to oversample Democrats. We did our normal, random digit dial way of calling people. And there were, these are likely voters. They had to pass a screen. Because it’s a presidential year, it’s not a particularly heavy screen.
HH: And so if, in fact, you had gotten a hundred Democrats out of a hundred respondents that answered, would you think that poll was reliable?
PB: Probably not at 100 out of 100.
HH: Okay, so if it was 75 out of 100”¦
PB: Well, I mean”¦
HH: I mean, when does it become unreliable? You know you’ve just put your foot on the slope, so I’m going to push you down it. When does it become unreliable?
PB: Like the Supreme Court and pornography, you know it when you see it.
HH: Well, a lot of us look at a nine point advantage in Florida, and we say we know that to be the polling equivalent of pornography. Why am I wrong?
PB: Because what we found when we made the actual calls is this kind of party ID.
HH: Do you expect Democrats, this is a different question, do you, Peter Brown, expect Democrats to have a nine point registration advantage when the polls close on November 6th in Florida?
PB: Well, first, you don’t mean registration.
HH: I mean, yeah, turnout.
PB: Do I think”¦I think it is probably unlikely.
HH: And so what value is this poll if in fact it doesn’t weight for the turnout that’s going to be approximated?
PB: Well, you’ll have to judge that. I mean, you know, our record is very good. You know, we do independent polling. We use random digit dial. We use human beings to make our calls. We call cell phones as well as land lines. We follow the protocol that is the professional standard.
HH: As we say, that might be the case, but I don’t know it’s responsive to my question. My question is, should we trust this as an accurate predictor of what will happen? You’ve already told me there”¦
PB: It’s an accurate predictor of what would happen is the election were today.
HH: But that’s, again, I don’t believe that, because today, Democrats wouldn’t turn out by a nine point advantage. I don’t think anyone believes today, if you held the election today, do you think Democrats would turn out nine percentage points higher than Republicans?
PB: If the election were today, yeah. What we found is obviously a large Democratic advantage.
HH: I mean, you really think that’s true? I mean, as a professional, you believe that Democrats have a nine point turnout advantage in Florida?
PB: Our record has been very good. You know, Hugh, I”¦
HH: That’s not responsive. It’s just a question. Do you personally, Peter, believe that Democrats enjoy a nine point turnout advantage right now?
PB: What I believe is what we found.
HH: Geez, I just, and an eight point in Ohio? I’m from Ohio. Democrats haven’t had an eight point advantage in Ohio since before the Civil War. I mean, that just never happens, but Peter, I appreciate your coming on. I’m not persuaded that Quinnipiac Polls haven’t hurt themselves today, but I appreciate your willingness to come on and talk about it.
To understand this you have to understand the concept of random sampling (that is, unless you think it’s all a pro-Democratic plot by pollsters, in which case that’s all you have to understand). If sampling is truly random, then the percentage of Democrats answering the poll (and who say they plan to vote) will accurately reflect the percentage of Democrats who would actually be voting if the election were held that day. To adjust for wild skews (for example, if the sample ended up being 75% Democrats), the pollsters could use something called “stratified sampling” to adjust for the problem.
I believe that’s correct, anyway. But I’m not a pollster, nor do I play one on TV.
By the way, here’s Gallup’s polling method:
The findings from Gallup’s U.S. surveys are based on the organization’s standard national telephone samples, consisting of directory-assisted random-digit-dial (RDD) telephone samples using a proportionate, stratified sampling design.
From what Brown says, Quinnipiac doesn’t stratify its polls unless there’s a huge red flag staring them in the face. But Gallup does. Its accuracy would depend on what parameters it uses for stratification, and how well they reflect reality.
Utterly incredible (Quinnipiac, I mean).
Utterly incredible.
The implication is that the inverse would also be correct; that the number of Republicans (or Democrats for that matter) not answering the poll would indicate the number of Republicans or Dems NOT voting on that given day.
There could be many criteria for not responding to a poll or to being unsure about voting on any given day. If the inverse does not follow from the premise (and it doesn’t), then the premise must be flawed to begin with. It’s like the old “All drug dealers tie their shoes. I tie my shoes therefore I am a drug dealer.”
T: I agree that the premise is flawed, as I think I indicated in my post yesterday when I discussed the problem with falling response rates. To believe it’s not flawed, you have to believe that responders don’t differ from non-responders in important ways. I find that somewhat hard to believe, although it’s possible it’s correct.
Peter Brown seems to be saying that because Quinnipiac’s results have conformed pretty well to the actual election results, there’s no problem.
How accurate has Quinnipiac been? In 2010, at least, they were the most accurate of all, with an error rate of 2%. That’s an average for them, though, which hides the fact that they made substantial errors in prediction as well.
I think what Quinnipiac is saying is that a lot of people don’t know what party they are actually registered in, so that if they are voting for Obama they identify themselves as Democrats. Sounds plausible.
roc scssrs: It sounds plausible until you consider that in Ohio the margin of victory for Obama in 2008 was 4%. Is it really likely he’s doubled his margin?
“It sounds plausible until you consider that in Ohio the margin of victory for Obama in 2008 was 4%. Is it really likely he’s doubled his margin?”
Especially given the fact that the Eastern and several Southern Ohio counties (basically the Ohio River border counties) are coal counties which he carried in 2008—IMO a snowball’s chance in Hell for a repeat of that in 2012.
Neoneocon, re: your response 1:42. It seems that we are on the same page here.
Random sampling is only meaningful if you have what is known as a stationary process, that is, a process whose properties are constant over time. People aren’t like that, they respond to circumstances. You’d think they’d try to take 2010 into account, an election where D identification dropped significantly and R rose to equality with D. Rasmussen combines 2004 and 2008, but even that seems questionable with all that has happened since 2008.
There’s a fly in the polling soup somewhere. And i think it comes down to all anti Obama forces tending to distrust pollsters and their agenda driven media brethren in general.
I would certainly be less likely to participate in a poll than i was 4 years ago.
If you’re not tired of polls you need to regroup, examine your immediate environment, improve it with a new fast and up the dose of any medication proven to render you unconscious until your neural circuits heal.
My guess is that Quinnipac reports an accurate poll just before the election but if you look weeks or months out, they are way off and consistently biased in favor of dems. They use the final poll to deflect criticism that they are biased. They use absurd assumptions to boost the dems in earlier polls. Ever wonder why the polls are all biased in the same direction? That is not random. They are dem operatives.
Make that “news fast….” and get to it fast.
“”They use the final poll to deflect criticism that they are biased””
Steve
Exactly. Sort of like the three networks will show their 3% of reports critical of Obama to prove they aren’t biased either.
SteveH,
I repeat my belief that there will be three categories of voters this cycle:
That second category will be made up of Independents (where Romney leads by, in some polls, 20%) and disaffected Democrats like the coal-miners of Appalachia (which will influence the Ohio and PA, and perhaps VA votes).
I never have given much thought to this whole process. However, if I was asked as a physicist to design a poll I might try something like the following:
1) Obtain voter registration records for a state. 2)I need to sample 1000 people in order to have a statistically valid set with an uncertainty of about 3%.
3)Randomly call Dems, Reps, and unaffliated until I reach a 333 actual poll responses for each of the 3 groups.
4) One question for each person: Who do you support for President: Obama, Romney, Other? and vary the order is which they are listed on a rotating basis for each successive interview.
I really don’t know how well this would work, but it seems reasonable to me.
Pingback:"I think…I think it is probably unlikely." » Da Tech Guy's Blog
Physicsguy (I’m a physics guy too) , you’d get the wrong answer because the hard part is to get the R D I weighting right. My guess is that the R’s and D’s would be nearly 100% for their candidate with only the I’s showing a split. The problem is that the electorate is never 1/3 each and it shifts from one election to the next. A seminal election like this one falls into the Black Swan category, totally unpredictable by ordinary statistical forecasting because the situation today is so different from the past.
Despite the lack of reporting by the MSM, there really is a Depression going on with greater than 16% of the working population unemployed, the U6 number. This numer outweighs any poll. No way Obama gets re-elected, and the Democratic party will get hammered too.
Steve: there’s a problem or two with your reasoning.
The first is that you need to go back and actually look at the record of each pollster over time, and by that I mean over a number of elections over a number of years, and for the trends within each election year as well. Is there a tendency to oversample Democrats earlier in the year and then to use fewer Democrats as you get very close to the time of the election? I haven’t seen an analysis that indicates that, although I suppose it’s possible. Is there a tendency for the poll results to change a lot from pro-Democrat to less so within a certain year, as the election draws near? If not, I don’t think your theory would hold water, although I suppose it’s possible it could be something that’s started in 2012.
The other thing that makes it problematic is that each poll is just a snapshot at a certain time. Is there any way to prove that any change in poll results as the campaign moves closer to the election doesn’t just represent a real change of opinion on the part of the public, rather than being a function of pollsters’ biased manipulations?
When I look at the results of Gallup polls over many years and over time within each single election year, I don’t see a lot of evidence of that sort of manipulation (at least with Gallup), although there are a few years when it might be happening (i.e. early leads for Democrats are larger, and then the race tightens—but that could just as easily be a reflection of an actual change in voter perceptions).
Paul,
True, my method doesn’t have the proper proportion of R,D, I, but at least it samples each group in equal numbers. which overcomes the objection that D’s are oversampled, and that they seem to be ones that answer the phone when the poller calls. I can definitely see flaws in my way… Oh well, first attempt.
There seems there should be a better methodology than just randomly calling numbers out of the phone directory.
Physicsguy,
One problem with the method you state is that R/D/I do not necessarily show up in equal proportions at the polling place to vote. Thus any result your 1/3. 1/3, 1/3 split creates does not necessarily predict an election result.
This causes the same problem that tinkering with the poll weighting does; how accurately will it predict voting trends?
As Yogi Berra said, predictions are hard, especially when they’re about the future.
neo, remember the 2010 midterms? The MSM was predicting the dems would hold the House. As this article notes it was the biggest swing in power in 50 years:
http://www.nytimes.com/2010/11/07/weekinreview/07marsh.html
Is it likely that the electorate is any more friendly to dems now than it was in 2010? Do current polls reflect the swing in women, independents, Catholics, etc (as described in the article)? As far as I can tell they all oversample dems using 2008 as their benchmark. What a joke!
I heard someone (I think Peter Kirsanow) say this morning that the oversampling was on purpose to reflect the turnout of the last election.
Which of course was highly dem.
But won’t be this time.
A break from all the poll analysis. I offer that this linked video is like Obama’s gravity defying lead. It looks accurate from only one point of view. However, the minute that point of view changes even the least little bit, the entire illusion falls apart and we realize that traditional gravity still prevails.
Enjoy:
http://illusioncontest.neuralcorrelate.com/2010/impossible-motion-magnet-like-slopes/
Steve: but I was making the point that Quinnipiac was only off 2% (on average) in 2010. Its record that year was the best of all the polls.
2% off sounds pretty good, until you observe that, as an average, it could include a lot of Congressional races that it called in significant error greater than 2%. However, the errors couldn’t all be in the same direction to get the 2% figure.
So how can we reconcile all this oversampling with the fact that the Quinnipiac polls were fairly accurate in 2010? If it had really oversampled I would expect them to have been more inaccurate that year. Of course, that doesn’t mean they’re not oversampling this year. Nor does it mean that other pollsters aren’t oversampling. I just don’t see much strong evidence for it.
Personally, I think they are oversampling Democrats this year, but I don’t know how much. If it’s only a little, their accuracy would only be a little off. And I think most pollsters, if they are oversampling Democrats this year, aren’t doing it to mislead. I think it’s just, as Paul in Boston says above, that this election falls into a black swan category.
Here’s an interesting take on “the polls:”
“How can you justify dutifully writing reports about a poll without mentioning it is skewed by a projected Democrat turnout larger than even the margin Obama had in 2008? How can you ignore the clear evidence that there are more registered Republicans than registered Democrats by 4.3 points (and most of that edge has come about just since June)?
A thirty-second search would tell you that the 4.3 point Republican lead in registration is more than THREE TIMES the 1.3 point lead Republicans had in 2010 when we crushed the Democrats, taking 63 House races.”
From
http://www.westernjournalism.com/why-conservative-pundits-are-the-lefts-useful-idiots/
Fox News has thrown in the towel too, for what its worth. Let’s say the polls are somewhat correct- wouldn’t these ridiculously large margins for The One candidate tend to suppress his turnout?? If you’re a honey booboo independent with not much enthusiasm for politics, and you keep hearing your preferred guy is a lock- it’s not going to be close- do you vote, or sit it out? it’s hard to scare the hell out of an electorate and motivate them to action, when victory has been assured by every news outlet in America.
I would think for every discouraged republican, there would be a lot more dems who feel no threat whatsoever when the leads are 8,9,10 points.
Just a thought.
Traditionally weekend polls tended to favor Democrats. Theoretically no one knew why, but Republican/Conservatives didn’t seem to be home over the weekend.
Lately the impression I’ve gotten from my limited circle of friends & acquaintances is that the Republican/Conservative voter is more vigilant at screening calls. Anecdotal, but very consistent in my experience.
datechguy updates with:
Update 4: Neo Neocon notes that Quinnipiac has a different defination of Big Red Flag than I do
From what Brown says, Quinnipiac doesn’t stratify its polls unless there’s a huge red flag staring them in the face. But Gallup does. Its accuracy would depend on what parameters it uses for stratification, and how well they reflect reality.
Or maybe a dem skew is considered a “huge blue flag” so that’s OK.
How many times do you get to be surprised by an event with the same cause and still call it a black swan?
Here’s an interesting point of view from the left on the Chick fil a non-boycott where it was said to be a black swan. Sorry to send traffic to HuffPo: http://www.huffingtonpost.com/ron-galloway/the-chickfila-effect-on-t_b_1731161.html
The turnout and results of 2010 are neither insignificant or inapplicable. If nothing else the Chick fil a incident of a couple months ago is a good indication that the these voters haven’t gone anywhere. If they’ll stand in line for hours for a chicken sandwich, they’ll show up to vote.
Susan: Excellent point. Although chicken sandwiches are nice, winning this election will be epic!
I am having problems imagining that the D/R/I split the polls are showing are real.
I’m not sure about the methodology that produces these results, but it seems strange.