In the Republican presidential nomination contest, the polls have been the outsiders' best friend. Billionaire Donald Trump and physician Ben Carson gained early momentum and critical debate exposure thanks almost entirely to their consistently high rankings in any number of public opinion surveys.
What those poll numbers say about Trump's and Carson's chances in the Iowa caucuses — not to mention beyond — is far murkier. Iowa conservative radio host Steve Deace gave voice to a common feeling. Deace said the polls are meaningless.
"The record of polling in Iowa is a joke," Deace said on MSNBC's News Nation on Oct. 8. "You've got the pollsters now coming out in numerous articles in recent days saying 'don't actually believe our own data.' You have Gallup yesterday made the announcement, they're pulling out of polling in this primary. They may not even poll the general election because they don't trust their methodology."
A lot of people wonder about the accuracy of the polls, and it would be a big deal if the pollsters themselves were just as skeptical. As we dug into this, we found that the reality is less dramatic. It's harder to do good surveys today, and it costs more. But it can be done. That said, pollsters have some explaining to do.
The polls didn't botch most of the 2014 races, but they consistently underestimated the Republican margin of victory. The average results showed the Democrats running about 3 percentage points stronger than what the voters delivered.
In past elections, the bias has swung the other way and favored the GOP. The statisticians might say these shifts don't prove that polling is fundamentally flawed, but try telling that to a public that has troubling wrapping its collective brain around the idea that you actually can learn something valid about 100 million voters by asking 1,100 people. (Done right, that's a pretty good sample size.)
This 2016 primary season has put even more pressure on the polling profession.
News organizations sponsoring debates have turned to the polls to decide whom to bring on the stage and whom to cut. With over a dozen Republican candidates, many of them clustered in the single digits, the differences between any two are nearly certain to be smaller than the polls' well-published margin of error.
If just 1 percentage point separates Candidate A from Candidate B, and the margin of error is 3.5 percent, you have a problem.
It's a formula for discontent.
Just because networks might be asking surveys to do more than they should, that doesn't mean polling doesn't face serious challenges.
The twin goblins that afflict pollsters are the rising fraction of cellphone-only households and a growing percentage of people who refuse to answer pollsters' questions.
The best public opinion operations employ banks of callers who are fed randomly picked phone numbers. Cellphones make it tougher to determine where the interviewee actually lives, which matters a lot for state-level polls. Plus, federal rules about calling cellphones mean you must hand-dial each one, rather than use a faster automated dialer. That's a significant cost factor.
And then there's the response rate. When more people hang up, that raises questions about whether the people who participate are unrepresentative of the public at large.
"As response rates have declined, the need to rely on risky mathematical maneuvers has increased," a recent Bloomberg article said.
Survey operators have a bunch of formulas to guide them to likely voters and reach enough people from different walks of life. Some of those methods work better than others, but you don't know which until after people vote.
Pollsters aim to neutralize response rate headaches by calling back the same number (when the person simply never picked up) over a span of several nights and making sure there's the right mix of cellphone and landline interviews.
Charles Franklin is the co-founder of Pollster.com and director of the Marquette University Law School Poll. Franklin said the survey firms, including his, spend more time getting full interviews.
"We have to call many more numbers to get a big enough sample size, but there's plenty of evidence that we're reaching enough," he said.
Historical trends collected by the National Council on Public Polls, a trade association of pollsters, track the gap between polling and actual voting results in presidential years. In 2012, the average error was 1.46 percentage points. That was slightly higher than in the previous three elections in 2008, 2004 and 2000, but less than the error in 1996 when it was 2.1 percentage points.
Jon Krosnick, a public opinion researcher at Stanford University, has assessed polls for many years. Krosnick distinguishes between polling organizations that dot their i's and cross their t's, and those that don't.
"Our evaluations of survey accuracy indicate that surveys done using best practices, which is expensive, continue to be remarkably accurate," Krosnick said. "If you look at the average of the final pre-election polls done by the neutral news organizations that used high quality methods just before the 2008 and 2012 presidential elections, they predicted the national popular vote within 1 percentage point."
Krosnick said the danger lies in the proliferation of firms that do polling on the cheap.
Online polls are everywhere, particularly on websites with an ideological agenda. In the 2004 election, 16 million people said they participated in some kind of web survey. The risk isn't that those voters know such dodgy polls mean next to nothing. The risk is that they think all polls are no better.
All that said, Cliff Zukin, a Rutgers University political scientist and former president of the American Association for Public Opinion Research, warns that no one should underestimate the problems polling faces today. Zukin said there's a huge issue of figuring out who will actually vote and all the issues only get worse at the state level where much of the attention is focused during the primaries.
"We are less sure how to conduct good survey research now than we were four years ago, and much less than eight years ago," Zukin wrote in an op-ed.
Get what you pay for
The general message is it takes more time, more savvy, and more money to do good polling. Internet polls offer some relief, and according to Steven Smith, political scientist and pollster at Washington University in St. Louis, they have proven their mettle. Smith said the best ones, such as YouGov and Harris Interactive, cost more to set up, but as the cost of the phone polls rises, "the good online polls are looking quite affordable."
But if the candidacies of Trump and Carson prove nothing else, it's that this campaign season will stretch pollsters. And the scrutiny and skepticism are unlikely to abate.
Contact Jon Greenberg at email@example.com. Follow @jonzgreenberg.