The polls say it’s neck and neck. But can you even trust them?

Presidential election night in 2016 and a high-spirited crowd has gathered at the Javits convention centre on New York’s Eleventh Avenue for Hillary Clinton’s victory party. A few blocks uptown, at the Hilton hotel, Donald Trump’s supporters have settled in for a night of drinking, commiserations and jeering before finally calling it quits on the “Make America Great Again” experiment.

Clinton’s event begins with “jubilation and festive chaos”, Nathan Heller later observes in the New Yorker. A queue snakes around the building, vendors do a brisk trade in badges declaring “Madam President” and, for some unknown reason, a man wearing striped red stockings dances in the street with “a huge stuffed fish”. Inside the venue, the first results are declared for Clinton and the crowd cheers. It is as expected: pre-election opinion polling has declared Clinton ahead by a country mile, with Reuters, CNN and The New York Times polls estimating her chances of victory at around 90 per cent.

Around 7pm, though, results dribbling in from Florida are not quite as the Clinton campaign had predicted. At 10.53pm, Florida falls to Trump: a shock. By 1.30am, Trump has sealed the key battleground of Pennsylvania. In the Javits Centre, the mood shifts. At 2.35am, a shell-shocked Clinton phones Trump to concede. She later tells the BBC: “He was very obviously ebullient, and I wished him well and said if I could help I would do so – and it was painful.” Clinton later reveals she had not even prepared a concession speech.

The other loser was the polls, which had clearly failed spectacularly. Everyone knew they were not infallible: UK surveys earlier in the year had wrongly found voters would not support Brexit. Still, the likelihood they would so badly underestimate Trump’s support was, for the most part, brushed aside. Today, the polls have Kamala Harris and Trump in a dead heat. Yet the recent track record of polls in the United States invites the question, should we trust them? How do the polls work? Why do we pay attention to them? And how do they get things wrong?

What polls are we talking about?

Quinnipiac University, set amid the verdant hills of Hamden, Connecticut, conducts a US presidential election poll. So does Marquette Law School, a minnow with just 195 enrolled students in Milwaukee, Wisconsin. So do Reuters-Ipsos, CNBC, The Wall Street Journal, The New York Times and literally hundreds of other organisations. They’re all taking the pulse of the US public during election campaigns in an effort to forecast the result.

Some of these scientific soothsayers predict how the nation will vote as a whole. Some focus on individual states, either out of local interest or to see how a state will shake down in the so-called electoral college votes (which are doled out by each state and territory and which, ultimately, determine the presidency). Some even poll individual voting districts, whose relatively small numbers of voters might, like the flap of a butterfly’s wing, swing a state and then the nation one way or the other. A handful of polls are considered “gold standard”; some are less reliable. Few, if any, do it in the same way. But right now, they are in furious agreement: this election is too close to call, whichever way you try to predict it.

Polling is commonplace worldwide. In Australia, the main operators include Roy Morgan, Freshwater Strategy, Essential, YouGov, Newspoll and Resolve, which polls the nation for this publication. Nowhere, though, do they conduct polls like they do in the United States. “We are a nation obsessed,” observes Richard Kreitner, a contributing writer in The Nation. “Americans are drowning in opinion polls. New surveys are released at a rate of about a dozen per day.”

‘Even for those who could not afford large stakes, betting in the run-up to elections was a cherished ritual.’

Economists Paul Rhode and Koleman Strumpf in a 2003 paper

The first modern-style “straw poll” was carried out by The Harrisburg Pennsylvanian in 1824, which correctly predicted that Andrew Jackson, who had fought in the War of Independence, would win the popular vote, although he didn’t, ultimately, snag the presidency. Still, the polling we recognise today took another century to properly emerge.

In the meantime, people gauged the likelihood of results by watching the betting markets. From 1896 until the 1920s, mass-market newspapers provided almost daily odds from early October until election day. “Even for those who could not afford large stakes, betting in the run-up to elections was a cherished ritual,” observed economists Paul Rhode and Koleman Strumpf in a 2003 paper. The published odds, they noted, “proved remarkably prescient and almost always correctly predicted election outcomes well in advance, despite the absence of scientific polls”.

From 1916, meanwhile, Literary Digest magazine had started its own crude poll by sending postcards asking about voting intentions to millions of names harvested from telephone directories and automobile registries. It correctly predicted presidential winners until its method unravelled in 1936, when it called the election for the Republican Alfred Landon, a millionaire oil baron, before Democrat Franklin Roosevelt achieved the largest electoral college margin in history – 523 electoral college votes to just eight. (Literary Digest folded in 1938, its failure proving a prescient warning to emerging pollsters.)

Enter George Gallup, a former advertising executive, remembered as the father of modern polling methods. Unlike Literary Digest, with its scatter-gun requests, Gallup focused on smaller groups who were tailored to better represent the wider population, according to characteristics such as age, gender and race. He correctly predicted the 1936 race – and launched an industry. Others followed, each with their own methodology, including the pioneering Simulmatics Corporation, which deployed an IBM 704 computer the size of a large room and offered its services to help a lagging John F. Kennedy win the White House in 1960.

Today, pollsters employ an array of methods to reach appropriate sample groups and then to crunch the data. Typically, says Jim Reed, creator of the Resolve poll published in this masthead, people are chosen at random and contacted either online, by phone call or text. For polls on voting intention, Reed gets people to fill out an online ballot paper that appears as it would in the actual poll because emulating the reality of voting is key to accuracy. “It’s the arty bit of the science; understanding what the real choice is and how to reflect that,” he tells us.

The next revolution in polling will be AI, says Reed. “It’s not here yet, but it’s not far away. AI can be somebody who looks and sounds real and engaging and it understands what they’re saying. We will hopefully get rich and accurate data in a cost-effective way.”

‘The forecast is objective, but the way people look at probabilities is very contingent on what their assumptions are.’

Nate Silver from aggregation site FiveThirtyEight

Others, such as the Washington Post and The Economist, combine the results of multiple polls to average out a result. Nate Silver left his job in management consulting in 2004 to analyse the careers of baseball players after designing a prediction system. Ahead of the 2008 US election, he began to notice flaws in how polls were interpreted, and set up the aggregation site FiveThirtyEight, named after the total number of electors in the electoral college. It correctly predicted all but one state that year and forecast them all correctly in 2012.

Like most polling outlets, the website underestimated Trump in 2016, but gave him a 29 per cent chance, a much greater chance than most predictions. “The one thing that said Donald Trump could become president was the polls. The polls showed him within the margin of error,” Silver said in 2020. “The forecast is objective, but the way people look at probabilities is very contingent on what their assumptions are.”

Why are polls useful (or not?)

Many of today’s polls are never made public. Instead, they guide political parties’ election campaigns, identifying whose votes they need to win and with which policies. “Polls leading up to an election are similar to the process of a student submitting a draft essay to a teacher for feedback, giving them a chance to revise their writing before the final submission and grading,” says Professor Scott Sisson, a data scientist at the University of NSW. “The polls allow the election parties to get real-time glimpses of how their messaging is resonating with the voting population.”

‘What we have demonstrated is how data from past situations can be used to simulate a future situation.’

The Simulmatics Corporation’s recommendation to John F. Kennedy in 1960

That was key to Kennedy’s campaign in 1960. As Jill Lepore reported in The New Yorker in 2020, the fledgling Simulmatics Corporation recommended Kennedy try to improve his popularity with African-American voters by leaning into his Catholicism – aligning him with other oppressed minorities – and take a stronger position on civil rights. He convinced enough previously sceptical African-American voters to support him, taking six of eight essential states he needed to win. The company subsequently boasted: “What we have demonstrated is how data from past situations can be used to simulate a future situation.” George Gallup, however, notes Lepore, “had notoriously failed to include African-Americans in his surveys, not least because he depended on the financial support of Southern newspapers, which were unwilling to print the opinions of Black Americans, especially about race relations.”

Polls give the public a chance to be heard ahead of an election, says Shaun Ratcliff, president of polling group Accent Research. “The people that go out of their way to make sure they’re talking to candidates, or elected representatives, tend to be people that are either really well-organised or have a burning issue they’re passionate about. Polls give elected representatives and candidates a sense of what the electorate as a whole is thinking.”

Reed believes that as long as public polls are accurate they serve a democratic purpose, especially in cases where a voter might mostly receive information that already aligns with their views. “Published polling is a real reset to the echo chamber. Voters get to see the truth that there are lots of people with different perspectives, and it gets us to question our own views on politics and issues of policy. An informed electorate is healthy.”

It’s a view that echoes that of James Wechsler, who wrote in 1940 that polls gave voice to the common man. “What have the answers to good questions, addressed intelligently to the right cross-section, proved about America?” Wechsler asked in his piece for The Atlantic. “Above all, they have revealed an incredibly animated populace anxious to articulate its fears, resentments, and loyalties.”

‘People ... are looking for instances of whether or not ‘your side’ is going to win or lose, be the vanquished or the vanquisher.’

Lloyd Cox, Macquarie University

Others see polling as amplifying the horse race aspect of politics. “There’s a certain satisfaction and entertainment value that comes from following polls and attempting to divine what might or might not be the outcome,” says Lloyd Cox, a senior lecturer in US political history at Macquarie University. “I think people find it fascinating and are looking for instances of whether or not ‘your side’ – and increasingly we take sides – is going to win or lose, be the vanquished or the vanquisher,” he tells us. “Politics, like entertainment, has villains and heroes, storylines and outcomes. It has revenge and all of that stuff that we watch Netflix for every night. There’s a line of thought that suggests US politics is increasingly like the World Wrestling Federation. And it’s no accident, therefore, that Donald Trump was part of that entertainment world before he became part of the political world.”

More pragmatically, poll results can help people prepare for potential outcomes, says Sisson. “For example, predicted election outcomes tend to get priced into financial markets, where the market anticipates a particular outcome and so is not surprised when the full results are known. Reliable polling information is critical for this to work well.” (The Australian dollar has fallen ahead of the US election while global investments in gold and bitcoin have risen.)

Then there are the potentially negative effects of polls on the election process. Simulmatics, for example, was branded immoral by some in 1960 for helping to shape Kennedy’s policy to suit how voters would be likely to respond. Political scientists, meanwhile, have theorised about a so-called contagion, or bandwagon, effect where voters rally behind parties that are doing well in polls, or don’t bother to vote because they think it’s unnecessary, a theory that could help explain the 2016 Clinton-Trump result.

Others believe polls can influence people to shift their votes at the margins. “If it’s a close contest it focuses people on who they really want to win,” says Reed. “But if it’s a clear win then you can see people starting to send a message with their vote with a minor party, for example.”

Some polls are designed to cater to a particular voter group, says Scott Sisson. “This includes shoring up support amongst a party’s core voting base by presenting results that support what that group wants to hear, or targeting particular segments, such as the female vote, or first-generation migrant vote, in swing areas, or just trying to demoralise the opposition voter base by claiming the opposition party are performing poorly and so deterring voters from turning out for the opposition – ‘It doesn’t matter if I vote because they have already lost.’”

How accurate are polls?

“Scepticism of the polls is entirely justified,” writes G. Elliott Morris in his 2022 book Strength in Numbers: How Polls Work and Why We Need Them. “In the past 20 years, they have routinely underestimated the attitudes of conservative Americans, who often refuse when pollsters ask by phone or email to interview them. And while the polls, on average, are relatively unbiased, they are fantastically bad in some situations.”

Still, any pollster worth their salt will tell you the poll can’t entirely predict the outcome on election day. “Strictly speaking, polls are not predictions, they’re a snapshot in time,” Reed says. Shaun Ratcliff agrees: “It’s a bit like a weather forecast, it’s not going to be 100 per cent right all of the time. But, if done well, what it will do is give you useful information that’s an estimate of what the reality is.”

‘You can ask loads of people, How are you going to vote? but judging who will get off their sofa and cast a vote adds an extra dimension.’

Pollster Jim Reed, creator of the Resolve poll

Famous misses also include: 2020, when the polls correctly predicted a Biden victory but were way wide of the mark on the margin; 1980, when Ronald Reagan trounced Jimmy Carter as the polls pronounced it too close to call; and 1948, when pollsters were so sure of the result that the Chicago Tribune printed the next day’s edition with a banner headline declaring that Thomas Dewey had defeated Harry S. Truman. Fate was not to be tempted: Truman triumphed in a landslide.

We often hear about “margins for error”. In a tight race like this US election, this is critical. Given that polls might include between a few hundred or few thousand people, their likelihood of accurately representing the general population must include a margin of error, or a sampling error, that takes into account inadvertently not asking exactly the right kind of people the right kind of questions. If this margin for error is typically taken to be 3 per cent in either direction, contests that are closer than that are impossible to call. The New York Times-Siena national poll, which spoke to 2516 voters nationwide, had Harris and Trump tied at 48 per cent of the vote just days ahead of the election, meaning that either candidate could actually be ahead. “Unfortunately, for everyone who wants to know the result before it happens, it’s very close and within polling margin errors, so we can’t tell who is actually going to win,” Reed says, though he notes, “Without polling you wouldn’t know that. It’s telling us this particular race is close.”

There are other potential pitfalls. Online polls potentially skew away from voters without internet access and towards a better-educated or wealthier average. What’s known as social-desirability bias can see respondents not tell the truth to pollsters because they are embarrassed or feel it’s socially unacceptable to, say, admit they intend to vote against a minority candidate. Then there’s the challenge of reaching people who don’t want to engage in the election until, possibly, the actual day itself.

It can be difficult work. Researchers had to place more than 80,000 calls to fill out enough respondents for the New York Times-Siena poll. Ratcliff says caller scepticism of spam calling, which occurs at higher rates in the US than Australia, makes “trying to get respondents to actually answer and complete the survey tricky”. Non-compulsory voting also makes polling in the US a challenge. There’s no wizardry to predict who will come out to vote on the day. Says Reed: “You can ask loads of people, ‘How are you going to vote?’ but judging who will get off their sofa and cast a vote adds an extra dimension.”

And it involves careful course correction. As the results are gathered, researchers might contact particular groups so the sample better mirrors the overall population, say, by making sure there are enough young men (who are, apparently, particularly unlikely to respond to online surveys). They then “weight” the final results to smooth over any groups who’ve answered disproportionately. “You hope to get to the end with minimal weighting,” Reed says. This year, polls in the US are more likely to boost the weight of responses from people without college degrees, to make up for the poorer Republican voters who are less likely to respond to surveys.

(Republican voters are notoriously difficult to reach. “There is no reason to assume pollsters can reach the least politically interested voters in sufficient numbers, and there is plenty of reason to think they will back Mr Trump in November,” suggests Nate Cohn in The New York Times. “If this challenge remains just as great, the polls might miss badly yet again. Perhaps the polls could fare even worse if Mr Trump fares even better among less engaged voters than four years ago, as polls have shown all cycle.” )

What are the alternatives?

In the US there has been a resurgence in betting on events such as elections through a mechanism called “prediction markets”, online gambling hubs where you essentially take a position on one side of a bet to somebody with a different view. One of the largest, Polymarket, claims that studies show “prediction markets are often more accurate than pundits because they combine news, polls and expert opinions into a single value that represents the market’s view of an event’s odds”. A week out from election day, it was reporting that US$2.7 billion had already been wagered on the outcome, with around 65 per cent of bettors predicting a Trump win.

“In theory, prediction markets have potent strengths,” writes the FT’s Oliver Roeder. “They harness the wisdom of an incentivised crowd, they return instant results, and they are sensitive seismographs of a spasmodic campaign season.” Shaun Ratcliff is more sceptical. “Normally, the people betting in those don’t have any inside information that would give them a better idea of who’s likely to win than what we would see from the published polls,” he says. “They tend to be shallow markets, so a number of bets from individuals are perhaps not based on good information, but they’re betting based on what they want to see happen. Or in some cases, they’re actually trying to sway the markets.”

‘My gut says Donald Trump … But I don’t think you should put any value whatsoever on anyone’s gut – including mine.’

Nate Silver

US historian Allan Lichtman, known as the “Nostradamus” of US politics, eschews polls altogether, instead using 13 “keys” that are either true or false about the candidate and their party, including projections for the economy, foreign affairs and whether the candidate is charismatic. He has predicted all but one presidential election since 1984, miscalculating Al Gore’s loss in 2000, which famously came down to a few hundred votes in Florida, and says eight of his keys are true for Harris this year, enough to assure a win.

What about gut feeling? Nate Silver, who today has his own website called Silver Bulletin, declared in an op-ed for the New York Times two weeks out from this election that “50-50 is the only responsible forecast”. Still, he addressed the inevitable question: what does your gut say? “My gut says Donald Trump … But I don’t think you should put any value whatsoever on anyone’s gut – including mine.”

In Australia, pollster Jim Reed has delivered the news to politicians in the past that their prospects are too close to call. “If that’s the honest answer to come out of your research, that’s what you’ve got to say.” There’s no reassurance or opportunity for soul-searching in this, he adds, but “it’s a huge motivator to campaign as hard as you can until the ballot boxes close”. And for voters? “It tells them their vote counts. Maybe we might see a higher turnout than we have previously in the US for this reason.”

Get fascinating insights and explanations on the world’s most perplexing topics. Sign up for our weekly Explainer newsletter.