Why are public opinion polls failing?

Generally speaking- and opinion polls of opinion pollsters agree – the failure of public opinion polls is entirely due to the people polled.

US President George W Bush put it accurately, if somewhat rather clumsily, when he said:

It's no exaggeration to say that the undecided could go one way or the other.

Quite frankly, the people aren't being fair to public opinion pollsters – mind you, that is just my opinion.

The American Association of Public Opinion Research (AAPOR) has released its autopsy of the 2016 US presidential election and. given the fact that AAPOR is the industry association for public opinion polling organisations, it was unlikely to be severely condemnatory of the self-evident fact that their member organisations very largely – in fact, overwhelmingly – got the actual outcome catastrophically wrong. Hillary Clinton was so bedazzled by these pre-election polls that she didn't give a nanosecond's thought to preparing a concession speech on election night last November.

The Executive Summary of the AAPOR report begins:

The 2016 presidential was a jarring event for polling in the United States. Pre-election polls fuelled high-profile predictions that Hilary Clinton's likelihood of winning the presidency was about 90 percent, with estimates ranging from 71 to over 99 percent.

Having got that self-evident truth off its corporate chest, the AAPOR report then tried to defend its own member organisations. Frankly many of these defences are specious while many are demonstrably absurd but overall the defence is that lots of people either fibbed to the pollsters or just changed their mind on election day or – mostly probably - both.

The report states:

National polls were generally correct and accurate by historical standards. National polls were the most accurate in estimating the popular vote since 1936. Collectively, they indicated that Clinton had about a 3 percentage point lead, and they were basically correct; she ultimately won the popular vote by 2 percentage points. Furthermore, the strong performance of national polls did not, as some have suggested, result from two large errors canceling under-estimation of Trump support in heavily working class white states and over-estimation of his support in liberal-leading states with sizable Hispanic population.

If anybody should be acutely aware of the US Electoral College system of electing their President you would think American polling organisations would be which, of course, makes national polling figures almost meaningless and, in this case, deceptive.

It continues, "State-level polls showed a competitive, uncertain contest" yet it goes on to say, in apparent denial of this uncertain contest conclusion that "Clinton appeared to have a slim advantage". Does a "slim advantage" actually mean an "uncertain contest"?

Then AAPOR gets to the real nitty-gritty. "There are a number of reasons as to why polls under-estimated support for Trump" with the three main reasons – excuses – being:

Real change in vote preference during the final week or so of the campaign, adjusting for over-representation of college graduates was critical, but many polls did not do it and some Trump voters who participated in pre-election polls did not reveal themselves as Trump voters until after the election, and they outnumbered late-revealing Clinton voters.

In its heroic defence of its members' failures, AAPOR states:

A spotty year for election polls is not an indictment of all survey research or even all polling.

Describing monumental failures as spotty is certainly creative. I'm surprised AAPOR didn't publish a poll "proving" that 97.93% of Americans thought that public opinion polls were God's honest revealed truth.

And, again addressing another key nitty-gritty question:

About those polls that Clinton was 90 percent likely to win AAPOR bravely asserts. However well-intentioned these predictions may have been, they helped crystallize the belief that Clinton was a shoo-in for president, with unknown consequences for turnout.

It's as close to an admission that pollsters shot themselves in the foot that AAPOR will come.

AAPOR still struggles with the fact that Trump won asserting "Donald Trump's victory in the 2016 presidential election came as a shock to pollsters, political analysts, reporters and pundits". Well, yes it certainly did come as a shock to the overwhelming majority of these people whose natural inclination was for Clinton. They could not – or would not – countenance the idea that they might be wrong. This admission by AAPOR inadvertently reveals a deep-seated hostility to Trump and a sublime belief in their own particular prejudices. None of this lot would have been among those voters dismissed and denigrated by Clinton as deplorables.

The media doesn't escape criticism: for example, "The pattern in early voting in key states were described in numerous, high profile news stories as favorable for Clinton, particularly in Florida and North Carolina. Trump won both states". So sloppy, unprofessional journalism is to blame as well, right? The AAPOR is silent on whether or not this media bias towards Clinton so aggravated likely Trump supporters that they flocked to vote late in the day.

In what could point to some possible outcomes in Australia both for the same sex postal ballot and the possible appeal of One Nation and other minor parties, the report determined that voters with higher levels of formal education tended to be Clinton supporters and that these elites were more likely to respond to surveys than people with lower levels of education. Post election, this was called the "shy Trump" factor. The fact is however low their level of education these deplorables as Clinton disparaged them know when they are being told what to think and what to do by those who consider themselves their betters.

Recent years have not been all that wonderful for polling organisations around the world.

In 2014 in Scotland's independence referendum, most British polls showed independence losing in a close race – in fact, it was lost overwhelmingly 55.7% to 44.3%. In 2015, Prime Minister David Cameron won a stunning victory at a time when polls suggested his chances of winning were, at best, very slim.

Cameron then was fooled into believing that a Brexit poll would back his stance that the UK should remain in the European Union. False opinion polls suggesting Brexit was unpopular and that voters wanted the UK to stay in the union were decisively wrong and cost him the premiership.

And opinion polls that showed that Prime Minister Theresa May would win the UK general election this year in a canter were also horribly wrong and she came within a whisker of being defeated by Jeremy Corbyn's Labour Party. She lost 13 seats and the Conservative majority, Corbyn won an extra 31 seats and now May has to rely on the Northern Ireland's DUP for confidence and supply.

In France, opinion polls ahead of the 2017 presidential election showed the top three candidates in contention but none of these made it through to the runoff between Emmanuel Macron (who won) and Marine Le Pen.

French opinion polls were so badly wrong that the leading newspaper La Parisienne took a break from writing stories based on polling results.

Pollsters in France excused the woeful results of their polls by saying that only 12 percent had confidence in political parties and were voting all over the place. How do they know that? Why public opinion polls of course.

You would have to feel sorry for outgoing President Francois Hollande who decided not to seek a second term because of poor public opinion results. Perhaps, just maybe, he was panicked by some false polls.

Kevin Rudd who liked to present himself as a man of inflexible principle once candidly admitted:

Of course I base all of my election promises and policies on opinion polls.

Perhaps that is why he ultimately lost so badly.

Mind you, that's just my opinion.