What was wrong with the polls?

Being this far “wrong” is a commercial disaster. If you can’t call an election right, how do we know you are telling us the truth about what sort of baby wipe people are looking for.

So their explanations are diversions meant to get them through to the spot where clients have forgotten the failure.

Ipsos is suggesting establishing a polling council, to police the accuracy of polls. Like they have in the UK. Because the British Polling Council was so successful in ensuring that British polls accurately picked the Brexit vote. OK, well maybe that isn’t a solution after all, just another level of bureaucracy.

Labor pollster John Utting suggests spending more on polling to get a more representative sample, and more transparency so we can see what pollsters are cooking-up. Fine, except even in our meta sample the pollsters were out, so the larger sample didn’t help, and who is going to judge the quality of the calculations seeing none of them can get them right in the first place?

Utting claims the Australian Bureau of Statistics gets its polling right. One wonders how he knows this. They might have larger samples, but this just reduces sampling error, and makes it possible to do more fine-grained dissections, which may be just as wobbly as what we’ve seen in the last weeks.

It appears that the political parties didn’t do much better themselves. Labor used Galaxy to do its tracking polling, so it is not an effective check, but the Liberals used Crosby Textor. You could tell by the chatter coming out of the coalition, combined with the seats Scott Morrison chose to campaign in, C|T had no idea they were sitting on 10 per cent swings in central and northern Queensland, for example.

So, back to what went wrong?

There are a number of candidates. It is unlikely to be anything to do with young people and mobile phones, as some have suggested. The four pollsters we looked at use different methods. Ipsos and Newspoll use telephones, but this includes mobile phones. Essential is online only – no mobile problem there, and Roy Morgan is face to face. If there were a mobile phone problem it would only show up in half of them.

There is a problem however with all polling being essentially opt-in, unlike elections which are compulsory. When I ring someone, or approach them face-to-face, they can choose not to talk to me. Internet panels are incentivised, and again, are restricted to people who want to be involved.

This means that no polling sample is going to be representative of the population at large. And it is less representative of one group more than most – young men, followed by young women – because they are the least likely to want to talk. Even with huge resources it is very difficult to get enough of them.

Most pollsters explicitly “fix” these and other sampling problems by “weighting” except that weighting doesn’t fix anything. By inflating the size of an insufficient sub-sample you don’t cure the inaccuracy inherent in the small size, you amplify it.

Not only are polls opt-in, but generally part of their sample is invented.