What was wrong with the polls?

Living with uncertainty is the thing most humans find most difficult.

And while it’s virtually impossible to be certain even about the exact details of the past, it is completely impossible to be certain about the future.

Yet that is the standard we demand of our pollsters, even when they don’t attempt to measure the future.

In this last election every single one of the published national polls failed to get the exact numbers right for what would happen in the future.

So how well did they do at measuring the past, given the uncertainty inherent even here? Newspoll had the ALP on 52 per cent of the vote two days out; Ipsos a bit earlier had them at 5, as did Essential and Roy Morgan.

All of these pollsters use slightly different sample sizes and different methods. Morgan and Ipsos have samples of around 1500 and 1800, but the rest are around 1000.

In the final result the ALP won 48.39 per cent of the party preferred.

Now the sample error on 1,000 is ±3.1 per cent at the 52 per cent mark (Newspoll’s result), which means 95 per cent of the time the actual figure on the whole of your population could be anywhere from 55.1 to 48.9 per cent. And five per cent of the time it could be even further out. So Newspoll was in the five per cent zone, and the rest got results that a competent pollster would get, based on the thing they weren’t measuring – election day results.

There might be a systemic thing happening here though because they are suspiciously close together.

Because we have four pollsters we can also do some meta-analysis and combine their results to effectively give us a larger sample with Labor vote, by weighting the individual results, of 51.2 per cent. The sample error is now 1.49 per cent, and which is well-outside election day as in 95 per cent of cases, we would expect that to range between 49.68 per cent and 52.68 per cent.

Either the pollsters are incredibly unlucky, or something is wrong, but what?

Don’t pay any attention to the industry. The reason they spend so much time on elections is that this is one of the few times you get a proof of concept where you can measure the accuracy of your polling against an actual outcome.