The polls may be wrong but they are certainly not a waste of time

“Howard surges to a winning position”, the headline in The Sydney Morning Herald gushed a couple of weeks ago (week ending September 11, 2004).

The paper was reporting on the latest opinion poll that it had commissioned ACNielsen to do. On the same day, The Australian reported on its Newspoll saying, “The first-week fear campaign on interest rates has lifted the Howard Government’s primary vote above the ALP”.

Well, maybe. Maybe not.

Both newspapers reported changes between their previous polls and the latest one, and drew their conclusions on a few percentage points of recorded change.

In The Sydney Morning Herald’s case, the “surge” was based on a 3 per cent increase in the Coalition’s two-party preferred vote and a 4 per cent increase in its primary vote. This was based on a sample size of 1,415. The margin of error with such a sample size is a tad over 2.5 per cent either way. So if the polling was done 20 times you would expect the results to be within 2.5 per cent either way 19 times and one time beyond that range. You could be 95 per cent confident that total population’s voting intention is within 2.5 per cent either way. But if you took 2.5 per cent from the Coalition and gave it to Labor, there would be no “surge” at all.

Similarly with Newspoll. Its sample size of 1,734 is slightly higher so its margin of error is slightly lower at 2.35. Again, the lift of 2 per cent in the Coalition’s primary vote was less than the margin of error of the poll. Yet that was the basis of the report’s conclusion.

The pollsters are quite clear about the margins of error and level of confidence. But the newspapers put such details in the fine print and place the journalists’ conclusions from the poll in large type on page one.

Incidentally, the margin of error does not change much with increasing voting populations. It would not be much different if only 100,000 people were voting, rather than 13 million. But the margin of error increases dramatically if you reduce the sample size. This is why statements about the aged vote or the young vote can be way out. As can state-by-state or marginal-electorate figures.

But more than the margin of error can go wrong.

The margin of error is a mathematical calculation that presumes a random sample. But polling samples are not randomly selected, in the same way, say, as a roll of a dice or a toss of a coin. The pollsters do their best, but they would be the first to acknowledge that a random sample is not possible in a voting population of 13 million, given the constraints of time and money.

For a start pollsters use the phone. That biases the sample as the sample includes only people who can afford a phone; who have a landline rather than a mobile; who are more likely to be at home; who have broadband internet or a dedicated internet line (allowing the phone to be free for the pollster to ring); who live in households where the phone is answered (rather than going straight to answering machine) and so on. Maybe those biases correlate to a tendency to vote for one party or the other. Maybe young, mobile voters are less likely to get polled.

Pollsters attempt to select randomly within a household rather than taking the person who happens to answer. Even so, practicalities require pollsters ultimately to take someone, skewing towards the person who picks up the phone. Those people may be more likely to vote one-way or the other. Voting intention is a difficult thing to poll, unlike, say, income level, whether one is likely to buy a car, be interested in cable television, and so on. It is based upon many factors, some of which might change before polling day. Newspoll said only 55 per cent of those polled said their stated preference was final (and even those could still change their mind).