The polls may be wrong but they are certainly not a waste of time

“Howard surges to a winning position”, the headline in The Sydney Morning Herald gushed a couple of weeks ago (week ending September 11, 2004).

The paper was reporting on the latest opinion poll that it had commissioned ACNielsen to do. On the same day, The Australian reported on its Newspoll saying, “The first-week fear campaign on interest rates has lifted the Howard Government’s primary vote above the ALP”.

Well, maybe. Maybe not.

Both newspapers reported changes between their previous polls and the latest one, and drew their conclusions on a few percentage points of recorded change.

In The Sydney Morning Herald’s case, the “surge” was based on a 3 per cent increase in the Coalition’s two-party preferred vote and a 4 per cent increase in its primary vote. This was based on a sample size of 1,415. The margin of error with such a sample size is a tad over 2.5 per cent either way. So if the polling was done 20 times you would expect the results to be within 2.5 per cent either way 19 times and one time beyond that range. You could be 95 per cent confident that total population’s voting intention is within 2.5 per cent either way. But if you took 2.5 per cent from the Coalition and gave it to Labor, there would be no “surge” at all.

Similarly with Newspoll. Its sample size of 1,734 is slightly higher so its margin of error is slightly lower at 2.35. Again, the lift of 2 per cent in the Coalition’s primary vote was less than the margin of error of the poll. Yet that was the basis of the report’s conclusion.

The pollsters are quite clear about the margins of error and level of confidence. But the newspapers put such details in the fine print and place the journalists’ conclusions from the poll in large type on page one.

Incidentally, the margin of error does not change much with increasing voting populations. It would not be much different if only 100,000 people were voting, rather than 13 million. But the margin of error increases dramatically if you reduce the sample size. This is why statements about the aged vote or the young vote can be way out. As can state-by-state or marginal-electorate figures.

But more than the margin of error can go wrong.

The margin of error is a mathematical calculation that presumes a random sample. But polling samples are not randomly selected, in the same way, say, as a roll of a dice or a toss of a coin. The pollsters do their best, but they would be the first to acknowledge that a random sample is not possible in a voting population of 13 million, given the constraints of time and money.

For a start pollsters use the phone. That biases the sample as the sample includes only people who can afford a phone; who have a landline rather than a mobile; who are more likely to be at home; who have broadband internet or a dedicated internet line (allowing the phone to be free for the pollster to ring); who live in households where the phone is answered (rather than going straight to answering machine) and so on. Maybe those biases correlate to a tendency to vote for one party or the other. Maybe young, mobile voters are less likely to get polled.

Pollsters attempt to select randomly within a household rather than taking the person who happens to answer. Even so, practicalities require pollsters ultimately to take someone, skewing towards the person who picks up the phone. Those people may be more likely to vote one-way or the other. Voting intention is a difficult thing to poll, unlike, say, income level, whether one is likely to buy a car, be interested in cable television, and so on. It is based upon many factors, some of which might change before polling day. Newspoll said only 55 per cent of those polled said their stated preference was final (and even those could still change their mind).

The voting system does not help the pollster. The party with the most seats wins. A party can get a majority without winning the popular vote. That has happened on several occasions in the past 50 years - most recently in 1998 and, before that, in 1990. Further, the preferential system muddies the water. The number of people voting for minor parties and independents is usually much greater than the polls’ margin of error. Yet their vote trickles through to the major parties and determines the outcomes. Just how that vote trickles through is difficult to measure. The closeness of elections also makes polling difficult. Since World War II, the widest two-party-preferred split was 43/57; the narrowest was 49.9/50.1; and in more than half the elections the margin was less than 2 per cent each way.

It makes it easy for the pollster to get the result wrong, while being basically correct about the state of Australian political opinion as in the place is fairly evenly divided.

So are opinion polls a complete waste of time? Certainly not. They provide a great deal of reader fascination, even if their meaning is stretched. More importantly, they help the working of democracy, especially polls that ask voters’ attitudes to certain issues. Polls help political parties respond to voter concerns while leaving room for a good leader to make decisions which are unpopular in the short term but can be explained and accepted in time. Polls can prompt parties to dump leaders who have no chance of winning, such as Labor’s Simon Crean.

Last and least, they are of great value to the polling companies. Political polling puts their opinion-gathering techniques to the test. Their sampling is tested (and in difficult conditions) because on polling day the whole voting population is counted. If the polling companies get it right, businesses can be confident that the company will also get research on products and consumer attitudes right, and so the polling company is worth paying.

ACNielsen says, “More than 9,000 clients in over 100 countries rely on ACNielsen's dedicated professionals to measure competitive marketplace dynamics, to understand consumer attitudes and behaviour, and to develop advanced analytical insights that generate increased sales and profits.”

Newspoll advertises itself as having “the best record of any Australian market research company in election surveys”. And there are plenty of opportunities - 2000 is the only calendar year since World War II in which there was not a federal, state or territory election somewhere in Australia.

They were the polls that counted.