Scientists, politicians and public policy

On July 6, 2008 Australia’s Prime Minister, Kevin Rudd, announced the imminent release of “a new report on the impact of climate change on drought”. The CSIRO and the Bureau of Meteorology had presented the government with some “very disturbing” findings:

What they say in two short points is this, and this will come out later today - firstly that when it comes to exceptional or extreme drought, exceptionally high temperatures, the historical assumption that this occurred once every 20 years has now been revised down to between every one and two years, that's the first point.

Secondly, with exceptional circumstances drought conditions, under scenarios within it, that that will occur twice as often, and with twice the area of droughted parts of Australia included. Now this is a serious revision of the impact of climate change on drought and the Agricultural Minister will make that clear in the report that he releases later today ...

At a press conference some hours later, the Minister for Agriculture, Fisheries and Forestry, Tony Burke, said that some of the predictions of drought he had received in the new report “read more like a disaster novel than a scientific report”. The Minister claimed that the conclusions of the report meant that “we now know what would happen if we did nothing”, and that “we know for certain from today ... that the ground rules, because of climate change, have themselves changed” (emphases added).

Asked how confident he could be that predictions based on modelling were accurate, Mr Burke replied that “What we’ve done is taken the best climate scientists in Australia and asked them to come up with their best information”. The scientists had looked at a range of projections, and “we all hope that we can work our way forward to reducing the greenhouse gases in the atmosphere and start to ameliorate what some of these projections are based on”. However, “the bottom line ... is that this has been handed to the government, but it’s entirely written from beginning-to-end by scientists”.

In a further press conference on the following day Minister Burke repeated that he’d taken his information from “the best available science we can get” and that “At every level, the news is simple: it’s just getting tougher and likely to continue to get tougher in the years to come”. Averring that all of the projections went to the same conclusion that there would be “varying levels of increased hardship”, Mr Burke reiterated that his “priority on all of this is to make sure we go with the best available science and I do believe that’s what the CSIRO and the Bureau of Meteorology gave us yesterday”.

Predictably, the Minister’s “disaster novel” metaphor made headlines in Australia and overseas.

According to The Guardian (London), the new report predicted that Australia “will be hit by a 10-fold increase in heatwaves and ... droughts will almost double in frequency and become more widespread because of climate change”. Together with the Garnaut report published some days earlier, the CSIRO/Bureau of Meteorology report “would put pressure on Prime Minister Kevin Rudd to act swiftly on his pledge for Australia to lead the world in tackling polluters”.

The report’s findings were accepted without question by the government and by most commentators, despite the fact the report had not been subjected to any external review. However, Dr David Stockwell, author of Niche Modeling and host of the Niche Modeling website, saw a need to check on the tests that the authors had conducted to determine the statistical significance of their conclusions. For this purpose, he asked CSIRO for the results from each of the 13 models that had been used to produce the drought projections. To his surprise, this request was initially refused. However, following extensive canvassing of the issue on several other websites, the data were eventually released some three weeks after the publication of the report.

Dr Stockwell has now published the results of his review of the report, using the detailed information that is available here. He has found that:

• all 13 climate models failed internal validation tests for regional droughted area in Australia over the past century;
• simulations showed increases in droughted area over the last century in all regions, while the observed trends in drought decreased in five of the seven regions identified in the CSIRO/Bureau of Meteorology report;
• in almost all cases, the correlation coefficient between simulated and observed values was very low, and not significant;
• in all cases the “coefficient of efficiency” was negative, indicating that the climate models simulated drought area worse than simply using the mean;
• in almost all cases the difference between the means of the return periods was significant, indicating that the frequency of droughts in the models has no relationship to the actual frequency of droughts;
• as the model simulations have no resemblance to observed droughts in the last century, the models have failed internal validation and no further testing is warranted;
• contrary to statements in the report, there is no credible basis for claims of increasing frequency of Exceptional Circumstances declarations; and
• as there is no logical connection between the extreme values of models and simulations using different global warming scenarios, the report’s claim to have performed analysis using high global warming scenarios is illogical and invalid: this is true, irrespective of whether or not the underlying climate projections are themselves invalid.

Dr Stockwell recognises that some of the models show apparent skill at some statistics in some regions, but says that this is of no importance:

For example, the climate model miroc-h from the Centre for Climate Research, Japan, shows good agreement with observed return period in most regions. However, it has not been subjected to more rigorous external validation, and it has performed poorly in other tests ... used in a study including three co-authors of the DEC report.

In the light of these findings, Dr Stockwell suggests to the authors that “Studies of complex variables like droughts should be conducted with statisticians to ensure the protocol meets the objectives of the study”, and offers some examples of how known statistical methods might be used to improve drought projections. More generally, Stockwell concludes that “Policy making based on science should not be influenced by studies that appear scientific but do not adhere to the generally accepted validation practices”.

Dr Stockwell’s critique of the CSIRO/Bureau of Meteorology report does not constitute a rejection of the need for the application of the scientific method in climate science modelling. On the contrary, his analysis is in itself an affirmation of the need for a rigorously scientific approach in this area - and his successful efforts to bring about the publication of all supporting information will not only facilitate the external scrutiny of the Drought Exceptional Circumstances Report that its importance demands, but will also enable other experts to seek to reproduce Stockwell’s analysis. This can only lead to more enlightened public policy.

It should be stressed that Dr Stockwell’s study does not demonstrate that droughts in Australia will not become more frequent and/or more severe. But if his analysis withstands expert scrutiny he will have shown that the report on future drought events by Australia’s two leading climate science research bodies does not meet basic tests of climate model validity and cannot serve as a guide to policy.

It is truly unfortunate that the CSIRO and the Bureau of Meteorology did not undertake a prior evaluation of the models used in the first study of its kind in Australia in order to ensure that those models were able to reproduce the past situations that were relevant to the study.

The website of the Statistical Society of Australia Inc lists the names of more than 100 accredited statisticians who wish their contact details to be made public. There is now an opportunity for those experts who have the requisite skill and inclination to carry out their own analysis of the data used in the CSIRO/Bureau of Meteorology study and to report the results. In doing so, they would be performing a valuable public service.