How to fix the broken scientific system

The legal and scientific processes of western democracies are based upon an adversarial system. In a court of law a prosecutor is appointed to make a case against the accused and a defence counsel is appointed to defend. Even in cases where it is completely obvious that the accused is guilty, they are still entitled to a defence, if for no other reason than to force the prosecution to prove their case. The strength of the prosecution argument is increased by its ability to stand up to the test posed by the defence. Hence there is far more public trust in the Guilty verdict. In science, scientists fight out intellectual battles in journals so that only the most battle-tested ideas emerge - or so we would hope.

The process of argument is as essential to the scientific system as it is to the legal system. A big difference is that argument is guaranteed in the legal system with the two sides of the argument formally recognised in the legal system itself, but because of the structures of the present systems in science, a robust argument cannot be guaranteed. Because of this there cannot be a sufficiently high level of faith that some of the big scientific issues of our time such as Anthropogenic Global Warming, the fate of the Murray-Darling, or the imminent demise of the Great Barrier Reef, have been properly tested in the scientific equivalent of a court of law.

The job of a defence counsel is to defend, not directly to try to find the truth. They may concede obvious points of the prosecution case, but they will mainly try to tear apart the arguments of the prosecution. The consequence is that the chance of indirectly ‘getting at’ the truth is increased. We need a similar system in science - an organization the sole charter of which is to find out what may be wrong or debatable with current scientific propositions, and that hence need critical, sceptical testing. In the case of climate change, we presently rely greatly upon the Intergovernmental Panel on Climate Change (IPCC) to be the prosecution, defence, jury and judge. There is no independent defence. This is akin to a court case in China, or, it seems, in Russia where the government controls all aspects of the judicial system. If the system is fundamentally deficient, one cannot have faith in the verdict.

Where is the organization that should be defending the potentially ‘criminal’ molecule CO₂ against the allegations made against it? Such an organization does not exist. Mostly a few maverick scientists together with a large number of knowledgeable bloggers have done the surprisingly successful defence of CO₂ .

But surely, one may ask, what about all the famous debates in science? Don’t all scientists spend their lives arguing and defending their ideas? Aren’t they mostly sceptical about all scientific theories and discoveries? The answer is yes - up to a point. To understand where it goes wrong, one needs to understand the way that scientific results are published and how funding is given.

After a scientist does some new and original work, he/she submits to a journal a paper describing the work and its conclusions. The journal is usually owned by one of the big publishing houses or by one of the learned societies such as the Royal Society, American Geophysical Union etc. The editor of the journal has a quick look at the paper and selects reviewers who are considered to be knowledgeable scientists in the relevant field. This is peer review. The reviewers then make comments on the paper and, most importantly, advise the editor if the paper should be rejected or accepted for publication. If the paper is accepted, it is duly published and other scientists will read it. The work is now ‘peer reviewed’, the gold standard in science. If after publication a different scientist thinks there is a problem with a paper, he/she may write a comment to the editor who may decide to publish the comment together with reply by the original author, but there is no guarantee that either of these will happen.

The problem with this system is that a robust argument about the paper is not guaranteed. Argument is possible, perhaps even likely, but certainly not guaranteed. Why not?

The journal editor may not be neutral on the issue and may not send the paper to reviewers who will be sufficiently critical and objective. Journal editors do not get into these jobs by being system knockers. In addition to being good scientists, they are usually good at networking and operating within the system.

Reviewers are also not necessarily neutral and may not be truly critical in the proper sense of that word. They may not act like counsel in a court case.

Reviewers can block the publication of work antagonistic to their own views and the conventional wisdom. By blocking publication the reviewer prevents argument starting once the paper is published.

It is worth remembering that scientists are only human. It is hard for a reviewer to be objective about a paper that attacks some line that is dear to the heart (and mind) of the reviewer.

Once the paper is published, another scientist who thinks there is a problem with the paper may decide not to try to publish an argumentative comment for a number of reasons including the following.

They don’t want to cause a fuss and antagonise a potentially powerful group of scientists who could affect their funding and future job prospects. It’s also not much fun deliberately knocking, and the scientist will get no thanks or extra pay for doing it. For an easy life, it is best to say nothing.

Laziness. It is hard work getting a comment published and scientists are busy doing other things.

The scientist may personally know the original author and would feel that it could destroy a friendship to publicly show that a mistake has been made.

The scientist may also decide not to try to publish a critical comment because past experience has shown that getting comments published is very difficult because in the end, the editor may have to admit that he or she has made a mistake.

Similar but probably worse problems exist in the system of peer review of funding applications. If a scientist is asking for funds to work on something that is opposing the conventional wisdom, they will likely find that their reviewers will rate their application poorly and not be funded. Funding from bodies such as the Australian Research Council is very difficult to obtain with typical success rates being around 20%. On the other hand, for the publication of a paper there is little cost and so journals may publish work that has had less than wholly supportive reviews. For funding applications, funds are strictly rationed and even the hint of controversy from a reviewer can sink the application.

I often wonder what would have happened if Albert Einstein had, in 1905, applied for funding from an equivalent of the Australian Research Council to work on his theory of relativity where the expected outcomes (to use the jargon of these application processes) were to hopefully prove that mass and energy was equivalent (E = mc²), light could be bent by gravity, the size of the galaxy could be shrunk to a few metres for a fast travelling particle, and that twins could end up being different ages. I think he would have had problems. Fortunately theoretical physics does not have so many political or ideological ramifications as issues such as climate change and theoretical physicists can and often do their work with little more than pen, paper, and some time to think. However most of the rest of science needs funding, and funding bodies, because of peer review, are likely to entrench the conventional wisdom.

The flaws in the scientific system mean that once an idea takes hold in the scientific community, the self-reinforcing mechanisms in peer review of publication and funding, and the self-interests of organizations relying upon funding, mean that debate is likely to be stifled. The conventional wisdom can control the system and make it difficult for opposing ideas to be heard or funded.

This system of self-reinforcement is sometimes called a positive feedback mechanism. It is ironic that in climate science, hotly disputed positive feedback mechanisms are necessary to cause the nightmare scenarios of climate change where temperature will change by more than 2 degrees. Foremost amongst these mechanisms is the water vapour feedback where a temperature change caused by the ‘criminal’ CO₂ will evaporate more water, which is itself a strong greenhouse gas, thus increasing the temperature further and causing more evaporation and therefore even more warming etc. In the same way that the climate may spiral out of control, the scientific system, when applied to climate matters, may be considered to be spiralling out of control through powerful positive feedback mechanisms inherent in the system.

Some organizations such as the IPCC attempt to subject the science to the blowtorch of argument but it is done in a halfhearted manner. Contrarian scientists will rapidly find themselves on the outer in the IPCC. There are many other organizations involved in the positive feedback process for example the Australian Greenhouse Office (AGO). I would like to see how climate sceptics such as Professors Bob Carter or Ian Plimer would go if they applied for a job in that organization. The AGO doesn’t appreciate any criticism of conventional climate change theory – it is after all their lifeblood.

I have no problem with the IPCC prosecuting CO₂ . There is more than enough prime facie evidence to warrant a trial. But they mustn’t pretend also to act as the defence, then weigh the case for and against, and finally pass a verdict. We need a defence counsel that has major resources to fund scientists and take the time to formulate arguments. We cannot continue to rely upon a few mavericks, amateurs, bloggers and retired scientists.

We need an organization the role of which is not to directly try to determine the truth but just to make the best case against the conventional wisdom. I will call this organization the Scientific Challengers Office (SCO) for want of a better name. The organization should run its own journal and have significant resources to fund external scientists or to employ its own. It must be government funded and possibly a statutory authority to minimise political interference.

The SCO and its workers should be a valued part of the scientific system in the same way as in the legal system where it is a noble and honourable thing to defend a rapist, a nazi war criminal, or a drug lord. We need an organization where scientists can make a career out of being professional sceptics where they need not worry about what is right about a scientific theory – only what is wrong.

It should start on Global Warming, the supposed threats to the Great Barrier Reef and the debate about the Murray Darling. If the conventional wisdom on these issues is solid, it will emerge the stronger for the challenge. If the conventional wisdom is found to be shaky, we can concentrate on other pressing issues. Either way we are better off.

Peter Ridd is Professor of Physics at James Cook University and a scientific advisor to the Australian Environment Foundation (AEF). This article is an extract from a talk given by Ridd to the AEF 2010 annual conference.