How to fix the broken scientific system

The legal and scientific processes of western democracies are based upon an adversarial system. In a court of law a prosecutor is appointed to make a case against the accused and a defence counsel is appointed to defend. Even in cases where it is completely obvious that the accused is guilty, they are still entitled to a defence, if for no other reason than to force the prosecution to prove their case. The strength of the prosecution argument is increased by its ability to stand up to the test posed by the defence. Hence there is far more public trust in the Guilty verdict. In science, scientists fight out intellectual battles in journals so that only the most battle-tested ideas emerge - or so we would hope.

The process of argument is as essential to the scientific system as it is to the legal system. A big difference is that argument is guaranteed in the legal system with the two sides of the argument formally recognised in the legal system itself, but because of the structures of the present systems in science, a robust argument cannot be guaranteed. Because of this there cannot be a sufficiently high level of faith that some of the big scientific issues of our time such as Anthropogenic Global Warming, the fate of the Murray-Darling, or the imminent demise of the Great Barrier Reef, have been properly tested in the scientific equivalent of a court of law.

The job of a defence counsel is to defend, not directly to try to find the truth. They may concede obvious points of the prosecution case, but they will mainly try to tear apart the arguments of the prosecution. The consequence is that the chance of indirectly ‘getting at’ the truth is increased. We need a similar system in science - an organization the sole charter of which is to find out what may be wrong or debatable with current scientific propositions, and that hence need critical, sceptical testing. In the case of climate change, we presently rely greatly upon the Intergovernmental Panel on Climate Change (IPCC) to be the prosecution, defence, jury and judge. There is no independent defence. This is akin to a court case in China, or, it seems, in Russia where the government controls all aspects of the judicial system. If the system is fundamentally deficient, one cannot have faith in the verdict.

Where is the organization that should be defending the potentially ‘criminal’ molecule CO₂ against the allegations made against it? Such an organization does not exist. Mostly a few maverick scientists together with a large number of knowledgeable bloggers have done the surprisingly successful defence of CO₂ .

But surely, one may ask, what about all the famous debates in science? Don’t all scientists spend their lives arguing and defending their ideas? Aren’t they mostly sceptical about all scientific theories and discoveries? The answer is yes - up to a point. To understand where it goes wrong, one needs to understand the way that scientific results are published and how funding is given.

After a scientist does some new and original work, he/she submits to a journal a paper describing the work and its conclusions. The journal is usually owned by one of the big publishing houses or by one of the learned societies such as the Royal Society, American Geophysical Union etc. The editor of the journal has a quick look at the paper and selects reviewers who are considered to be knowledgeable scientists in the relevant field. This is peer review. The reviewers then make comments on the paper and, most importantly, advise the editor if the paper should be rejected or accepted for publication. If the paper is accepted, it is duly published and other scientists will read it. The work is now ‘peer reviewed’, the gold standard in science. If after publication a different scientist thinks there is a problem with a paper, he/she may write a comment to the editor who may decide to publish the comment together with reply by the original author, but there is no guarantee that either of these will happen.

The problem with this system is that a robust argument about the paper is not guaranteed. Argument is possible, perhaps even likely, but certainly not guaranteed. Why not?

The journal editor may not be neutral on the issue and may not send the paper to reviewers who will be sufficiently critical and objective. Journal editors do not get into these jobs by being system knockers. In addition to being good scientists, they are usually good at networking and operating within the system.

Reviewers are also not necessarily neutral and may not be truly critical in the proper sense of that word. They may not act like counsel in a court case.

Reviewers can block the publication of work antagonistic to their own views and the conventional wisdom. By blocking publication the reviewer prevents argument starting once the paper is published.

It is worth remembering that scientists are only human. It is hard for a reviewer to be objective about a paper that attacks some line that is dear to the heart (and mind) of the reviewer.

Once the paper is published, another scientist who thinks there is a problem with the paper may decide not to try to publish an argumentative comment for a number of reasons including the following.