Science death spiral

Having brought mankind so far, has traditional science finally outlived its usefulness? Many seem to think so, finding the rules of the scientific method—the strict guidelines a researcher must follow to actually practice science—far too restrictive and cumbersome. The requirement that evidence be empirical, which is to say, actual measurements of nature itself, is found too burdensome to new age scientists. They prefer clean, clinical computer models to messy, often uncooperative nature. Over reliance on models, misapplication of statistical methods, and lack of repeatability are the hallmarks of the new pseudoscience that is replacing the traditional practice of science, real science. As one critic recently wrote: “The problem with ­science is that so much of it simply isn’t.” Has science entered a death spiral, as indifferent, inept scientists raise up new generations of even poorer researchers? The facts look grim.

There has been a flood of recent articles addressing the failure of modern science in both scientific journals and online. The respected journal Nature posted a new article entitled “Online debate erupts to ask: is science broken?,” addressing just this subject. In it, results from a panel discussion, held at the University College London, were reported. These included claims that some dubious laboratory practices, such as tweaking statistical analyses to make results seem significant, are widespread.

From psychology and medicine to the physical sciences, alarms are being raised that science in its evolving modern form is failing. It is as if the rules, which have served science so well for centuries, have become optional. Requirements for judging good science from bad have become flexible, expanding to embrace sloppy research and even outright fakery. I have written about the scientific method and the philosophy of science before (see “Cherry Picking, Black Swans and Falsifiability”). I have stressed how theories must be testable, meaning they must make predictions that can be either confirmed or refuted by actual measurement. Beyond that, these measurements must be repeatable by others. In other words, an experiment can not just work for the originator of a theory. That is a good place to start investigating modern science's failures.

Writing on the First Things blog, William A. Wilson has penned a scathing critique of published research in the field of psychology. In “Scientific Regress,” Wilson reveals modern psychological research to be a house of cards, filled with unreproducible results:

The problem with ­science is that so much of it simply isn’t. Last summer, the Open Science Collaboration announced that it had tried to replicate one hundred published psychology experiments sampled from three of the most prestigious journals in the field. Scientific claims rest on the idea that experiments repeated under nearly identical conditions ought to yield approximately the same results, but until very recently, very few had bothered to check in a systematic way whether this was actually the case. The OSC was the biggest attempt yet to check a field’s results, and the most shocking. In many cases, they had used original experimental materials, and sometimes even performed the experiments under the guidance of the original researchers. Of the studies that had originally reported positive results, an astonishing 65 percent failed to show statistical significance on replication, and many of the remainder showed greatly reduced effect sizes.

According to Wilson the problem is not limited to psychology. An investigation into the pharmaceutical industry at Bayer, focused on cancer biology research, researchers found that more than 75% of cases the published data did not match up with in-house attempts to replicate. As Wilson points out, such conformation failure can be because the reproduction was in someway flawed or the original finding was false. Moreover, the failure of an experiment may well be because of failure on the part of the researchers to understand the subtleties of Bayesian statistics. “A tremendous amount depends on the proportion of possible hypotheses which turn out to be true, and on the accuracy with which an experiment can discern truth from falsehood,” he writes.

John Ioannidis, a professor at Stanford University’s School of Medicine, recently published a paper with the shocking title “Why Most Published Research Findings Are False.” In it, he details why the validity of an experiment depends on the proportion of possible hypotheses which turn out to be true, and on the accuracy with which an experiment can discern truth from falsehood. Ioannidis shows that for a wide variety of scientific settings and fields, the values of these two parameters are not at all favorable. In many cases, approaching even 50% true positives requires unimaginable accuracy. Ioannidis's 2005 paper has become the most downloaded technical paper from the journal PLoS Medicine.

But it is not just the medical and social sciences that are at risk. Physics, that most concrete of the physical sciences, is supposedly based on measurable experiments: dropping objects from a tower, the bending of light by an intervening galaxy, Millikan and his oil drops, etc. Surely physics lies at the other end of the scientific spectrum from sciences like psychology and sociology where interpretation plays a much larger part. According to Wilson that would be a false assumption.

[M]easured any way you like—volume of papers, number of working researchers, total amount of funding—deductive, theory-building physics in the mold of Newton and Lagrange, Maxwell and Einstein, is a tiny fraction of modern science as a whole. In fact, it also makes up a tiny fraction of modern physics. Far more common is the delicate and subtle art of scouring inconceivably vast volumes of noise with advanced software and mathematical tools in search of the faintest signal of some hypothesized but never before observed phenomenon, whether an astrophysical event or the decay of a subatomic particle. This sort of work is difficult and beautiful in its own way, but it is not at all self-evident in the manner of a falling apple or an elliptical planetary orbit, and it is very sensitive to the same sorts of accidental contamination, deliberate fraud, and unconscious bias as the medical and social-scientific studies we have discussed. Two of the most vaunted physics results of the past few years—the announced discovery of both cosmic inflation and gravitational waves at the BICEP2 experiment in Antarctica, and the supposed discovery of superluminal neutrinos at the Swiss-Italian border—have now been retracted, with far less fanfare than when they were first published.

So even the hard sciences are susceptible to irreproducibility. In “Why Science Is Broken, and How To Fix It,” scientist William M Briggs weighed in on the problem. “There’s been a spate of lamentations that science is broken,” he writes. “I am a credentialed, working scientist, and I’m here to tell you that, with some exceptions, these cris de coeur are right. Science is a mess.”

Decrying “nitwit, avaricious, power-hungry politicians” who promote the “science is settled!” mantra, Briggs points the finger at bad politics, bad money, and bad philosophy. He complains that too many indifferent students have swollen the ranks of researchers and too much (not too little!) money is driving mediocre scientists to do mediocre research which gets published in content hungry journals. Climate science gets particular mention.

First problem is over-certainty. Science works where it is forced to confront reality. This is why meteorological forecasts have improved so greatly, and why climatological forecasts haven’t. Weathermen have to perform. Climatologists have to please politicians and funding agencies. Which group is cockier?

That climatology, and some other fields, succeed when they shouldn’t is a philosophical problem. It used to be understood that scientific theories should match reality. But that is a philosophical and not a scientific proposition. Indeed, all the principles that define science aren’t themselves scientific.