Are Most Research Findings False? Is That A Bad Thing?

I have some thoughts on this essay that Melinda posted in Comments:

Why Most Published Research Findings Are False, Ioannidis J, PLOS Medicine, August 2005.

I didn't understand what Ioannidis meant by "false," and to what it was being applied. What findings are false? Data results? Interpretation of data results? Hypotheses generated from interpretation of data results? And what is meant by "false?" I feel uncomfortable characterizing things as "true" and "false." Things just aren't that absolute.

After scanning the paper, I still don't see how he defined "findings," although by "false" I'm going to say he was referring to "findings" that are unsupported.

All studies are flawed. All studies are biased. Every single study that exists can be justifiably criticized. This is why science and research exist in the first place, to make sense of a seemingly nonsensical universe.

The risk in emphasizing the idea that studies are flawed, biased, or that findings are false, is that we, consumers of research, might not give study findings due consideration. We might not apply our minds in judging the adequacy of the research design (e.g. does what they measured answer the question they asked?) or the validity of the authors' data interpretation. We might not task ourselves with investigating author affiliations and research sponsorship.

There's always something to learn from research, no matter how flawed it is. Many of the studies that reported an association between cigarette smoking and lung cancer were criticized for being flawed (notably by tobacco companies), yet the body of evidence today continues to support the notion that smoking increases the risk for lung cancer.

After I wrote the above, I saw this reply to Ioannidis' essay by the editors of PLOS Medicine:

Minimizing Mistakes and Embracing Uncertainty, PLOS Medicine, August 2005

It said much the same. About "findings:"

"Ioannidis doesn't define “findings,” but it seems important to attempt to separate data (“in this study 5% of people examined who lived in San Francisco from 1965–1970 developed lung cancer compared with 20% of people studied who lived in Anchorage”) from conclusions (“lung cancer rates are higher in Anchorage than San Francisco”) and hypotheses (“cold weather exacerbates the consequences of smoking”)."

And about continuing to give research due consideration against a backdrop of not-knowing:

"Consumers also need to become comfortable with uncertainty, and understand the strengths and weaknesses intrinsic to every study conducted and published."

Finally, the idea that most findings in research are false, or flawed, or wrong, and that we could achieve much by embracing uncertainty was iterated in this video by physicist Brian Greene, where he said:

"The most amazing thing about the scientific journey is that we spend most of our life in the dark. We spend most of our life trying to figure things out and most people don't recognize that 99.9% of everything we do is flat out wrong. It's not wrong because we make mistakes, it's wrong because the universe is such a rich source of mystery that our attempts to understand it are usually off the mark. ... It's all about being comfortable searching in the dark."

A friend of mine says, "You can't be criticized for being wrong if you never do anything." So, let's do things! Flawed as they may be.*

* For instance, one good outcome from this Stanford study is that lots of people are talking about it, and lots of people are probably learning the reasons why food produced with fewer chemicals may be more healthful and better for the environment.
________