Wednesday, September 12, 2012

Are Most Research Findings False? Is That A Bad Thing?

I have some thoughts on this essay that Melinda posted in Comments:

Why Most Published Research Findings Are False, Ioannidis J, PLOS Medicine, August 2005.

I didn't understand what Ioannidis meant by "false," and to what it was being applied. What findings are false? Data results? Interpretation of data results? Hypotheses generated from interpretation of data results? And what is meant by "false?" I feel uncomfortable characterizing things as "true" and "false." Things just aren't that absolute.

After scanning the paper, I still don't see how he defined "findings," although by "false" I'm going to say he was referring to "findings" that are unsupported.

All studies are flawed. All studies are biased. Every single study that exists can be justifiably criticized. This is why science and research exist in the first place, to make sense of a seemingly nonsensical universe.

The risk in emphasizing the idea that studies are flawed, biased, or that findings are false, is that we, consumers of research, might not give study findings due consideration. We might not apply our minds in judging the adequacy of the research design (e.g. does what they measured answer the question they asked?) or the validity of the authors' data interpretation. We might not task ourselves with investigating author affiliations and research sponsorship.

There's always something to learn from research, no matter how flawed it is. Many of the studies that reported an association between cigarette smoking and lung cancer were criticized for being flawed (notably by tobacco companies), yet the body of evidence today continues to support the notion that smoking increases the risk for lung cancer.

After I wrote the above, I saw this reply to Ioannidis' essay by the editors of PLOS Medicine:

Minimizing Mistakes and Embracing Uncertainty, PLOS Medicine, August 2005

It said much the same. About "findings:"
"Ioannidis doesn't define “findings,” but it seems important to attempt to separate data (“in this study 5% of people examined who lived in San Francisco from 1965–1970 developed lung cancer compared with 20% of people studied who lived in Anchorage”) from conclusions (“lung cancer rates are higher in Anchorage than San Francisco”) and hypotheses (“cold weather exacerbates the consequences of smoking”)."
And about continuing to give research due consideration against a backdrop of not-knowing:
"Consumers also need to become comfortable with uncertainty, and understand the strengths and weaknesses intrinsic to every study conducted and published."
Finally, the idea that most findings in research are false, or flawed, or wrong, and that we could achieve much by embracing uncertainty was iterated in this video by physicist Brian Greene, where he said:
"The most amazing thing about the scientific journey is that we spend most of our life in the dark. We spend most of our life trying to figure things out and most people don't recognize that 99.9% of everything we do is flat out wrong. It's not wrong because we make mistakes, it's wrong because the universe is such a rich source of mystery that our attempts to understand it are usually off the mark. ... It's all about being comfortable searching in the dark."
A friend of mine says, "You can't be criticized for being wrong if you never do anything." So, let's do things! Flawed as they may be.*

* For instance, one good outcome from this Stanford study is that lots of people are talking about it, and lots of people are probably learning the reasons why food produced with fewer chemicals may be more healthful and better for the environment.


Anonymous said...

The use of epidemiology is whats so FALSE,its a statistical wonderland that can be manipulated to any extreme one with like it used for. Why is epidemiology even used,its quite simple,when you cant prove anything you use statistical magic and create your own evidence and especially in the case of LC.

To this day not one claim against tobacco has ever been proven:

Of course this comment wont be posted but at least you who moderate it will see it and go OMG!

It destroys everything anti-tobacco stands for and this cant be posted!

7 October, the COT meeting on 26 October and the COC meeting on 18
November 2004.

"5. The Committees commented that tobacco smoke was a highly complex chemical mixture and that the causative agents for smoke induced diseases (such as cardiovascular disease, cancer, effects on reproduction and on offspring) was unknown. The mechanisms by which tobacco induced adverse effects were not established. The best information related to tobacco smoke - induced lung cancer, but even in this instance a detailed mechanism was not available. The Committees therefore agreed that on the basis of current knowledge it would be very difficult to identify a toxicological testing strategy or a biomonitoring approach for use in volunteer studies with smokers where the end-points determined or biomarkers measured were predictive of the overall burden of tobacco-induced adverse disease."

In other words ... our first hand smoke theory is so lame we can't even design a bogus lab experiment to prove it. In fact ... we don't even know how tobacco does all of the magical things we claim it does.

The greatest threat to the second hand theory is the weakness of the first hand theory.

Ronald said...

I agree. There are also no studies that PROVE that we are not living in the Matrix or are part of a Holodeck program. End program.

Bix said...

I have to agree with Ronald here, science doesn't prove things, it disproves them.

Regarding The Matrix: I mean, even the theoretical physicist Brian Greene in the video I just posted said there was a possibility we are elements of a computer program. Nothing has proven we aren't.

Bix said...

From what I can tell, this (JOINT STATEMENT ON ...) is a paper assessing "toxic exposure and harm" from tobacco-based Potentially Reduced Exposure Products (PREPS). I guess these PREPS are the new "safer" cigarettes?

It started out by saying...

The Committees agreed that it was important to state that the ideal way forward to reduce risks and hazards of tobacco smoke was to encourage smokers to stop or people not to start in the first place.

Bix said...

Epidemiology is useful in that it allows us to generate hypotheses.