Jump to content

Is Most Published Research Wrong? (p-hacking, etc.)


Recommended Posts

Really good video Khurram. Thanks for posting it.


There is a big problem across all branches of science (but especially health and medicine) with researchers deluding themselves, colleagues and the public (often unintentionally) via techniques like 'P-hacking' - post hoc analysis of experimental data in many different ways until you can demonstrate a result that is statistically significant, e.g. P < 0.05.


In fact, we've got a thread devoted to exposing particularly egregious examples of P-hacking and other research shenanigans.


I'm a big fan of the method of "pre-registration" mentioned at the end. The Center for Open Science is sponsoring a contest they call the Preregistration Challenge. It is open to scientists who register the precise experimental design and statistical analyses they plan to do ahead of actually collecting any data - to discourage P-hacking. Researchers who submit their experimental design, follow it to the letter, and then publish their results (significant or not) in any one of a long list of eligible journals, receive an award of $1000. 


Of course, it remains to be seen whether the $1000 reward will outweigh the big the incentive researchers have to P-hack (i.e. career advancement, more likely for significant results to get published regardless of how contrived they were, etc.). It would really help if journals were more willing to publish replication studies (positive or negative) as well as null results. Perhaps as important, it would help if tenure/promotion committees were more willing to reward solid science that replicates (or refutes) earlier results, not just reward researchers when they appear to break new ground, sometimes using dubious experimental designs and analysis techniques to make it look that way.


Interestingly, there is some empirical evidence the pre-registration does have an impact. This study [1], found that the likelihood of null effects being published as a result big, government-sponsored studies has been increasing over time, and this was particularly true of studies that were pre-registered at clinicaltrials.gov.


So that's a good sign, but as far as I know, the results of [1] have yet to be replicated...





[1] PLoS One. 2015 Aug 5;10(8):e0132382. doi: 10.1371/journal.pone.0132382.

eCollection 2015.
Likelihood of Null Effects of Large NHLBI Clinical Trials Has Increased over
Kaplan RM(1), Irvin VL(2).
Author information: 
(1)Agency for Healthcare Research and Quality, U.S. Department of Health and
Human Services, Rockville, Maryland, United States of America. (2)Oregon State
University, Corvallis, Oregon, United States of America.
Comment in
    BMJ. 2015;351:h4304.
BACKGROUND: We explore whether the number of null results in large National Heart
Lung, and Blood Institute (NHLBI) funded trials has increased over time.
METHODS: We identified all large NHLBI supported RCTs between 1970 and 2012
evaluating drugs or dietary supplements for the treatment or prevention of
cardiovascular disease. Trials were included if direct costs >$500,000/year,
participants were adult humans, and the primary outcome was cardiovascular risk, 
disease or death. The 55 trials meeting these criteria were coded for whether
they were published prior to or after the year 2000, whether they registered in
clinicaltrials.gov prior to publication, used active or placebo comparator, and
whether or not the trial had industry co-sponsorship. We tabulated whether the
study reported a positive, negative, or null result on the primary outcome
variable and for total mortality.
RESULTS: 17 of 30 studies (57%) published prior to 2000 showed a significant
benefit of intervention on the primary outcome in comparison to only 2 among the 
25 (8%) trials published after 2000 (χ2=12.2,df= 1, p=0.0005). There has been no 
change in the proportion of trials that compared treatment to placebo versus
active comparator. Industry co-sponsorship was unrelated to the probability of
reporting a significant benefit. Pre-registration in clinical trials.gov was
strongly associated with the trend toward null findings.
CONCLUSIONS: The number NHLBI trials reporting positive results declined after
the year 2000. Prospective declaration of outcomes in RCTs, and the adoption of
transparent reporting standards, as required by clinicaltrials.gov, may have
contributed to the trend toward null findings.
DOI: 10.1371/journal.pone.0132382 
PMCID: PMC4526697
PMID: 26244868
Link to comment
Share on other sites

Dean: thx for the other thread link!


The word "WRONG" from the video/thread title "Is Most Published Research Wrong?" reminded me of another similar synthesis from a book I read in 2010.


WRONG:Why experts* keep failing us--and how to know when not to trust them  (by David Freedman)


*Scientists, finance wizards, doctors, relationship gurus, celebrity CEOs, ... consultants, health officials and more.


The fact that "gurus" are mostly wrong is not surprising. What is a bit concerning is the claim that "90% of medical studies" are ultimately proved wrong. 

Or "...two-thirds of the findings published in the top medical journals are refuted within a few years."

This quick video sums some key point of Freedman's book:


Link to comment
Share on other sites


This topic is now archived and is closed to further replies.

  • Create New...