KHashmi317 Posted August 27, 2016 Report Share Posted August 27, 2016 This was uploaded on the popsci YouTube vlog Veritasium about two weeks ago: It's simpler that you watch/absorb the short vid, so I won't comment further in this post. Link to comment Share on other sites More sharing options...
Dean Pomerleau Posted August 27, 2016 Report Share Posted August 27, 2016 Really good video Khurram. Thanks for posting it. There is a big problem across all branches of science (but especially health and medicine) with researchers deluding themselves, colleagues and the public (often unintentionally) via techniques like 'P-hacking' - post hoc analysis of experimental data in many different ways until you can demonstrate a result that is statistically significant, e.g. P < 0.05. In fact, we've got a thread devoted to exposing particularly egregious examples of P-hacking and other research shenanigans. I'm a big fan of the method of "pre-registration" mentioned at the end. The Center for Open Science is sponsoring a contest they call the Preregistration Challenge. It is open to scientists who register the precise experimental design and statistical analyses they plan to do ahead of actually collecting any data - to discourage P-hacking. Researchers who submit their experimental design, follow it to the letter, and then publish their results (significant or not) in any one of a long list of eligible journals, receive an award of $1000. Of course, it remains to be seen whether the $1000 reward will outweigh the big the incentive researchers have to P-hack (i.e. career advancement, more likely for significant results to get published regardless of how contrived they were, etc.). It would really help if journals were more willing to publish replication studies (positive or negative) as well as null results. Perhaps as important, it would help if tenure/promotion committees were more willing to reward solid science that replicates (or refutes) earlier results, not just reward researchers when they appear to break new ground, sometimes using dubious experimental designs and analysis techniques to make it look that way. Interestingly, there is some empirical evidence the pre-registration does have an impact. This study [1], found that the likelihood of null effects being published as a result big, government-sponsored studies has been increasing over time, and this was particularly true of studies that were pre-registered at clinicaltrials.gov. So that's a good sign, but as far as I know, the results of [1] have yet to be replicated... --Dean ---------- [1] PLoS One. 2015 Aug 5;10(8):e0132382. doi: 10.1371/journal.pone.0132382. eCollection 2015. Likelihood of Null Effects of Large NHLBI Clinical Trials Has Increased over Time. Kaplan RM(1), Irvin VL(2). Author information: (1)Agency for Healthcare Research and Quality, U.S. Department of Health and Human Services, Rockville, Maryland, United States of America. (2)Oregon State University, Corvallis, Oregon, United States of America. Comment in BMJ. 2015;351:h4304. BACKGROUND: We explore whether the number of null results in large National Heart Lung, and Blood Institute (NHLBI) funded trials has increased over time. METHODS: We identified all large NHLBI supported RCTs between 1970 and 2012 evaluating drugs or dietary supplements for the treatment or prevention of cardiovascular disease. Trials were included if direct costs >$500,000/year, participants were adult humans, and the primary outcome was cardiovascular risk, disease or death. The 55 trials meeting these criteria were coded for whether they were published prior to or after the year 2000, whether they registered in clinicaltrials.gov prior to publication, used active or placebo comparator, and whether or not the trial had industry co-sponsorship. We tabulated whether the study reported a positive, negative, or null result on the primary outcome variable and for total mortality. RESULTS: 17 of 30 studies (57%) published prior to 2000 showed a significant benefit of intervention on the primary outcome in comparison to only 2 among the 25 (8%) trials published after 2000 (χ2=12.2,df= 1, p=0.0005). There has been no change in the proportion of trials that compared treatment to placebo versus active comparator. Industry co-sponsorship was unrelated to the probability of reporting a significant benefit. Pre-registration in clinical trials.gov was strongly associated with the trend toward null findings. CONCLUSIONS: The number NHLBI trials reporting positive results declined after the year 2000. Prospective declaration of outcomes in RCTs, and the adoption of transparent reporting standards, as required by clinicaltrials.gov, may have contributed to the trend toward null findings. DOI: 10.1371/journal.pone.0132382 PMCID: PMC4526697 PMID: 26244868 Link to comment Share on other sites More sharing options...
KHashmi317 Posted August 27, 2016 Author Report Share Posted August 27, 2016 Dean: thx for the other thread link! The word "WRONG" from the video/thread title "Is Most Published Research Wrong?" reminded me of another similar synthesis from a book I read in 2010. WRONG:Why experts* keep failing us--and how to know when not to trust them (by David Freedman) *Scientists, finance wizards, doctors, relationship gurus, celebrity CEOs, ... consultants, health officials and more. The fact that "gurus" are mostly wrong is not surprising. What is a bit concerning is the claim that "90% of medical studies" are ultimately proved wrong. Or "...two-thirds of the findings published in the top medical journals are refuted within a few years." This quick video sums some key point of Freedman's book: Link to comment Share on other sites More sharing options...
Sthira Posted August 27, 2016 Report Share Posted August 27, 2016 https://www.youtube.com/watch?v=FJWKNr8mqO4 This was great, thank you. And I like the nutrition advice: "eat fruits and vegetables for 70 years" end of nutrition science story. Link to comment Share on other sites More sharing options...
Recommended Posts
Archived
This topic is now archived and is closed to further replies.