Mindfulness research is often “spun” to appear positive, says article in Public Library of Science
Researchers at McGill University in Montreal, Canada, analyzed 124 published studies of mindfulness-based therapy (MBT), and found studies reported positive findings 60% more than is statistically likely. Only three of the studies reported negative outcomes and those were often “spun” to appear positive. Also, 62% of registered, completed MBT studies are not being published–suggesting that negative or non-findings are omitted from publication and mindfulness research is overly positive.
The study, Reporting of Positive Results in Randomized Controlled Trials of Mindfulness-Based Mental Health Interventions, was published 8 Apr 2016 in PLOS One, a peer-reviewed open access scientific journal published by the Public Library of Science (PLOS) since 2006.1
“A bias toward publishing studies that find the technique to be effective withholds important information from mental-health clinicians and patients,” says Christopher Ferguson, a psychologist at Stetson University in Florida, who was not involved in the study. “I think this is a very important finding,” he told Nature.
McGill psychiatry professor Dr. Brett Thombs, one of the researchers, told Health News Review there is a “massive push to support and propagate it [the mindfulness message],” and that “it isn’t dissimilar to [the drug industry] pushing cures that don’t work like they say they do.”
Thombs and team examined the evidence to support the health benefits being pushed by advocates of MBT. “While I’d agree that those selling wares carry out all sorts of shenanigans to promote their work,” he said, “could this be happening with mindfulness?” Let’s look.
Survey of Mindfulness Studies
The McGill researchers searched the medical and scientific databases for published papers on randomized controlled trials of MBT. Their initial search yielded 1,183 unique publications. After systematic review and excluding duplicates, they were left with 124 unique randomized controlled MBT trials: 4 (3%) published before 2000, 40 (32%) between 2000-2009, and 80 (65%) in 2010 or later.
“For 124 trials, the researchers calculated the probability that a trial with that sample size could detect the result reported.” Nature explained, “Experiments with smaller sample sizes are more affected by chance and thus worse at detecting statistically significant positive results. The scientists’ calculations suggested that 66 of 124 trials would have positive results. Instead, 108 trials had positive results. And none of the 21 registered trials adequately specified which of the variables they tracked would be the main one used to evaluate success.”
The team also examined another 21 trials that were registered with databases such as ClinicalTrials.gov; of these, 62% were unpublished 30 months after they finished. The findings suggest that non-findings or negative results are going unpublished.
Cherry-picking “positive” findings
Serious risks of bias are introduced when researchers fail to pre-register or don’t specify the variables to be used to evaluate treatment success. Pre-registration introduces transparency into the scientific process. Researchers and publishers are then less likely to selectively pick data to support “findings” and to underreport non-findings or negative outcomes.
“Reporting biases are said to occur when statistically significant or ‘positive’ outcomes have been preferentially published compared to non-significant or ‘negative’ outcomes.” wrote McGill team.
Biases skew mindfulness research
A concern expressed in McGill study was the overwhelmingly significant results in favor of MBT and mindfulness-based stress reduction. The authors say MBT reporting is influenced by:
- Publication bias: positive studies tend to be published, whereas negative not
- Selective outcome bias: studies are selected for publishing that show positive outcomes
- Selective analysis bias: data is analyzed using numerous methods and only positive results are reported
- Other biases: non-significant outcomes are glossed-over and made to appear positive
Thombs and team point out data dredging may be playing an important role in biased MBT research. Data dredging2 (also called p-hacking, data fishing, data snooping, and equation fitting) mines data to uncover patterns in the data and then presents findings as statistically significant. Dredging of data may be avoided by scientists who first devise a specific hypothesis as to the underlying causality of treatment, and by pre-registering trials before collecting clinical data.
Widespread bias and publication omissions occur, not only in MBT studies, warns LiveScience.com in Dark Side of Medical Research. Medical journals and researchers have a strong incentive to report only “positive” results, leaving out non-findings or negative findings when a therapy or procedure may have proved more harmful than helpful.
What are we to conclude?
Alan Cassels, a health policy researcher with University of Victoria and contributor to Health News Review writes, “Researchers in mindfulness, like almost anywhere, are capable of cherry-picking studies, some of which may make outrageous claims…”.
Mindfulness and meditation techniques may not be as miraculous as many want us believe. Certainly there always will be charlatans or delusional teachers who stand to profit from selling sham treatments and false promises.
Thombs admits: “I don’t believe that mindfulness training is completely ineffective or is harmful. I do believe–and I am supported by the evidence that we are publishing–that we don’t have a very good idea of how effective it is.”
Mindfulness studies are low quality and overly positive. Important health decisions require good research not pseudoscientific hype. “For the health-care system,” says Thombs, “it’s just as important to know what doesn’t work.”
Science has yet to prove the medical benefits of meditation are better than a placebo, a pretend treatment that one believes makes them feel better.
“If your particular form of meditation makes you feel good, do it!,” says John Horgan, writer at Scientific American.3 “But don’t kid yourself that its medical benefits have been scientifically proven.”
Many questions surround the practice of meditation and mindfulness. Seldom discussed are adverse side effects, including suggestibility, anxiety, psychosis and suicide. There is a cultish aspect to meditation: followers may be more vulnerable to psychological manipulation by charlatans and delusional teachers. (Watch Kumare, a disturbing and captivating documentary film of a fake Indian guru who built a following of real American devotees. We Westerners are so gullible for yogis and gurus.)
Before embracing or dismissing meditation, recognize the biases of its pushers and detractors. Thoroughly, intelligently, and methodically investigate claims.
Decide for yourself, read my posts on:
1 PLOS ONE, Wikipedia, https://en.wikipedia.org/wiki/PLOS_ONE
2 Data Dredging, Wikipedia, https://en.wikipedia.org/wiki/Data_dredging
3 Research on TM and Other Forms of Meditation Stinks, John Horgan, Scientific American, 8 Mar 2013, http://blogs.scientificamerican.com/cross-check/research-has-not-shown-that-meditation-beats-a-placebo/
Reporting of Positive Results in Randomized Controlled Trials of Mindfulness-Based Mental Health Interventions, Stephanie Coronado-Montoya, Alexander W. Levis, Linda Kwakkenbos, Russell J. Steele, Erick H. Turner, Brett D. Thombs, PLOS One, 8 Apr 2016, http://dx.doi.org/10.1371/journal.pone.0153220
Power of positive thinking skews mindfulness studies: Trials of mindfulness to improve mental health selectively report positive results, Anna Nowogrodzki, Nature, 21 Apr 2016. Accessed 27 Apr 2016 at
The marketing of mindfulness and why that matters, Health News Review, Alan Cassels, 12 Apr 2016. Accessed 27 Apr 2016 at http://www.healthnewsreview.org/2016/04/the-marketing-of-mindfulness-and-why-that-matters/