Mindfulness research is often “spun” to appear positive, says article in Public Library of Science
Researchers at McGill University in Montreal, Canada, analyzed 124 published studies of mindfulness-based therapy (MBT), and found studies reported positive findings 60% more than is statistically likely. Only three of the studies reported negative outcomes and those were often “spun” to appear positive. Also, 62% of registered, completed MBT studies are not being published–suggesting that negative or non-findings are omitted from publication and mindfulness research is overly positive.
The study, Reporting of Positive Results in Randomized Controlled Trials of Mindfulness-Based Mental Health Interventions, was published 8 Apr 2016 in PLOS One, a peer-reviewed open access scientific journal published by the Public Library of Science (PLOS) since 2006.1
“A bias toward publishing studies that find the technique to be effective withholds important information from mental-health clinicians and patients,” says Christopher Ferguson, a psychologist at Stetson University in Florida, who was not involved in the study. “I think this is a very important finding,” he told Nature.
McGill psychiatry professor Dr. Brett Thombs, one of the researchers, told Health News Review there is a “massive push to support and propagate it [the mindfulness message],” and that “it isn’t dissimilar to [the drug industry] pushing cures that don’t work like they say they do.”
Thombs and team examined the evidence to support the health benefits being pushed by advocates of MBT. “While I’d agree that those selling wares carry out all sorts of shenanigans to promote their work,” he said, “could this be happening with mindfulness?” Let’s look.
Survey of Mindfulness Studies
The McGill researchers searched the medical and scientific databases for published papers on randomized controlled trials of MBT. Their initial search yielded 1,183 unique publications. After systematic review and excluding duplicates, they were left with 124 unique randomized controlled MBT trials: 4 (3%) published before 2000, 40 (32%) between 2000-2009, and 80 (65%) in 2010 or later.
“For 124 trials, the researchers calculated the probability that a trial with that sample size could detect the result reported.” Nature explained, “Experiments with smaller sample sizes are more affected by chance and thus worse at detecting statistically significant positive results. The scientists’ calculations suggested that 66 of 124 trials would have positive results. Instead, 108 trials had positive results. And none of the 21 registered trials adequately specified which of the variables they tracked would be the main one used to evaluate success.”
The team also examined another 21 trials that were registered with databases such as ClinicalTrials.gov; of these, 62% were unpublished 30 months after they finished. The findings suggest that non-findings or negative results are going unpublished.
Andreas-photography, Flickr, CC BY-NC 2.0
Cherry-picking “positive” findings
Serious risks of bias are introduced when researchers fail to pre-register or don’t specify the variables to be used to evaluate treatment success. Pre-registration introduces transparency into the scientific process. Researchers and publishers are then less likely to selectively pick data to support “findings” and to underreport non-findings or negative outcomes.
“Reporting biases are said to occur when statistically significant or ‘positive’ outcomes have been preferentially published compared to non-significant or ‘negative’ outcomes.” wrote McGill team.
Biases skew mindfulness research
A concern expressed in McGill study was the overwhelmingly significant results in favor of MBT and mindfulness-based stress reduction. The authors say MBT reporting is influenced by:
- Publication bias: positive studies tend to be published, whereas negative not
- Selective outcome bias: studies are selected for publishing that show positive outcomes
- Selective analysis bias: data is analyzed using numerous methods and only positive results are reported
- Other biases: non-significant outcomes are glossed-over and made to appear positive
Thombs and team point out data dredging may be playing an important role in biased MBT research. Data dredging2 (also called p-hacking, data fishing, data snooping, and equation fitting) mines data to uncover patterns in the data and then presents findings as statistically significant. Dredging of data may be avoided by scientists who first devise a specific hypothesis as to the underlying causality of treatment, and by pre-registering trials before collecting clinical data.
Widespread bias and publication omissions occur, not only in MBT studies, warns LiveScience.com in Dark Side of Medical Research. Medical journals and researchers have a strong incentive to report only “positive” results, leaving out non-findings or negative findings when a therapy or procedure may have proved more harmful than helpful.
Image: honor the gift , Flickr, CC BY-NC 2.0
What are we to conclude?
Alan Cassels, a health policy researcher with University of Victoria and contributor to Health News Review writes, “Researchers in mindfulness, like almost anywhere, are capable of cherry-picking studies, some of which may make outrageous claims…”.
Mindfulness and meditation techniques may not be as miraculous as many want us believe. Certainly there always will be charlatans or delusional teachers who stand to profit from selling sham treatments and false promises.
Thombs admits: “I don’t believe that mindfulness training is completely ineffective or is harmful. I do believe–and I am supported by the evidence that we are publishing–that we don’t have a very good idea of how effective it is.”
Mindfulness studies are low quality and overly positive. Important health decisions require good research not pseudoscientific hype. “For the health-care system,” says Thombs, “it’s just as important to know what doesn’t work.”
Science has yet to prove the medical benefits of meditation are better than a placebo, a pretend treatment that one believes makes them feel better.
“If your particular form of meditation makes you feel good, do it!,” says John Horgan, writer at Scientific American.3 “But don’t kid yourself that its medical benefits have been scientifically proven.”
Many questions surround the practice of meditation and mindfulness. Seldom discussed are adverse side effects, including suggestibility, anxiety, psychosis and suicide. There is a cultish aspect to meditation: followers may be more vulnerable to psychological manipulation by charlatans and delusional teachers. (Watch Kumare, a disturbing and captivating documentary film of a fake Indian guru who built a following of real American devotees. We Westerners are so gullible for yogis and gurus.)
Before embracing or dismissing meditation, recognize the biases of its pushers and detractors. Thoroughly, intelligently, and methodically investigate claims.
Decide for yourself, read my posts on:
Escaping the psychological trap of meditation techniques
Notes
1 PLOS ONE, Wikipedia, https://en.wikipedia.org/wiki/PLOS_ONE
2 Data Dredging, Wikipedia, https://en.wikipedia.org/wiki/Data_dredging
3 Research on TM and Other Forms of Meditation Stinks, John Horgan, Scientific American, 8 Mar 2013, http://blogs.scientificamerican.com/cross-check/research-has-not-shown-that-meditation-beats-a-placebo/
References
Reporting of Positive Results in Randomized Controlled Trials of Mindfulness-Based Mental Health Interventions, Stephanie Coronado-Montoya, Alexander W. Levis, Linda Kwakkenbos, Russell J. Steele, Erick H. Turner, Brett D. Thombs, PLOS One, 8 Apr 2016, http://dx.doi.org/10.1371/journal.pone.0153220
Power of positive thinking skews mindfulness studies: Trials of mindfulness to improve mental health selectively report positive results, Anna Nowogrodzki, Nature, 21 Apr 2016. Accessed 27 Apr 2016 at
http://www.nature.com/news/power-of-positive-thinking-skews-mindfulness-studies-1.19776
The marketing of mindfulness and why that matters, Health News Review, Alan Cassels, 12 Apr 2016. Accessed 27 Apr 2016 at http://www.healthnewsreview.org/2016/04/the-marketing-of-mindfulness-and-why-that-matters/
Interesting article, Scott. You touch on several difficult and important issues: accuracy, completeness, and veracity of scientific research and clinical techniques; distinguishing personal religious/spiritual practices from clinical techniques (i.e. a yoga class is not the same as Mindfulness Based Therapy, and it’s a mistake to think a study on MBT supports one’s yoga practices); and problems with new data-driven research methods: the large databases available to current researchers provide the possibility of sculpting data sets that justify their hypotheses. Nice work.
@My Other Feet: You’ve concisely summed up several important points I hope others will benefit from. Thanks for your feedback.
Scott,
I watched Kumare a few years back.It was quite illuminating.
Even with a group that I considered to be well grounded, and on the whole of it, quite on the reasonable side, I still found myself disenchanted with the group I joined due to an overarching skepticism of the very touted benefits of meditation.
I agree, Corey, the Kumare film is an illuminating and shocking documentary. Glad to hear you retained skepticism while in your group.
I wonder what group joined and what were the touted benefits. The foundational premises of these groups, gurus, and their followers is often frightfully similar. I wonder if you have read my post Duped by Meditation? for examples of these foundational premises.
Scott,
I would like to tell you about my own enlightenment. Yes it really happened. I went to a talk given by a direct disciple of Paramahansa Yogananda. He gave the group detailed instructions on how to meditate. Needless to say I was thrilled. I drove home that night with one thought on my mind. “I will experience enlightenment this very nIght!” So I immediately ran to my room, turned the lights down and sat in a lotus position on my bed. I closed my eyes and began to meditate. It was slow at first but I kept focusing and then it happened. I was literally “bathed” in an intense white light. I couldn’t believe it! Meditation works! I slowly opened my eyes to just get “feel” for what was happening to me. I realized that I had sat on the bed facing out toward the driveway and a car had driven up with its headlights on and pointing directly into my bedroom window.
So now you know someone who has really become enlightened. Or better said, head-lightened…
@Brent: I love your story! Your punch line made me laugh out loud, at the ironic truth.
Our minds and background beliefs fabricate our interpretations of our internal and external sensations. The guru implanted the idea that meditation could bath you in light…Even if meditation “headlights” were internal to you only, your mind and imagination probably fabricates what you “saw”. I share one of my experiences of the inner light in my post Natural Causes of the Spiritual Eye. My mind was conditioned, by my years of yogic indoctrination, to make my inner light out to be the spiritual eye–that’s what I saw and felt a great peace, bliss. Blood and oxygen deprived in my body and brain from cycling caused (I believe) this particular experience.
Hi Scott, Pretty funny right? True story.
I reread your Spiritual Eye post. Science can explain many of the experiences people report as part of the meditative process. In my opinion, there are two reasons why, in some cases, it can’t.
1) The experiences are hearsay. Are they really reported correctly? What kind of bias is brought into the report (religious, cultural, etc)
2) They are real phenomenon, but science does not yet have an answer for it. Because science hasn’t studied something yet and come up with answers for, does not mean it won’t.
So I don’t think there is a really good scientific answer for a lot of meditation experiences like the “spiritual star”, but I think that there will be. Right now the main place you read about it is in spiritual literature. But if you wrote a book about cycling and experiencing the “cycling star” people would begin looking for it on their tough climbs and some would actually report it. We’re humans and that’s what we do.
I think the most “damning” thing that your site has made me think about is the whole samadhi experience. If you take that out of the equation, who would even want to meditate? Who would even listen to a guru? If there is no pot of gold at the end of the rainbow, why chase it?
The need for samadhi is also associated with the need to escape reincarnation. That if you have the samadhi experience you don’t have to reincarnate. But there is no proof of reincarnation.
When I reflect on the samadhi/reincarnation association, it isn’t altogether different from the christian dogma of “you are going to hell, but if you accept Christ you don’t have to go to hell”. So “you are going to reincarnate unless you meditate”
Both state “your future is going to be horrible, unless you do what I say”
In the mean time, where is God in all of this? The Christians say “God came 2k years ago”. The Hindus say “God comes in the form of a Guru/Avatar” . If God really exists, why does He have to play these games? Why doesn’t He just show up “in person” and tell us what he wants? Both Christians and Hindus will have answers to these questions, but in the end the answer is “just trust me”. People have done dark and terrible things based on these 3 words.
Cheers, Brent
Scott,
Sorry, not trying to “litter up” your site ,but this relates exactly to the science of meditation. or should I say science of medication (dmt).
http://www.sbs.com.au/news/article/2013/11/08/dmt-psychedelic-drug-produced-your-brain
What is so fascinating is that the DMT drug produces experiences of other worlds and beings. And that it may be produced natually by our pineal gland (or “3rd eye”) Remember that Yogananda said that it wasn’t enough to “see” the spiritual eye, but one must merge with it or “go through it”. And if you “go through it” , according to Yogananda” you will be able to visit astral worlds and commune with other beings. What if the “merging” process Yogananda talked about causes the physical “release” of dmt from the pineal gland.
So it is possible that what he said about the “spiritual eye” experience was totally valid. He really experienced exactly what he said he did. But what he didn’t know is that there was a possible scientific reason for this. That the experience is real in every way except that it was a dmt drug trip. People who take the DMT drug claim that their experiences are “more real” than normal experiences. That makes perfect sense because these experiences “originate” in the brain, they do not first go through the 5 senses and then get “interpreted ” by the brain.
So if dmt is the samadhi experience what about other positive experiences like “bliss”? My bet is that it is endophines/nuetransmitter release. That’s why your assertion that people report the same “bliss high” from running, listening to music etc.
It is also apparently true that samadhi experiences are fairly rare. I think we can assume this. The release of dmt “should be” difficult for humans. It must be because of survival of the species alone. If a bear is running after you, you shouldn’t have a dmt induced spiritual experience in the midst of it. You will die. And if it did happen to a person, that person got deselected from the gene pool…by the bear.
So bliss feelings should be fairly easy to get. You ran from the bear, you are in pain from the exhaustion, but you are safe, now the release of endophins/neurotransmitters. Ahh I feel better!
Samadhi(dmt) experiences are harder to get, in fact it is rare no matter how much you meditate. But yes it can happen. But samadhi is a dmt based hallucination.
This dmt theory may also explain detailed near death experiences and why they are so life changing…they are so “real”.
So it’s not the “science of religion” it’s just the “science of …er… uh…it’s just science”.
Hey Brent: I’ll reply to your two comment posts, first post first and then your second.
I agree Science nor humans know everything there is to know. However, science may never know about lots of “experiences” or perceptions that are made up in the human imagination and which have no basis in actuality. Suppose I say I have an invisible dragon in my garage. I see the fire breathing dragon in my spiritual eye and feel its flaming presence in my garage. The fact that you can’t see or feel it doesn’t make it not true. It just means you and others have no good reason to believe it is true. If it doesn’t exist, how can one “discover” it?—supposedly it requires faith or surrender to the dragon master.
Yes, samadhi is another false promise, and extraordinary claim (like the invisible dragon in my garage) of which there is no good reason to believe that it is anything real than overactive imagination or religious interpretation of natural phenomenon.
Regarding your reference to DMT: You write, “It is also apparently true that samadhi experiences are fairly rare. I think we can assume this.” I would not assume this. I don’t grant that so-called “samadhi” is even a coherent concept or clearly defined. What are we talking about when you use the label “samadhi”? Go into the meaning and you will find like I did only some vague, incoherent concept devoid of meaning except and until you include the background belief of notions such as salvation, liberation, transcendence, gurus, masters, or gods. So, what’s our motivation for calling something samadhi? Religious? Salvation? Sloppy, lazy thinking to slap a label onto something that feels good, satisfies our cravings of thinking we are special or tapped into some hidden, inexplicable mystery? Answering one mystery with another is what this yogic nonsense is all about–getting us to follow and trust in what the guru or spiritual authority wants us to believe.
Thanks for taking the time to share your comments
Hi Scott,
I guess we disagree. (by the way, I’ve gained a lot from your site and your pugnacious and dogged defense of the “truths” you have come to.)
I agree that Samadhi is a myth. Samadhi does not exist. However it is defined.
I disagree that people don’t “think” that
they have had a samadhi experience. Go here:
http://www.aypsite.org/forum/pop_printer_friendly.asp?TOPIC_ID=4633
There are a number of people on that site that claim to have experienced Samadhi. Do some of your own searches.
Now, let me make my point again. No matter how you define samadhi, there are people who “think” they have experienced it. Let me try my own metaphor.
You and I can agree that there are no such things as UFOs. We agree on this. We agree no spaceships have come to our planet from other planets. We agree on this.
However, people still report UFOs. Time and time again. People still believe they have “seen” one or many interplanetary spaceships. We don’t know why. But they do.
You and I probably agree that Gurus have selfish objectives. They want to control other people to get money, fame and in many cases sexual favors. Happens all the time. Some gurus are worse than others, but they still want something from others. One of the controlling factors is to convince people that there is a state they can get to that is the end all, be all, state. (samadhi). But it doesn’t actually exist. So few people report it. But my point is that some people do report it. Erroneously, but report it none the less.
The people who report samadhi do so because of something, Psychotic break, intermittent manic state, lying to gain status in a larger group, spontaneous release of neurotransmitters including DMT? … I don’t know why, but they do.
My point about DMT is that if science proves that it does exist and can get released, it might explain why some of these people report samadhi.
Cheers, Brent
@Brent. Respectfully, I don’t understand your point. We could go around and around in circles for ever on this topic.
I strongly recommend, if you wish to have a constructive conversation about “samadhi”, that you define what you mean by “samadhi”. I asked you to in my prior reply.
After decades of meditation practice, and questioning its premises, I find the term “samadhi” to be vague and vacuous of objective meaning.
What is Samadhi? Is there a logically coherent or consistent definition outside a specific religious tradition? I have yet to find one. That doesn’t mean I think there’s no samadhi, or that its false or wrong. But that doesn’t make it real or valid. Indeed the more I objectively researched definitions of Samadhi the more contradictions and vagaries I discovered. (Read my post Contradictions with Samadhi).
My thoughts on meditation practices and especially yoga traditions have evolved and changed over time. Who knows? Maybe someone will sometime demonstrate, in a credible way, that there’s something useful in “samadhi”. Until then to me “samadhi” is only useful inside the context of a religious tradition, yogic belief system, or spiritual teacher whose followers must take the mahatmas claims on faith.
I admit that all that I know of “samadhi” is what I’ve read on this blog; it is basically a foreign concept to me. However, it seems to me that what Brent was trying to say is that the subjective experiences that some meditators interpret as “samadhi” may be nothing more than a neurochemical trick that is induced by fluctuations in the chemical DMT. It’s on the subject of DMT that I wish to chime in.
N,N-dimethyltryptamine, or DMT, is structurally similar to serotonin. So far, it has been found in rabbits and rodents, but has not yet been isolated as a naturally occurring substance in the human brain. It is found in certain plants, however, and many of these plants, such as ayahuasca, have been consumed for centuries in order to produce psychedelic experiences. Another group of plants with natural concentrations of DMT is the Desmodium genus, several species of which are found throughout India, where many of these meditation practices originated. Some of the psychoneurological effects of DMT include the feeling of visiting a parallel reality, sensory and spatial distortions, depersonalization (possibly akin to “anatta”) and visual and auditory hallucinations. We don’t know what substances the gurus, teachers, “masters” and Buddhas of the past may or may not have been exposed to. So for my money, it’s possible that some of them were meditating while taking one long strange trip. In turn, this may have set the standard for certain ideals or norms of spiritual experience; so basically people now go around unknowingly trying to replicate through meditation what may very well have been psychedelic drug experiences while believing themselves to be on some sort of spiritual or transcendental quest.
Oh, and someone brought up UFOs somewhere, and how people continue to believe they have seen them…DMT can induce hallucinations of contact with UFOs and aliens too.
I hope this has been somehow thought-provoking or clarifying.
@Red,
That’s an intriguing theory–the subjective experiences some have samadhi could have been caused by yogi-meditators ingesting hallucinogenic substances.
I agree with you: It’s our background beliefs that interpret “trips” as spiritual, transcendent, or ordinary. Heck, during sleep it’s ordinary for us to have dreams that transcend “this world”.
Thanks for your comments.