Mental Health

Publication Bias in Psychotherapy for Depression

by Michael D. Anestis, M.S.

 I want to jump start our conversations by discussing an interesting study that has been coming up in a number of different forums in recent days (including the comment section here at PBB).  The article, just published in the British Journal of Psychiatry by Pim Cuijpers, Filip Smit, Ernst Bohlmeijer, Steve Hollon, and Gerhard Andersson takes a look at the degree to which the impact of psychotherapy for depression might be overstated due to publication bias.  I’ll explain what I mean by publication bias in a moment, but first I’d like to direct you to two links.  The first (click here) will take you to our coverage of recent work indicating that the impact of antidepressant medications has been overstated, particularly with respect to mild to severe cases of depression.  The second (click here) will take you to our coverage of other recent work by Dr.Cuijpers and his colleagues looking at potentially inflated effect sizes in psychosocial treatments for depression.  An important point to keep in mind as you read those articles and this one is that nobody is saying these treatments are ineffective, but rather that we have not been attentive enough to the details with respect to empirical data and the publication process and, as such, we have potentially developed a somewhat inflated view of our various interventions.

March 15, 2010

Publication bias in psychotherapy for depression

 

by Michael D. Anestis, M.S.

Welcome back to PBB, everyone.  Today, I want to jump start our conversations by discussing an interesting study that has been coming up in a number of different forums in recent days (including the comment section here at PBB).  The article, just published in the British Journal of Psychiatry by Pim Cuijpers, Filip Smit, Ernst Bohlmeijer, Steve Hollon, and Gerhard Andersson takes a look at the degree to which the impact of psychotherapy for depression might be overstated due to publication bias.  I’ll explain what I mean by publication bias in a moment, but first I’d like to direct you to two links.  The first (click here) will take you to our coverage of recent work indicating that the impact of antidepressant medications has been overstated, particularly with respect to mild to severe cases of depression.  The second (click here) will take you to our coverage of other recent work by Dr.Cuijpers and his colleagues looking at potentially inflated effect sizes in psychosocial treatments for depression.  An important point to keep in mind as you read those articles and this one is that nobody is saying these treatments are ineffective, but rather that we have not been attentive enough to the details with respect to empirical data and the publication process and, as such, we have potentially developed a somewhat inflated view of our various interventions.

Okay, so what do I mean by publication bias?  Actually, it can mean a number of things, but in the context of psychotherapy research, the key component can be described as follows:

  • Randomized controlled trials of psychotherapy are expensive, lengthly, and intricate endeavors.  As such, the vast majority of these trials include a very small number of participants.
  • As we have discussed before on PBB, when you have a small sample size, it is very difficult to find results that are statistically significant.  This is because it is harder to dismiss the possibility that the results of the study are due purely to chance.  Consequently, the only results that meet criteria for statistical significance (p < .05) are huge effects.  In other words, if the one treatment is slightly more effective than the other (or the control condition), we would likely end up with a false negative – a non-significant result.
  • Peer reviewed journals are not inclined to publish null results – the results of studies that are not statistically significant – which means that studies in which treatments only modestly outperform one another are unlikely to even be submitted in some cases and unlikely to be accepted for publication when they are.
  • The result is that the literature is flooded with studies in which one treatment performed substantially better than the other while other studies with more muted effects go unpublished (if they exist).

So, how does this matter in psychotherapy research?  Unfortunately, it means that there is reason to believe that the actual impact of psychotherapy on depression (and likely other diagnoses) is not quite as large as we have come to believe it to be.  Again, keep in mind, this does not mean that treatment does not work or is barely effective.  It simply means that we are not being entirely accurate in our understanding of the phenomenon.

Cuijpers’ research group (2010) addressed this issue by conducting a meta-analysis of treatment studies for depression.  Now, I have expressed my reservations about meta-analyses in general in the past; however, my position is certainly open to debate and, regardless, it is imperative that we discuss these types of studies when they come up, as they are quite influential within the field.  Anyway, Cuijpers and colleagues examined nearly 10,000 abstracts in compiling a list of studies that fit the inclusion criteria.  Ultimately, 117 studies in which 175 comparisons between treatment conditions were made met criteria and were included in the analyses.

The researchers used a number of statistical techniques to estimate the likelihood that publication bias was present.  These approaches, which include examining funnel plots, do not conclusively demonstrate that less impressive studies exist but have not been published.  Instead, they indicate the likelihood that this is the case.  That being said, although ambiguity remains and the authors openly admit this to be the case, their conclusions are not based upon random guesses but rather a thorough examination of probability as determined through the pattern of results displayed by published manuscripts.

Looking at psychotherapy for depression in general, the authors found substantial evidence of publication bias.  There was no list in the article detailing every single form of therapy examined (although a list of all of the studies is available, so this information is not being hidden); however, these analyses presumably covered a wide range of psychosocial interventions.  Because so much research has been conducted on cognitive behavioral therapy (CBT), the authors had enough studies to run a sufficiently powered analysis examining potential publication in this specific therapeutic modality and here again, they found substantial evidence of publication bias, with numbers that were essentially identical to those for other therapeutic approaches.  The only therapeutic modalities that did not show any evidence of publication bias were interpersonal psychotherapy (IPT) and psychotherapy for women with postpartum depression; however, the authors noted that “these results have to be considered with caution, however, because these subsamples were relatively small and may have been the result of random error” (p.176).

So, ultimately, the authors found compelling evidence that publication bias might be causing us to overestimate the impact of psychotherapy for depression and that this point is as true for CBT as it is for other approaches.  That being said, they also acknowledged some limitations.  As I mentioned earlier, Cuijpers and colleagues’ (2010) analyses do not provide direct evidence of publication bias, as they did not uncover unpublished studies with muted effects.  They simply highlight the likelihood that such studies are out there.  Along these lines, the authors noted a number of alternative interpretations of their data.

Given these findings, you might find yourself wondering what to make of clinical research.  After all, if we are overstating the impact of psychotherapy (and pharmacotherapy) for depression, is the picture here actually a gloomy one?  First of all, the answer to that question is no.  Treatment for depression is imperfect but strong.  These findings simply highlight the idea that we should not become complacent in our search for something better and the importance of identifying which components of therapy directly impact recovery so that the most pivotal aspects can be further emphasized in the future.  Having said that, there are a number of things that we, as a field, could be doing in order to better avoid this type of problem.  One option suggested by the authors in this paper is for all researchers to register clinical trials in a public forum regardless of the outcome so that we can better understand the degree to which what is published accurately reflects what is going on in “real life.”  Along these lines, the Dr.Cuijpers and others have developed a website – www.evidencebasedpsychotherapies.org – that includes a growing registry of clinical trials and I highly recommend that you explore it.  Another option would be for the field to develop a number of journals specifically devoted to publishing non-significant (null) results.  These journals, which would need to screen out null findings that were the result of design problems, poor implementation of the treatment, and other such variables, would allow people to have access to instances when research did not go as expected.  Typically, such studies are locked away in a file drawer, as they are not capable of being published and, as a result, nobody knows that they exist.  The thing is, a non-significant finding provides important information and, if nobody knows about them, we are all less informed as a result.

So what’s the take home point here?  As we often proclaim here at PBB, Dr.Cuijpers’ research group highlighted the importance of making sure that our message is actually supported by the evidence.  In this case, the evidence seems to indicate that we have been overstating the impact of psychotherapy for depression.  This, of course, does not mean that psychotherapy for depression is ineffective, but rather that, for a number of reasons, we may have developed an inaccurate view of just how strong of an impact it has.

I look forward to hearing your thoughts on this study.  How do you think we can address publication bias and what do you believe the implications are of these findings?

************

If you would like to learn more about this and other topics discussed on PBB, we encourage your to consult our online store for scientifically-based psychological resources.

Mike Anestis is a doctoral candidate in the clinical psychology department at Florida State University and an incoming resident at the University of Mississippi Medical Center.

Leave a Reply