Journals have a responsibility to ensure ethical oversight of mental health research (and we do not currently have evidence that grad students are 6x as likely as the general population to have depression and anxiety)

I care deeply about mental health in academia (and have blogged about it in the past, including here and here and here). Given that, I was really interested when a recent paper by Evans et al. came out on graduate student mental health. However, when I read it, two things stood out to me: it didn’t mention IRB approval, and the most striking conclusion – that graduate students experience anxiety and depression at 6x the rate of the general population – is not supported by the study. The key messages of this blog post are:

1. the authors did have IRB approval to do this work, but Nature Biotechnology did not know that when they published the study. The editor of Nature Biotechnology claims that, since they published this in their Career & Recruitment section, it is not a research article and therefore didn’t require peer review or questions about IRB. This is problematic, as the study is clearly written and presented as presenting new findings, and journals have a responsibility to ensure ethical oversight of work they publish.
2. While the Evans et al. paper claims “Our results show that graduate students are more than six times as likely to experience depression and anxiety as compared to the general population,” that claim is not supported by their study. Their survey was not a representative sample of the graduate student body (it was a voluntary survey, distributed via social media and email), but they compare it to a representative survey of the general population to get the 6x statistic.

Again, I want to be clear: the authors did have IRB approval for the work, but I only know that because I wrote the authors directly (after being dissatisfied with the responsiveness at Nature Biotechnology), and Nature Biotechnology did not know they had IRB approval when they published the study. In addition, this study does not provide evidence that grad students are six times as likely as the general population to experience depression and anxiety.

Graduate student mental health is really important, so we need to get as accurate a picture as we can of the current situation regarding graduate student mental health. As discussed below, a study (by Levecque et al.) with a more carefully controlled comparison group found a 2.4 increase in risk in graduate students compared to the highly educated general population. This is definitely something that is still a problem and that still needs to be addressed, but it’s not a 6 times greater risk.

To expand on these points more:

As I said above, one thing that stood out to me immediately when I read the paper was that it didn’t include information about IRB approval. Part of why this stood out to me was because they used the PHQ-9 survey; this is a standard screening for depression, but its last question can lead to a higher level of IRB scrutiny. I know this from personal experience, as Michigan’s IRB had concerns about it being included in our survey of mental health in Intro STEM students, leading us to remove it.

Because IRB oversight is important, and because the Nature Biotechnology instructions to authors includes a link to a page that says “The manuscript must include a statement identifying the institutional and/or licensing committee approving the experiments, including any relevant details”, I wrote to the Editor of Nature Biotechnology, Dr. Andrew Marshall, to ask if the publication could be amended to include information about IRB approval. In his response, he stated, “The article you are referring to was published in the Career & Recruitment section of the journal, which is not research. The Instructions to Authors you refer to are for primary research articles.” I was flabbergasted. This article is clearly presenting new research. It’s the first time these data were reported, and their results are presented as new findings (including in the title – “Evidence for a mental health crisis in graduate education” – and header – “With mental illness a growing concern within graduate education, data from a new survey should prompt both academia and policy makers to consider intervention strategies”). Nature Biotechnology claiming that it wasn’t research because of what section they published it in – and that they therefore didn’t need to have asked about IRB approval prior to publication – does not seem responsible. In his email reply to me, Dr. Marshall did say that they agreed that it would have been preferable to ask about IRB approval, and were reaching out to the authors to do so. After not hearing back right away, I decided to reach out to the authors directly. (I had guessed they were likely to reply quickly, because it doesn’t take long to look up a protocol number.) The authors did indeed respond quickly and were happy to provide me with the IRB approval from the University of Kentucky. So, again: the work had IRB approval, but Nature Biotechnology did not know that prior to publishing the study.

In an email to Dr. Marshall last week, I asked: “Do you plan on making changes to your editorial process so that, in the future, you avoid publishing work on human subjects without ensuring the work had IRB approval?” I haven’t received a response to that question yet. I very much hope that Nature Biotechnology (and Nature Publishing Group, more broadly) address this.

My second concern relates to the 6x statistic, which has been very widely reported (including in an editorial in Nature, as well as other places). The Evans et al. paper claims that “graduate students are more than six times as likely to experience depression and anxiety as compared to the general population”, but it does not show this. The key is that the Evans et al. survey of graduate students was done online, circulated via email and social media. I know from personal experience with doing a similar survey that the responses will be biased – it’s pretty easy to imagine that people who have a mental health condition are more likely to invest the time in taking the survey than are people who do not. Evans et al. compare the results of their survey to a representative survey of the general population (done in Germany), where the researchers did face-to-face surveys of households. Because of the very different methodologies, it is not reasonable to compare the results of the two surveys.

I know that one common reaction to the study has been something along the lines of “We finally have a number to show how bad the problem is!” My response is that, no, this study doesn’t give us a number that tells us about the graduate student population as a whole. A study that did have a representative sample – at least, of one particular population – is the Levecque et al. 2017 study, which found “the prevalence of having or developing a common psychiatric disorder was 2.43 times higher in PhD students compared to the highly educated in the general population. It was 2.84 times higher compared to highly educated employees and 1.85 times higher compared to higher education students.” Those numbers aren’t great, but they are notably lower than the ones reported in the Evans et al. study. It’s possible that there’s variation between different countries, disciplines, etc. – the Levecque et al. study notes that the graduate students in their study were relatively well-supported financially, which might reduce stress. But, at present, the simplest explanation for why the Evans et al. rate is so much higher is that the comparison they are doing to get the 6x rate is flawed.

To me, this problem with the way the Evans et al. study interpreted the data was a really obvious issue, and I was surprised that it hadn’t been caught during peer-review. (I realize peer review isn’t perfect, but this seemed like a basic issue that I would have expected to be caught.) So, the email from the Nature Biotechnology editor indicating that it hadn’t been peer-reviewed helped explain that. But it was still surprising to me how few people were noting this very important caveat in their stories, tweets, etc. about the study. My guess is that it’s an example of confirmation bias – I suspect that, if it showed a 6x lower risk in graduate students, people would have immediately dug into the paper and recognized the issue related to the comparison.

Summary

Graduate student mental health is very important and I wish we had a better understanding of the scope of the problem and what can be done to improve graduate student mental health. But it is absolutely essential that this work be done with IRB oversight (as Evans and colleagues did) and that journals ensure that such oversight occurred before publishing work on mental health (which Nature Biotechnology did not). It is also very important that the results be interpreted carefully; in my opinion, peer review is likely to help ensure this. The response to the Evans et al. piece shows that there is a lot of interest in graduate student mental health, and also that there’s a general sense that there might be a mental health crisis among graduate students. Key next steps will be to review the evidence we currently have and to identify gaps in our knowledge that suggest important avenues of future research.

 

Update May 11, 2018: Dr. Marshall from Nature Biotechnology wrote (on May 2 — I’m slow to get the update posted!) saying that 1) the Evans et al. article is being amended to include the IRB information and 2) they are changing their editorial process to flag articles submitted to the Career & Recruitment section that involve human subjects research. I don’t have any more specific information than that, but it’s good to hear that they are changing their procedures.