Do individual ecologists review in proportion to how much they submit? Here’s the data! (UPDATEDx4)

One oft-voiced concern about the peer review system is that it’s getting harder to find reviewers these days (e.g., this editorial from the EiCs of various leading ecology journals). Which isn’t surprising, given that academics have strong incentives to submit papers, but much weaker incentives to review them.

A few years ago, Owen Petchey and I proposed a reform known as PubCreds, the purpose of which was to oblige authors to review in appropriate proportion to how much they submit. For instance, if each paper you submit receives two reviews, then arguably you ought to perform two reviews for every paper you submit.

Owen and I pitched PubCreds to various individuals and groups who might’ve had some power to make PubCreds happen, and we didn’t really get much traction. One reason among many we didn’t get much traction is that people questioned the need for PubCreds. Heck, even some of the authors of the editorial linked to above questioned the need for PubCreds! This was somewhat frustrating to Owen and I, but in retrospect I can understand it. The fact was, we didn’t have much hard data demonstrating the breakdown of the existing peer review system, at least not a breakdown so serious as to make major reform a matter of urgency.

So Owen and I decided to go get some data. The online ms handling systems that most journals have used for years compile data on how often individuals submit, how often they’re invited to review, and how often they agree to review. So we approached the EiCs and managing editors of something like 30 ecology journals, asking if they’d be willing to share (anonymized) data. The only journals from which we received a positive response that was then followed up were Population Ecology (which didn’t have enough data to be useful), and the journals of British Ecological Society (BES), of which Lindsay Haddon was managing editor at the time. Thanks to Lindsay’s hard work extracting the relevant data from Manuscript Central, we were able to compile an anonymized dataset on how often individuals submitted to, reviewed for, and were invited to review for, the four BES journals from 2003-2010. Our paper analyzing the data has just been published (Petchey et al. 2014; open access).

Here are the headline results (read the paper for details; it’s short).

  • Our main question of interest was whether individuals’ reviewing and submission activities were in balance. In our dataset, “in balance” meant “doing 2.1 reviews for every paper you submitted”, since the average paper in the dataset received 2.1 reviews (UPDATE: defining “in balance” relative to the mean number of reviews per paper corrects for rejection without review. If you were to do a similar analysis for, say, Science and Nature, you’d presumably find that the average paper receives much less than 2.1 reviews, because many papers are rejected without review UPDATE #2: In the comments, Owen jogs my memory, reminding me that we included in the analyses only submissions sent out for review.) For 64% of individuals in our dataset, the answer is “no”–they either did at least twice as many reviews as needed to balance their submissions, or less than half as many. So the majority of individuals are either “overcontributors” or “undercontributors” to the peer review system.
  • The relative abundance of over- vs. undercontributors depends on the assumptions you make about how to distribute “responsibility” for multi-authored papers (e.g., if you’re corresponding author on a multi-authored paper, does that mean you personally should do 2.1 reviews to balance that submission?) Depending on assumptions, 12-44% of individuals did at least twice as many reviews as needed to balance their submissions, while 20-52% did less than half as many.
  • Undercontributors mostly didn’t agree to do all the reviews they were invited to do. So undercontributors mostly didn’t undercontribute due to lack of opportunity to review, at least not completely.
  • Researchers who submitted more were more likely to accept invitations to review.

Obviously, few ecologists submit to, and review for, only the BES journals. But there’s no reason to think that this biases our results, as far as we can see. So we’re reasonably confident that our results wouldn’t change if someone were somehow able to compile a larger dataset from more journals. (UPDATE: I’m sure some people who are undercontributors to BES journals would be in balance, or even overcontributors, if you accounted for their reviewing and submitting for other journals. But I’m sure some people who are overcontributors to BES journals would be in balance or even undercontributors if you accounted for their reviewing and submitting to other journals. And I’m sure some people who are in balance in our dataset would not be in balance if you had data from many more journals. So when I say our results are unbiased as far as we can tell, what I mean is that Owen and I can’t see any reason why ecologists would tend to overcontribute to BES journals compared to ecology journals as a whole. Or tend to undercontribute to BES journals compared to ecology journals as a whole. Or tend to make more balanced contributions to BES journals than to ecology journals as a whole. But if you can see a reason why BES journals might represent a biased sample, please say so, I really do want to hear people’s thoughts on this!)

(UPDATE #4: In the comments, Douglas Sheil suggests a potential source of bias in our estimate of the proportion of people who are in balance. Briefly, the fact that we’re working with a sample of journals rather than a census of all journals might cause our data to underestimate the proportion of ecologists who are in balance, even if our data are a random sample (i.e. people react towards BES review requests the same way they react towards non-BES review requests). I just did a quick and dirty simulation to check out this suggestion and it looks like there might be something to it. Hard to say much more than that without a much more thorough simulation study. And even then it might not be possible to say more, since I’m sure that the existence and strength of any bias will be sensitive to the assumptions one makes, and many of those assumptions can’t be checked with the data we have. I doubt I’ll be able to make time to really look into this thoroughly, but if somebody wanted to pick up this idea and run with it, I could try to pitch in…)

Overall, I was pleasantly surprised by the results. I was too cynical–I thought we’d find a very large proportion of people reviewing very little relative to how often they submit, balanced by a small proportion reviewing a lot relative to how often they submit. In fact, overcontributors aren’t that rare relative to undercontributors, and might even be more common than undercontributors. I was also pleasantly surprised that, on average, people who submit more are more likely to accept invitations to review. So score another victory for data over anecdotal impressions.

But having said that, undercontributors aren’t rare in an absolute sense, and so the only thing that keeps the system from breaking down is that the undercontributors are balanced by a sufficient number of overcontributors. Obviously, our data don’t give us any basis for predicting whether or how the relative abundance of over- and undercontributors might change in future.

UPDATE #3: Let me emphasize a point made in the paper, which I should’ve emphasized more in the post. We have no information on why individuals over- or under-contribute. There could be many reasons, some better than others. For instance, some undercontributors may serve as editors, and so decline requests to review because they contribute to the peer review system via their work as editors. Whatever the reasons for individuals over- or undercontributing, it’s of practical interest to know how common such individuals are.

Owen and I haven’t done much with PubCreds for a while now, but I still find these data interesting in their own right, and hope others will as well. The dataset is on Dryad for anyone who wants to explore it.

Apologies for the self-promotional post, I don’t ordinarily post on my own papers. I’m only doing it because it’s related to a topic I’ve blogged about in the past, and because I’d be very interested to hear what folks think of our results. Looking forward to your comments.

27 thoughts on “Do individual ecologists review in proportion to how much they submit? Here’s the data! (UPDATEDx4)

  1. Another thing that surprised me about these data is just how much variation there is. I had no idea that there were people out there who review *that* much relative to how often they submit, or who review *that* little. It’s far more variation than can be explained by a model assuming that everyone has an equal probability of accepting every review request.

  2. That’s a very cool analysis. I think that starting a formal PubCred culture enforced by Journals could be good, but is unlikely to happen, as it require a big joint effort difficult to coordinate. However, the process would be easier if there is an informal PubCred culture. For example, if there exists a common recognized place where researchers can report their peer-review activity in order to get credit (e.g. in the CV) would be a tremendous step forward. That can be done based in authors self-reporting (good faith based system) or more formally by requiring Journal validation. ORCID would be probably the right place and that way over-contributors can get a fair credit for their labor. It can also create a cultural shift that lead to a more formal PubCred system in the future.

    • “However, the process would be easier if there is an informal PubCred culture. For example, if there exists a common recognized place where researchers can report their peer-review activity in order to get credit (e.g. in the CV) would be a tremendous step forward. That can be done based in authors self-reporting (good faith based system) or more formally by requiring Journal validation. ORCID would be probably the right place and that way over-contributors can get a fair credit for their labor. It can also create a cultural shift that lead to a more formal PubCred system in the future.”

      Owen would heartily agree, I think. He’s very much in favor of something along these lines. I confess I’m a bit skeptical that such self-reporting can create or maintain a culture in which everybody reviews in appropriate proportion to how much they submit. After all, we already have professional norms obliging scientists to do reviews. Insofar as some people no longer buy into those norms, I’m not sure that “peer pressure” will change their behavior. But don’t get me wrong, I don’t think self-reporting can hurt, and it’s possible that it might help.

  3. Interesting approach Jeremy. Does it matter to your results if an individual frequently reviews for BES journals but doesn’t (or rarely publishes in them? Does this not bring in a bias?

    • Oh, I’m sure there are some individuals who review often but submit little for BES journals. Those individuals look like overcontributors in our dataset, but they aren’t really (at least not to the extent they appear in our dataset) But I’m sure there are other individuals who submit much but review little for the BES journals, and so look like undercontributors in our dataset but aren’t really. And I’m sure there are some individuals who look balanced in our dataset, but who are actually over- or under-contributors. That’s what I mean when I say our data are unbiased as far as we can tell. There’s no reason to think that people are especially likely to overcontribute to BES journals, or undercontribute to BES journals, as compared to ecology journals as a whole.

      • Doesn’t it depend to some extent on pre-review rejection rates of the journals? So if you were to try a similar analysis for Science and Nature, you’d find that everyone was an over-contributor!

      • That’s why we define a balanced contribution relative to the average number of reviews each submission in our dataset received. Journals with high rates of rejection without review would have a low mean number of reviews per submission. So no, I wouldn’t expect that everyone would look like an overcontributor if you did the same analysis for Nature or Science.

  4. I’ve been hoping to see this study for a while – thanks for working so hard to make it happen. It’s unfortunate that you only had access to the anonymised data, as clearly there’s a lot of variables that you had to set aside. One big one that jumps to mind is being an editor – I know that plenty of academics see working as an editor as a valid reason to turn down review requests, as they are making a major contribution to the peer review process as it is. Being able to add data on how many editorial boards everyone was on would be very interesting, and may ‘mop up’ a considerable proportion of the under-contributors.

    Another complication is that more senior authors may get many more requests to review (e.g. here: http://scholarlykitchen.sspnet.org/2012/02/01/the-famous-grouse-do-prominent-scientists-have-biased-perceptions-of-peer-review/), but they also tend to be co-authors rather than submitting authors.

    One technical question – was the ‘submitting author’ the person who submitted the paper via Manuscript Central, or the first author on the published paper? We do see plenty (30%?) of senior academics act as the ‘submitting author’ when they’re not the first author on the paper.

    • “It’s unfortunate that you only had access to the anonymised data, as clearly there’s a lot of variables that you had to set aside. One big one that jumps to mind is being an editor – I know that plenty of academics see working as an editor as a valid reason to turn down review requests, as they are making a major contribution to the peer review process as it is.”

      Absolutely. In the paper, we’re careful to note that we have no information on why individuals over- or undercontributed. Nobody should read our paper as criticizing undercontributors. Indeed, even if people undercontribute simply because they have strong incentives to publish and few incentives to review, well, I personally have no problem with people responding to incentives. With regard specifically to service on editorial boards, our dataset is large enough (4055 individuals) that I doubt more than a small fraction of individuals in it were editorial board members, but obviously I don’t know.

      “Another complication is that more senior authors may get many more requests to review”

      Not something we can really get at. We can get at some aspects of seniority, by using the year in which an individual first entered our dataset as a proxy for seniority. For instance, we found that the positive association between “number of submissions” and “proportion of review requests agreed to” is especially strong for individuals who entered the database recently (so, presumably more junior individuals). It’s weaker for individuals who entered the dataset earlier on.

      “One technical question – was the ‘submitting author’ the person who submitted the paper via Manuscript Central, or the first author on the published paper?”

      I believe the submitting author is the person who submitted the paper via MS Central–the corresponding author.

  5. Great post, and I love the idea of PubCred – related ideas with more alliteration here: http://pacificsoutheast.wordpress.com/2014/03/19/the-responsible-researcher-reinforcement-index/

    I think authors need to balance not just the reviewers’ work, but also the editors’ work, so I think to play fair you actually need 3.1 reviews per reviewed submission (and 1 per non-reviewed submission, to cover the ed – which might also make people rethink shooting ridiculously high).

    • “I think authors need to balance not just the reviewers’ work, but also the editors’ work”

      When Owen and I proposed PubCreds, we actually had a lot of discussion between ourselves and with colleagues as to whether the editor’s work is equivalent to that of a reviewer. We encountered a range of opinions–including from editors! That old PubCreds post links to the old PubCreds petition website, where I think we discussed this issue.

      And editors-in-chief (who have quite different duties from handling editors) are still another ball of wax…

  6. Just some technical clarification.

    Jeremy is correct. The submitting author was the researcher that submitted to MS Central.

    We included in the analyses only submissions sent for review, by removing manuscripts where the number of invited reviewers was equal to zero. So the average of 2.1 reviews per paper was calculated across only the papers that were sent for review.

  7. The self-promotion is entirely forgivable, Jeremy — it’s an interesting topic and an interesting post/paper. My main question is about this sentence (from the abstract): “These finding[s] suggest overall that peer review of the four analysed journals is not in crisis, but only due to the favourable balance of over- and under-contributing researchers.” If I understand your approach correctly, the collective deficits of undercontributors HAD to be balanced out by overcontributors. You found that the overcontributor tail of the reviewing activity distribution was fatter than you (Jeremy) expected, but I have no sense that having a lot of overcontributors is better or worse (or neither) than only having a few. Can this dataset even be used to address the hypothesis that the peer review system is in crisis? If so, what would a “crisis” distribution look like?

    • I think a crisis distribution would look like what I cynically expected to see: lots of undercontributors balanced by a few people overcontributing a lot. Such a situation is fragile. If the overcontributors stop overcontributing, then suddenly journals can no longer get all the reviews they want to get.

      As an aside, there are other ways the system could break down that wouldn’t show up in our data at all. For instance, as noted in the updates, we threw out papers that were rejected without review. But that’s one big way that journals deal with the increasing difficulty of getting reviewers–don’t bother asking for reviews at all. I personally don’t like this. For me, a major motivation for PubCreds was to try to create a world in which rejection without review is very rare.

  8. Thanks Jeremy – very interesting!
    I wonder about that scatter though.
    Some people do more and some do less than their fair share of reviews for BES journals. OK, but of course the BES journals are a sample of a wider set of journals. I presume that if, when all journals were included, everyone was doing exactly their fair share, there would still be some broad scatter of over- and under-reviewing in any subset of journals due to sample noise. Could you/we make a few simplifying assumptions and estimate (or simulate) that “null scatter”? I suspect that the outliers will still be notable outliers, but how much of the rest is simply noise?

    • Hmm…we find many more over- and under-contributors than one would expect by chance if everybody knew what proportion of review requests they should accept in order to balance their own submissions, and accepted review requests with those probabilities. Off the top of my head, I don’t how that could just be due to noise introduced by taking a random sample of everyone’s reviewing and submission activities. Put another way, I don’t see why our sample should be biased towards finding a lower proportion of in-balance researchers than actually exists. But I could be wrong, and one certainly could do the sort of simulation you suggest–randomly simulate a world in which everyone is “in balance”, and then take a sample and see if the sample appears to contain lots of imbalanced researchers.

      • Thanks.
        Perhaps I am thinking of it wrongly, but let me try and be clearer why I expect a broad scatter (few points in balance). If the commitment to balance was broad, and not per journal, then which journals gets an author’s submissions and which get their reviews may be (near) random. Then a balance of counts would seem unlikely within any journal (or small collection of journals).
        Consider a set up where two Poisson event generators are set to produce events at a given ratio … (rate of submissions versus reviews) what are the chances of actually observing the counts that match that ratio in any sampling interval? I think it’ll be very noisy (assuming low counts for most authors). Anyway, just a thought.

      • Ok, I think I see better what you mean. Owen and I will definitely keep mulling it over and maybe have a go at simulating something like this. It’s the sort of thing that we could easily report on the blog.

      • @Douglas,

        Ok, I just spent an hour playing around with a simple simulation model, and yes, it looks like you could be right. It could be that the proportion of individuals who are in balance might be underestimated if you just look at their reviewing and submitting for a random sample of journals. Of course, there are various assumptions one has to make to do the simulations and I haven’t made any effort to explore the range of reasonable possibilities. In particular, I’ve only simulated a case in which, at a global level, everybody is in balance; I haven’t yet simulated cases where, at a global level, there are over- and under-contributors So I can’t really say if or how much we might have underestimated the proportion of individuals who are in balance. But it’s possible that we did underestimate it just due to the fact that we only had a sample of all journals to which people submit or review.

        Let me also note that, so far, I haven’t seen any reason why taking a random sample of journals would bias one’s estimate of the relative frequencies of over- vs. undercontributors. So I’m still confident in that aspect of our results.

  9. Very interesting post/paper… and not terribly surprising (based on my habits and those of some of my colleagues). Nonetheless, great to have some data. Thanks.

  10. Pingback: What we’re reading: Genomics for conservation, SNPs versus microsats, and imbalance in the peer-review ecosystem | The Molecular Ecologist

  11. Pingback: URLs of wisdom (30th March) | Social in silico

  12. Pingback: Peer review, reviewed | The Molecular Ecologist

  13. Pingback: FYI: rejected mss often get the same referees when resubmitted to a different journal | Dynamic Ecology

Leave a Comment

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s