A few months ago, Stephen Heard wrote a blog post that prompted us to have a brief twitter discussion on whether we sign our reviews. Steve tends to sign his reviews, and I tend not to, but neither of us felt completely sure that our approach was the right one. So, we decided that it would be fun for us to both write posts about our views on signing (or not signing) reviews. In the interim, I accepted a review request where I decided, before opening the paper, that I would sign the review to see whether that changed how I did the review. So, in this post I will discuss why I have generally not signed my name to reviews, how it felt to do a review where I signed my name, and what I plan on doing in the future.
First, here’s a link to Steve’s post. It’s great and, since I read it before working on my post, mine is partially a response to it.
I agree with Steve that I don’t think signing reviews tends to make them “better”. And, going further, I think you can make an argument that not signing reviews makes them better. To me, the most important reason for reviewers not to sign their reviews is so that they can be critical without worrying about retribution. I don’t want people who are reviewing my papers to be excessively harsh, but I also don’t want them not to point out an error or concern because they are worried about upsetting me. And, based on my experience as an associate editor (where I sign all my recommendation letters), I know that it can be hard to ignore that voice that is saying, “Boy, this is not going to make them happy”. I don’t think that voice has ever changed what I recommended, but it was a distraction and would have been even more of one if I were at an earlier career stage.
I did wonder, though, if having been an AE for several years would make me feel differently about signing a regular review, since I now have a lot of practice with signing things, even when I know it won’t be the outcome the author hoped for. So, as I said above, I recently accepted a review where I decided as soon as I accepted it that I was going to sign the review. I ended up really enjoying the manuscript and, while I had some suggestions for things to address, it seemed clear to me that the work was well done and that it was a good fit for the journal.
So, I was a little surprised that, when it came time to submit the review, I waffled for a bit about whether to go through with signing it. The biggest thing I was thinking was that it felt pointless – I didn’t see what benefit would come from signing it, so I wasn’t sure why I should change from my default of not signing it. I also kept thinking of what I had learned in grad school: that one should always sign reviews or never sign reviews. The idea behind this is that, if you sometimes sign reviews and sometimes don’t, that quickly leads to you signing only the positive ones, inviting a quid pro quo system. I’m not really sure how likely that is (is there data on this? I’d love to know), but it does seem plausible to me. (Note: after writing this post, I was reminded of this older post by Terry McGlynn where, among other reasons, he notes the potential for tit-for-tat as a reason not to sign reviews.)
Given that advice, I decided early on to not sign reviews, feeling like it would let me be more critical (while still being constructive) – but also, if I am being honest with myself, because reviewing has always brought out my imposter syndrome and anonymity helped with that. But, like Steve, I have not always gone with my default approach. When I do sign, it’s usually because I thought I needed to identify myself to help explain a question, suggestion, or critique. But that has been rare.
Now that I’ve done this little trial of signing a review, I plan on going back to my default approach of not signing reviews. I don’t think there’s much to be gained from signing them – as Steve suggests, there is some potential value to being able to contact the reviewer to ask a question but this is relatively rare (I think I’ve done this once in my career) – and I think there is something to lose. Steve wrote:
Occasionally, I remain anonymous precisely because I wouldn’t be comfortable saying something to an author’s face. But: this is not because I’m worried that in saying it, I’m behaving badly. Instead, it’s because I’m worried about an author behaving badly in response – whether active retaliation or just an unconscious bias against me when my own work is up for review. I think the former is very rare; but the latter probably isn’t. We all remember when our work is criticized; and it’s only human to let those memories sway your future judgement even if you’re intellectually resolved that it shouldn’t***. My occasional anonymity lets me be as critical as I think is needed. I still try to phrase that criticism as constructively as I can, of course, and most reviewers do the same.
This is an excellent summary of how I feel except that, unlike Steve, I have come to the conclusion that I should default to not signing my reviews. I find it very interesting that we seem to be pretty much in agreement about the pros and cons of signing reviews, yet have opposite defaults when it comes to whether we sign our reviews!
Steve concludes:
there seems to be little benefit to my signing. That leaves only the costs. I’ve figured this out, but I still (almost always) sign. I guess I have a choice: I can think of myself as foolish, for sticking with a practice that cost-benefit analysis can’t support; or I can think of myself as selflessly tilting at a worthwhile windmill. You can probably guess which way I lean.
I think my default position can similarly be framed positively or negatively. You could argue that I’m just being pragmatic – there are some clear potential downsides, and few upsides to signing reviews, so why bother? – or you could argue that I’m a wimp who lacks the courage of my convictions. I’m not sure which way I lean.
*** Steve’s footnote for this is:
I’m disagreeing here, I think, with Jeremy Fox over at Dynamic Ecology, who thinks we needn’t worry about “disagreeing publicly with Dr. Famous”. I’d like very much to agree with him, but can’t quite.
I guess this means I’m also disagreeing with Jeremy. Sorry, Jeremy!
Dynamic Ecology 
I don’t usually sign my reviews. But sometimes the authors can guess it is me. Most often that happens when they have cited my articles, misspelled my name and I have to correct it. 😉
Ah, I recognize that… How hard can it be to get the ä into the name..? I guess people do not realize it is a completely different letter. We should start consistently misspell the other way around (a–>ä)… On the issue of correcting it in reviews, I always try to find other names which are misspelled (there is almost always another one) to make it more difficult for the authors to guess 😉
I generally don’t sign either, but try to write as if I were.
N=1 rep of each gender when it comes to Meghan and and Stephen, but I wonder if some of it comes down to that.
I mostly don’t sign mine.
Not signing is much more common than signing in my experience as an editor and author. Which makes me hesitant to guess whether the rare people who sign are disproportionately likely to be men.
I think signing or not signing is a convention and not following the standard has more risks and consequences than following it. If you sign I strongly think you should sign all. Otherwise it could be you are just signing favorable ones. If everyone signed, then people would get used to it. If you sign and others don’t and you are negative, you will be remembered, even by people who don’t want to be vindictive. It is kind of like salary. It is secret a lot of places and open at others. If you tell where it is generally secret, it looks like you are trying to make some point. I do not sign any reviews, but I try to be kind. I imagine the paper writer is my own child or struggling student and try to help.
I mostly don’t sign:
-because not signing is the convention, as Joan says above. And because I think there are good reasons for the convention and so I’m not motivated to try to change the convention by signing.
-because I want to feel free to be honest without running the risk of authors getting annoyed with me. I’m not at all worried about retaliation, though, for reasons spelled out in the old post Meg and Stephen linked to in their footnotes. And like Meg and Stephen I don’t feel like I’d write reviews differently if I did sign. And I’m not even worried about the possibility of later having a brief awkward interaction with someone to whom I’d given a negative review. It’s vaguer, less calculated, and more subconscious than that. I think it’s just that, like most people, I’d rather have other people like me, so why sign a negative review and run a slight risk that the authors won’t like me?
-because I want the authors to focus on what I said, not who I am.
I used to make exceptions when the review was very positive but lately I’ve stopped doing that. These days I only sign if I know the authors personally. And I’m considering ceasing to do that in the interests of promoting the convention of reviewer anonymity. If I ever want to let the authors know that I reviewed their paper, I can tell them separately in a private email.
Also, the fact that Meg and Stephen weighed up the same considerations but come to opposite decisions suggests to me that the issue is not cut and dried, and that it in most cases it doesn’t matter whether you sign or not. Basically, we’re all struggling to weigh up mostly-hypothetical consequences of signing or not signing. That the consequences of both choices are mostly hypothetical suggests that, typically, there are *no* consequences to either choice.
On Twitter, someone suggested that it’s futile for Meg, Brian, and I to not sign our reviews, because this blog is widely read and so reviewers will guess our identities. To which, nah.
-This blog isn’t nearly as widely read as that; only a minority of ecologists read regularly enough to have a good sense of our writing styles.
-Even regular readers often pay no attention to who wrote which post. Meg, Brian, and I all have had the experience of being complimented for a post someone else wrote.
-Neither Meg, Brian, nor I writes reviews in the same style as we write our blog posts. Ok, my reviews probably tend to be longer than average, just as my blog posts tend to be longer than average. But if you guess that every longer-than-average review you get is from me, you’re almost sure to be guessing wrong, since I’m far from the only person who writes longer-than-average reviews.
-It’s a lot harder to guess reviewers than you’d think. As evidenced by the fact that reviewers trying to guess the author of an anonymized paper sometimes get it wrong, even though they typically have *much* more information to go on than authors trying to guess the identity of an anonymous reviewer.
I also have found that, when I’m copied on a decision email, I sometimes can’t tell if I was Reviewer 1 or 2!
In future, if I can’t tell who the review was from, I’m going to assume it was from you. 😉 Because that lack of distinctiveness seems awfully distinctive. 😉
Even your comments are longer than average, Jeremy :p
I strictly follow the journal’s suggested review format. If it encourages open review (e.g. Ethology, PLoS ONE), I’ll sign – regardless of being positive or negative. If the journal specifies single- or double-blind review, then I do not sign. The journal that asks for the review have typically, for whatever reason, decided on a review format – who am I to go against it? When I get signed reviews back, I usually get a little annoyed – because I don’t really know how to deal with it… Should I thank the open reviewer by name in the acknowledgements (I guess not, but it is quite common..)? I also worry about my semi-conscious bias (both positive and negative) if I ever get to review that person’s manuscript. Signing will likely force such bias upon the receiver of the review… In short, I have not made up my mind about what is best, so I just follow the guidelines.
So, I say we determine whether Meg or Stephen won this argument by looking at whose tweet of their post has the most retweets and likes! #badidea
So far Meg has a slight lead on both counts… 😉
Pingback: Why I sign (most of) my reviews | Scientist Sees Squirrel
Hi Meg,
I’ve enjoyed reading both this and Stephen’s views here, useful insight for young and less young researchers! I generally don’t sign reviews, but aim to write them as though I had (the ‘say it to the face’ approach). I’ve given up on trying to guess who writes reviews of my work, partly because it’s wasted energy, partly because I’m dabbling in new fields where I don’t know the likely reviewers.
In the example you’ve given here, can you clarify exactly when you decided to sign the review? Was it before reading the invitation (including title/authors/abstract) or only after any/all of these elements?
I’m just wondering if that subconsciously affected your decision to try the experiment – you say positive things about the work and its fit to the journal – perhaps that was already apparent from the earliest stage.
If you did do things in that order, how about repeating the experiment with your next request without knowing anything about the article, then reporting back on the experience?
My brain seems to have turned to mush and I don’t remember the exact order! But I think I decided to sign the review after I agreed to the review (meaning: after I had seen the title and authors) but before I’d looked at the paper. I do know that I had not formed any opinion when I decided to sign the review about whether I thought it would be a good fit to the journal, so I don’t think that changed anything.
Once I got a tenure track job (not yet tenure) I toyed with starting to sign reviews. But at least twice I had experiences where it was clear to me that the AE or EiC discounted my review because it was signed. Which was wrong in those cases – I was being objective – but I can understand the thinking. Once it is signed there is a possibility that somebody is playing games rather than being objective (which I hasten to say does not mean people are – I’m sure 90% of signed reviews are straight), but why leave the doubt?
At this stage in my career I am increasingly thinking double blind is the best choice so I am not sure why I would be pushing single-blind towards zero blind.
As for people guessing who wrote the reviews … I used to spend a lot of time playing that game then I had enough incidents where a year later or whatever, I found out how spectacularly wrong I was on a very regular basis and just gave up. It is very easy to remain anonymous as a reviewer. About the only tipoff is to steer somebody to 5 my own papers and few of anybody else’s. And even that is pretty weak to infer from. A couple of times I’ve steered an author to 4-5 of somebody else’s paper and then thought I better back off and refer to only 2-3 so they don’t think the person I’m referring to is the reviewer. So even trying to infer from that is pretty flimsy.
I have recently had my blog posts cited against me by a reviewer on a paper I was a co-author on though!
Oh – and like Mike said, I always write my review assuming it could be shared with the author by accident. I’ve had this happen to me both as an author when a reviewer was accidentally revealed to me and as a reviewer when I was casually chatting with somebody and they suddenly got this lightbulb over their head and said “so you were the mysterious reviewer #2” – I hadn’t intended to reveal myself and to this day I don’t know what I said that tipped them but they were right and it was not a favorable review. This is exceedingly rare for this to happen (I’ve done many hundreds of reviews and received many hundreds of reviews at this point in my career and these are the only two I know of), but its not just kind and ethical but also smart to write your reviews as if they could be signed.
Brian, do you think the % of unsigned reviews that are straight is different from the % of signed reviews? I’d guess there’s still some game-playing going on with unsigned reviews.
Oh you’re right there are games either way. But at least with unsigned reviews they cut both ways if not more often trying to kill it. Signed review games will be almost always positive. And I think it is much harder to detect and control for positive games.
“But at least twice I had experiences where it was clear to me that the AE or EiC discounted my review because it was signed. ”
Interesting, it had never occurred to me that editors might react that way to me signing a review. As an editor, I never thought anything one way or the other about authors choosing to sign their reviews.
Ok, your experiences on this have nudged me to the point where I’m going to stop signing any reviews. Although I still think that in practice it probably doesn’t usually matter at all one way or the other whether you do or not.
Related to double-blind and single-blind: I recently had a case at AmNat where the reviewers signed their reviews. Since AmNat blinds the identity of the author(s), this means that the author knows who the reviewers were but the reviewers do not know the author (yet — the paper was accepted, so they will find out). I found that amusing.
Related to suggesting someone else’s work: I have done that, too, and worried that they would then get “blamed” for my review. I’m pretty sure I’ve added a parenthetical note to say that the review is not by that person.
Re: double blind review, just saw this on Twitter from an author experiencing it for the first time:
To which, yeah, if in order to give yourself a good shot at remaining anonymous you’d have to write the ms differently than you otherwise would (say, by not citing your own previous work, on which your current paper builds), then I think you just have to live with a high likelihood of being recognized. Well, that, or hope that the mere fact that your name isn’t on the title page is a sufficient reminder of reviewers’ obligation to treat you fairly. I don’t think you want to go too far down the road of not referring to previous work that it’s important to refer to in order for readers to understand and evaluate your paper.
“Once I got a tenure track job (not yet tenure)…”
I think you could sum up why a lot of early-career scientists do not sign their reviews with that statement alone. I reviewed as an MS and Ph.D student, and the imposter syndrome can be overwhelming when you get a paper to review from some well-known authors in your field of study. The pressure is usually great enough when you know the AE/EIC. Add in knowing the authors and I ended up asking myself, “Why would they even ask ME to review this?” Now this insecurity has grown smaller over the course of my career, but I’m positive I’m not the only one who thinks this way. My similarly-career aged colleagues mostly do too, with the exception of some supremely confident ones.
That being said, I’ve received signed reviews (positive) and I did end up contacting one reviewer for clarifications and a little help. He was awesome, and career-wise it actually worked out quite well as the interaction earned me another contact.
“I ended up asking myself, “Why would they even ask ME to review this?””
I’m sure you’re right that that feeling is a very common one, especially in grad students and postdocs asked to do their first review, and yes, especially if the paper is by someone famous.
Here’s what I once told a student of mine who received a review request and came to my office expressing this worry: if an editor asks you for a review, that *shows* that you are a good person to do the review. Because if you weren’t, you wouldn’t have been asked. Editor’s don’t pick reviewers out of a hat, they always have reasons for picking the reviewers they do. And editors are good at picking reviewers, as evidenced by the fact that the vast majority of reviews they receive are useful to them.
(Aside: I think this is actually a specific instance of a general principle: the person in the best position to judge if you’re ready to do X, where X is something you’ve never done before, often isn’t you. Often, it’s someone who’s done X before, or someone who has evaluated many other people on their readiness to do X.)
No idea if that line of thought will help you or anyone else who feels the same way, but it seemed to help my student. We also talked about the specifics of how to do a good review (
https://dynamicecology.wordpress.com/2013/01/03/advice-how-to-review-a-manuscript-for-a-journal/), which probably helped as well.
I’ve been thinking of writing a post on reviewing and imposter syndrome. I often feel super impostery when reviewing, and then end up feeling even more impostery if the other reviewer (or editor) brings up something I missed. The point I plan to make in the post (if I ever write it!) is that the reason we invite multiple reviewers is because different people will notice/question/be confused by different things. So, it’s not a bad thing if other people involved in the review process bring up things you didn’t or had a different opinion of the paper, even though for many people (myself included), it feels like a problem when that happens.
It is my opinion that the Hamletic to sign or not to sign should come from a journal’s policy and not be a reviewer’s decision. Journals requiring signed reviews should also make the reviews public in order to discourage old boys’ club effects or timid approaches by junior colleagues, problems that are far too frequent to be considered a small nuisance. I think the latter is the way to go forward.
I wrote in an older post how comments in a pdf file allowed me to identify the author of a particular mean (but that was not the problem), unsubstantiated (that was the problem) review. I was not looking for the perpetrator, I just hovered over a comment. The following week I was asked to review a manuscript of the same reviewer; knowing that I would have delivered a very negative review that had nothing to do with the manuscript, I decided to decline to review the paper. The example was not to highlight my moral virtues, but to support the “thorny-issue” view, as: 1) that review would not have been written the same way had the review being signed (plus); 2) this was an extreme case, but rejecting papers very often causes some – and sometimes a lot of – resentment in the rejected (minus); 3) for junior colleagues, writing an enthusiastic signed review might become the next-generation networking (minus); 4) I guess far fewer people would accept to review manuscripts if they had to sign the review (minus); 5) the quality of reviews would improve and the bottom 10% of reviews in terms of quality would probably disappear (plus).
A few years ago, I started signing reviews, but then quickly realized it was much more likely to sign reviews when positive. I stopped signing reviews.
I have never signed my reviews and I don’t intend to, for all the good reasons stated in the two blogs and comments. I once reviewed a paper that I thought was weak and the other reviewer signed their review raving about it (the journal in question allow reviewers to see each other’s comments). I so happens that I know the reviewer who signed their review and the PI in the paper and I also know they are buddies. It felt weird and now I can’t help but recall that incident every time I meet that colleague. The paper was extensively revised and later accepted.
On the double-blind issue, I think that it is best although some people argue that reviewers will guess authors anyway. I don’t think that is a valid argument because uncertainty is not the same thing as the fact that author’s name is written on the ms. Brian McGill (comment above) shared how it is easy to be wrong guessing, which happened to me too.
The most serious potential bias, that could use some blind author thing, is the hurdle imposed by Editors of high impact journals possibly penalising early career scientists. I don’t know how to fix that but it is in my opinion as relevant, if not more, than hiding author’s identity from reviewers.
Re: blinding reviewers to author identity, we have some old discussions on that in old linkfests. Short version is that there’s a research literature on the effects of that (and on reviewer anonymity), but it’s tough to learn much from it because most of the designs have obvious and often serious problems and most of the sample sizes are small. IIRC, the broad conclusions of that literature are that author blinding does often get seen through (though of course sometimes reviewers merely *think* they’ve seen through it), and that its effects are pretty idiosyncratic (e.g., sometimes blinding favors papers by authors from underrepresented groups, sometimes it actually *disfavors* such papers). At a guess, the idiosyncracy probably reflects some combination of blinding having a zero or small effect on average, and blinding getting seen through non-randomly with respect to author attributes. For instance, if reviewers tend to be slightly more negative when they’re blinded to author identity (and there’s some evidence that they are), and it’s mostly papers by famous authors that tend to have blinding seen through, and famous authors tend to be white guys, well, there’s a plausible mechanism by which blinding might tend to slightly favor white guys on average.
I think it’s worth doing further research on this. I’m a supporter of Am Nat’s experiment on blinding reviewers to author identity and look forward to hearing what they learn.
I also would like to see more journals go through the exercise that Functional Ecology and New Zealand J. of Ecology went through to show that their peer review outcomes are gender-neutral, even though their review process does not blind reviewers to author identities: https://dynamicecology.wordpress.com/2015/11/18/gender-and-peer-review-at-functional-ecology/ and https://dynamicecology.wordpress.com/2015/11/18/gender-and-peer-review-at-functional-ecology/#comment-45831 It’s somewhat time consuming to wrangle those data from online ms handling systems, but it can be done and I think it’s worth doing. I’d rather see more journals first go to the trouble Functional Ecology and NZ J Ecol went to to check if there’s a problem that needs solving in their own peer review process rather than see journals all just switch to a double-blind review process on the grounds of “what harm could it do?” Because my reading of the existing, admittedly-limited and admittedly-flawed literature is that it’s about as likely to do (small) harm as it is to be of (small) help. I think if you can show that your existing processes are fair, as FE’s and NZJE’s seem to be, you should stick with them.
With the increasing acceptance of – and enthusiasm for – preprint posting by disciplines/journals outside of physics (eg in the biology preprint server bioRxiv), double blinding is going to become more difficult (unrealistic?) to achieve because versions of a manuscript can be found and authors identified.
Good write-up of a recent preprint workshop by Katherine Brown, Executive Editor of the journal Development at the Company of Biologists, with stats & links http://thenode.biologists.com/preprints-biomedical-science-publication-era-twitter-facebook/events/
Yes good point. In an old link fest post (sorry can’t find it now) I linked to someone who made the same argument: double blind review and widespread use of preprint servers can’t coexist.
As I said in response to Steve’s post, I am a non-signer, because of the unconscious obligation factor. We are only human, a good signed review will inevitably incline you to be kind(er) in return if asked to review a paper written by them, and vice-versa if you get a less favourable review.
I always sign my reviews. I do it primarily so that the authors can get in touch if they have questions, and also understand where any biases in my review might be coming from.
I like to think I write constructively and critically regardless of whether the review is signed, but do think that knowing my name goes out with each helps me with this process.
To my knowledge I have never seen any negative consequences arise from doing things this way.
Interesting point. I would think that the authors shouldn’t contact the reviewer directly even if the review is signed. Shouldn’t it all be done through the Editor so that there is a nice clean official reccord of exchanges. Ideally, it would be a lot easier to just contact the reviewer, and perhaps going through the editor discourages asking questions in a way that creates worse papers. I’m not so sure what I think about this.
Everett Fee, the former EIC at Limnology and Oceanography (L&O) put it to me best: he discouraged signing because with signed reviews, authors sometimes focus more on the qualifications of the reviewer to comment on their work than on the substance of the criticism.
There are a lot of AEs who read this blog. AEs often effectively give signed reviews, some of which are very critical. Anyone wonder whether the recipient of a signed negative review attached to a negative decision might remember if that person later reviewed a manuscript from the AE? I’ve wondered, though I don’t know if it’s actually happened. Still, the society that publishes the journal for which I serve has a rank or caste structure to it, and I’m nowhere near the top. I’ve had complaints to the EIC from Dr. BigName because I rejected their or their advisee’s paper, and they gave the EIC an earful about assigning it to an editor who clearly didn’t understand the science, had bad judgement, and once, even wrote that I was also a very bad person on top of my intellectual defects. I even heard about it from Dr. HigherUp at my own institution who happened to be close with Dr. BigName, and it had somehow come up in conversation. I found myself thinking, why on earth be an AE and make myself a target for these big egos? This has only happened twice out of about 200 manuscripts I’ve handled so far, but it left a deep impression. Why invite this with signed reviews as well? If the journal is set up for blind reviews, I do not recommend reviewers sign reviewers. I don’t strip it off when they do, but I really question it.
Interesting dueling perspectives and posts. I’ll vote by responding to one side of the argument. Thanks to both for taking time to kick off a good discussion.
You raise an important issue with AEs taking crap from authors. That’s why I think it’s very important to AEs to have a strong EiC who has their backs, so that AEs can feel confident in putting their names on all their correspondence with authors. Because the alternative (well, one alternative) is that AEs remain anonymous, perhaps unless and until the paper is accepted. Which I don’t like at all. AEs are the gatekeepers, they’re the ones with the power. They shouldn’t be anonymous, even if this sometimes exposes them to crap from authors: https://dynamicecology.wordpress.com/2014/10/06/should-journal-editors-be-anonymous/
AEs shouldn’t be anonymous for the same reason NSF program officers shouldn’t be anonymous, as Josh King points out in the comments on that linked post.
Another reason I don’t sign my reviews is that I want to feel free to ask the authors “stupid questions.” Most of the time, when I say I didn’t understand method X, definition Y or interpretation Z, it is a good comment that the authors use to imrpove the paper (the editor or other reviewer often have similar concerns). However, if hotshot professor in my field writes something I don’t understand, I might not want to point out that I don’t get it, in fear that it is me making the stupid mistake. This is different than fearing retaliation. Basically blind reviews help to prevent imposter syndrome from getting in the way of reviewers making good comments. This is very important given that many reviewers are ECRs. I have an say, getting the email with afirmation from an editor or second reviewer is perhaps the best medicine I’ve ever had for curing imposter syndrome.
Yes, the single thing that makes me most confident in my abilities as a reviewer is how often other reviewers and/or the editor say the same things I did.
I know you’ve seen this Matthew, but for any readers who haven’t, here’s an old post on how great it is that journals let reviewers see the other reviews: https://dynamicecology.wordpress.com/2016/05/19/i-love-that-journals-let-reviewers-see-the-other-reviews/
Re: the various subthreads on anonymity and openness in peer review, just found (again!) that comprehensive review of the literature that I was trying to recall upthread:
http://blogs.plos.org/absolutely-maybe/2015/05/13/weighing-up-anonymity-and-openness-in-publication-peer-review/
I think one thing missing from this conversation is the changing perspective as one publishes more and more (and reviews more and more).
In the story I told above where the name of one of the reviewers got revealed to me by accident, it was my 2nd paper ever, and I hard the hardest time looking the reviewer in the eye for the next year even though they were a generally friendly member of my own field. Now dozens of papers and hundreds of reviews later, I’ve been rejected so many times I have little to no issue when I find out somebody has recommended reject on a paper (especially if I perceive it as fair which I usually do). Odds are I have recommended reject on one of their papers in the past. At this point I have recommended both publish and reject on papers by most people in my field. Its the way of the world. Its part of academia. I think most other people move that direction as well. But it took me 5-10 years to get there.
I still don’t sign reviews for the reasons I gave above – basically it just seems more “pure” to me. But it is no longer a big deal when somebody does sign a negative review. Sometimes a year or two later I can even see that they were right. And if they were wrong, well I’m not perfect either and nobody should kid themselves that there is not a fraction (not the largest fraction but still a fraction) of the editorial process that is just luck.
I never sign reviews, and I don’t see that happening any time soon. The main reason? Bias against me. I’ve noticed that older scientists (men and women), and men in general (not all, of course), tend to dismiss or minimize the criticism from young women (“she doesn’t know what she’s talking about”), and I want my comments to be judged on content, not my sex, age or how far along I am in my career.
I always make my reviews as constructive as possible, and I am never rude or pushy about my comments. So it’s not fear of retaliation, it’s fear of being dismissed because of bias.
On the other side, I’ve had many issues with reviewers criticizing my paper based on who I am and where I am from, instead of content (“she is not from this country so how can she even do research about it”). Worst part? I AM from that country. Apparently googling is hard.
That’s a very good point that some people, if they sign their reviews, risk not being taken seriously based on their gender, career stage, or some other aspect of their identity. Thanks for raising it! (Of course, I wish it wasn’t an issue, but you’re right that it will be for some people.) It links with Chris Mebane’s story up above about why Everett Fee didn’t want people signing reviews.
I think there are benefits to an open review process that relate to improving a project. Putting all communication through a journal’s editorial office where the stakes are high and opportunities are low (take your best guess at what the reviewer/editor want, then resubmit and hope) does not seem best to me. Making it possible for authors and reviewers to have a more in depth conversation about ways to improve work would be useful. This has happened several times for me where I have been on either side of the manuscript, and I wish it would happen more.
I think the question of judgment of work around a reviewer’s identity is an important one – but not necessarily one we should get around by anonymizing all feedback. This effectively suggests that the context in which people are working and their background is not an important part of understanding their thinking. I think the burden is actually on the editors to create a review process where authors can and should respect feedback from whichever people the editors feel it is useful to obtain it. As Mariela suggests above, being a reviewer often feels like a precarious position to be in, especially in early career stages or when not from a dominant group – the editor can help us feel more confident in ensuring feedback is taken seriously.
I don’t ever sign reviews. But, I always sign as an AE, even though the majority of letters are negative.
As a reviewer, my name was inadvertently given to an author by an Editor. He (the editor) had recently started and was not familiar with the software. I got an email from the author thanking me for my review. I was horrified because the review was rather negative and I knew that I had not signed the review. During the revision process, I was assigned the same paper and I felt like I had to sign since the author now knew it was me. The second review was also negative so I had now cemented a bad taste in the mouth of the author. I don’t think I ever received another manuscript to review from that individual.
With respect to double blind reviews, I had a rather embarrassing incident. I reviewed a manuscript that my own postdoc was an author on. Had I known she was an author, I would not have accepted the review assignment. Here is an example where the double blind process actually led to a conflict, rather than avoiding a conflict of interest. Some have argued that the process worked in that I conducted an unbiased review, not knowing there was a potential conflict. But, even so, I was not happy when I found out the news well after the fact.
I have had two other double blind review requests where I declined the assignment. In both cases, it was a colleague who works at the same field site as me. The description of the study made it obvious in both cases so I recused myself.
“With respect to double blind reviews, I had a rather embarrassing incident. I reviewed a manuscript that my own postdoc was an author on. Had I known she was an author, I would not have accepted the review assignment. Here is an example where the double blind process actually led to a conflict, rather than avoiding a conflict of interest.”
Yeah, it’s incumbent upon editorial offices implementing double-blind review to be even more alert to potential conflicts of interest than they ordinarily are. I agree with you that it is not appropriate for people to be reviewing papers from authors with whom they have a conflict of interest even if blinded to author identity.
One problem I’ve run into as an AE is when I can’t tell if someone is collaborating with someone. I try to search webpages to figure it out, but those often aren’t super up-to-date. In those cases, I skip over that person as a reviewer, but that’s unfortunate because then it means, if they aren’t collaborators, I’m missing out on someone who would be a great reviewer.
Just in are you weren’t aware, this post was featured in a sciencegeist.com email 🙂
Thanks! I wasn’t aware of that site but, looking at it just now, it looks like an interesting aggregator.
I always sign. I have found that a) it has made me rephrase some things that I wrote in haste (and forces me to make all of my reviews as constructive as possible) and b) I really really want the authors to engage with me if they have questions or want clarifications. And I’ve had this happen a half a dozen times. It’s made the resulting manuscripts even better than they could have been initially. It’s this last bit, and the research – not much but some – showing that communication between authors and reviewers improves manuscripts that has convinced me, in the name of science, to always sign my reviews. I realize I do this from a position of privilege, but, if I can, I will.
“b) I really really want the authors to engage with me if they have questions or want clarifications. And I’ve had this happen a half a dozen times.”
Between you, Stephen Heard, and a couple of the commenters on his post on this, I’m now getting weirded out. As I said over there, it has never occurred to me to engage with a reviewer outside what I’ve always thought of as the usual channel (the response to reviewers that accompanies the revised ms). And over the years I’ve signed some reviews and never had an author contact me about them. Heck, until this week I’d never heard of an author contacting a reviewer to talk about the review (ok, except at the rare journals where that’s the official, formal review process.)
I think I’m going to do a poll on this. Because now I really need to know how common this is.
Also, Jarrett, since you’re here: what are your thoughts on whether signing one’s review creates at least the potential for impropriety? And that going further and communicating informally with the author who received your signed review strengthens that potential? Or do you think that’s no more of a issue than people reviewing papers by authors whom they’ve met and interacted with but with whom they have no conflict of interest (which is something that happens all the time)?
I mean, I love to weird you out, so, yay! I actually have a canned statement I put at all of the end of my reviews encouraging the authors to contact me. As often my comments are about nitty-gritty-nerdy analysis stuff, this has been particularly helpful a few times. It reads:
“Please note, I sign all of my reviews in the hopes that, if the authors wish, we can engage in a constructive dialogue about their work. I am more than happy to talk further about any issues raised in my reviews, and hope that they are helpful for the further development of any authors’ works.”
As for impropriety, that’s a great question. I don’t think it creates any new COIs. All communication centers around the paper, and my goal – and I think anyone’s goal – as a reviewer is to strengthen a paper. If anything, it lays all potentially perceived COIs on the table in a more fundamental way than anonymous review.
Then again, I also see the point that I am a white dude who is now an Asst. Prof. Were I not all of those things, I can see how it could potentially cause some authors to read my review in a less helpful way. My hope would be that it would be for the editor to adjudicate that, but, we are all human, so I don’t have full faith there. As I said, because I am those things, I know that my review will be valued (I hope?), and so I try and use that for good here. I completely understand that others might feel (and might be right) that choice is not the right one to achieve the same goal for them.
I suspect that canned statement may be why you get contacted by authors as often as you do. I suspect that anyone who signs their reviews in the *unspoken* hope that authors might contact them will be contacted only very rarely if ever.
Poll on this in the queue for Monday.
Pingback: Poll: have you ever contacted a reviewer about a review before responding to it? Or received such contact? | Dynamic Ecology
Pingback: To sign or not to sign: what the Replies taught me | Scientist Sees Squirrel
About a year ago I decided to sign all my reviews. And my experience has been entirely positive. Almost all authors got back to me saying that it was a positive thing to be transparent (probably because I usually also email the authors my review in parallel to submitting it to the journal). As a reviewer, I see my role more as an editor, not as a gatekeeper. I try to be critical but also constructive and I think the majority of scientists acknowledges this.
Pingback: Como revisar um artigo para uma revista | Sobrevivendo na Ciência
Pingback: Should figures go at the end of a manuscript or appear in line? And where should figure legends go? | Dynamic Ecology
As a main rule, I don’t sign reviews, but I make an exception, if I feel that it would be unfair not to disclose who I am, like in cases where I use my “power” as a reviewer in a way which might benefit myself. That has happened once so far: I suggested that they should cite my own paper.
Sorry for adding to this already long thread of comments, but it seems to me that one of the most convincing arguments for signing your reviews is entirely missing from the conversation. And that is the fact that signing your reviews allows you to take credit for the work you do as a reviewer.
In a job application today, you may point out that you are reviewing for journals X, Y, Z. But if you can actually list your signed reviews, they effectively become mini-publications that allow others to judge your knowledge of the field as well as your ability to provide constructive criticism on your colleagues’ work.
I do appreciate the valid arguments against signing reviews put forward here but I believe that this benefit of signing reviews needs to be part of the equation.
In my view, that the lack of incentives (besides moral obligation) to take the time to write high-quality reviews is a big problem and allowing reviewers to take credit for their work can be one way of alleviating it.
You can actually get a validated list of review-assignments, and publish reviews (if allowed by the journal) from Publons, without directly signing the reviews. It is really convenient and I strongly recommend it.
Wait, are you talking about the (very small at most) reputation boost you get from the authors of the papers that you reviewed? Or are you saying that signed reviews can be a valuable part of your job applications? Because if it’s the latter, I very much disagree. Nobody who evaluates job applications is going to treat your reviews as “mini-publications”. Nobody who evaluates job applications cares a bit about the quality of your reviews or treats them as evidence of your knowledge of the field or ability to provide constructive criticism or whatever.
Re: the reputation boost you might get with authors from signing your review, that depends entirely on whether the authors liked your review. If they didn’t, you get the opposite of a reputation boost in their minds.
I meant “the latter”. Perhaps I should have clarified that I was referring to applications for jobs in academia.
I am not in a position to argue with your opinion on how useful or useless a record of your reviews would be in such an application. All I can say is that if I were looking to recruit a postdoc, I would definitely check it out. And I don’t seem to be alone with the opinion that high-quality signed reviews can contribute to building your reputation as a scientist (see e.g. https://doi.org/10.3389/fncom.2012.00094).
Pingback: Revisando artigos – Mais Um Blog de Ecologia e Estatística
Pingback: My goal as a reviewer: pass the Poulin test | Dynamic Ecology