Choosing reviewers, recognition not recall, and why lists like DiversifyEEB are useful

Last week, I was assigned a paper to handle as an Associate Editor at American Naturalist. After reading through the paper and deciding it should go out for review, I began the task of finding potential reviewers. There were two people who immediately stood out to me as qualified reviewers. But AmNat likes to have a list of six potential reviewers to work from, so I continued through my standard process: 1) try to think of another person; 2) struggle with that; 3) think “surely I can write the editorial office folks and tell them we should just go with these two I already thought of because they’re perfect”; 4) decide I need to try harder before giving up; 5) after some more effort, end up at a list of six (or, in this case, seven) people who would be good reviewers, ranked in the order in which I’d like them to be asked. After going through that process, the two people I originally thought of were still on my list, but they were numbers 4&5. For some other papers, the initial people I thought of didn’t end up on my final list at all. And I have never had the first two people I thought of end up being the first two people on my list of six.

In other words: there were really good options who I only thought of after working at it for a while; those people were better options for this task than the people who initially occurred to me. To me, this is striking, but not really surprising. It’s what motivated the DiversifyEEB list that I created with my colleague, Gina Baucom. We all have biases, and those make it so that the people we think of first aren’t necessarily the best ones. And, moreover, our biases make it so that we’re more likely to think of well-known white men. That’s just how our brains work.

As I thought through this on my walk home, it reminded me of a story Kay Gross told me shortly after DiversifyEEB launched. Kay said that, many years back, she had a conversation with Margaret Davis. What Margaret told Kay is that, when she got a phone call asking her to recommend people for something, she would say, “Let me think about that and get back to you”. She did this because she had noticed that the first set of names she thought of were always men. But, if she thought it over more, she came up with more names and more diverse names. I found it especially interesting to learn that Margaret Davis had created a set of cards, adding a new card whenever she met an interesting woman scientist; during the time between getting the call and getting back to the person, she consulted that set of cards (her own personal DiversifyEEB list!) to think through people who were well-suited but who didn’t initially occur to her.

Back to the specific topic of finding reviewers: Charles Fox and colleagues have done a set of really interesting studies related to gender and the publication process. In one, they found that just 25% of the reviewers suggested by authors were women. In another, they found that only ~27% of the reviewers invited by associate editors were women. I initially thought that perhaps one solution to the problem of lack of gender diversity in reviewers would be to have more journals ask for lists of 6 potential reviewers — perhaps thinking longer about who should review something would increase the diversity of who they think of? But it turns out that Functional Ecology already asks their AEs to come up with 6 potential reviewers, so clearly that, on its own, will not solve the gender balance problem.

After more reflection, perhaps it makes sense that the lists are still pretty biased, even if they have more people on them: these potential reviewer lists still rely a lot on recall (that is, who I think of as I think about a particular topic), not recognition (that is, choosing from a list of names that might be suitable). And the original motivation for DiversifyEEB was learning (from Joan Strassmann) about psych research showing that the best way to come up with more diverse groups is to rely on recognition, not recall. (If you remember nothing else from this post, remember “recognition, not recall” as a strategy for increasing diversity!)

So, if you are an associate editor for a journal (or, really, in any other position where you are trying to come up with a list of scientists for something):

1. It’s worth the effort to try to come up with a longer list. In that process, you are likely to think of people who are better options. This will lead to better reviews (or a better seminar series or candidate list or whatever it is you’re trying to do.)
2. Once you have your list, consider the diversity of it. Does it include diversity in terms of race, gender, career stage, and institution type (including non-academic ones)? In some cases, your list might intentionally be lacking a form of diversity (e.g., a candidate list for an endowed chair probably won’t include many early career folks). But, in most cases, a lack of diversity will reflect our inherent biases. (We all have them! The key is to recognize them and work to counter them.)
3. If your list seems to be lacking in diversity, try to find lists that will give you more ideas. DiversifyEEB is one, but you can also look other places (e.g., if you are trying to think of Darwin Day speakers, a scan of the editorial boards of journals like Evolution, AmNat, J. Evolutionary Biology, etc. might give you ideas). Another great strategy, especially for looking for reviewers, is to go to the webpage of the person you first thought of and look at their grad students & postdocs. This includes looking at recent grads who have moved on to other positions.

As I said above, the key is being aware of the biases we have, recognizing when outcomes indicate biases are at work, and working to counter them. Lists like DiversifyEEB are one way to try to do that, and I love knowing that Margaret Davis had created her own version of a DiversifyEEB list long ago! I’d love to hear from readers about what strategies you use to try to increase diversity when coming up with potential reviewers, seminar speakers, etc!