Ask us anything: reviewing papers for small journals

A while back we invited you to ask us anything. Here’s our answer to our next question, from Richard Feldman. The question has been paraphrased for brevity, click through for the original.

How do you fairly review a paper for a smaller or more local journal? Does one just look at recent issues and say ‘oh, that’s how it’s done at this journal’, or does one push the authors to be more rigorous and correct mistakes?

Jeremy: I have some old advice on how to review a paper that’s broadly relevant here.

I think you adjust your review to the journal with regards to judgment calls like fit, impact, importance, and interest. Looking at recent issues of the journal can indeed help with that. I don’t think you ever let clear-cut technical mistakes slide. You shouldn’t ever recommend statistical machismo, though (as Richard himself noted in his original question).

Semi-relatedly, I don’t think you adjust your review based on the author. I don’t review papers any differently when the lead author is a student, even an undergrad. Anyone who submits to a professional journal is agreeing to have their work evaluated by professionals, according to professional standards.

If you’re unsure by what standards to judge the ms, explain where you’re coming from in you confidential comments to the editor. Editors always welcome that sort of context, it aids their decision-making.

But I may not be the best person to answer this question, because it’s never come up for me. I’ve never done a review for a small local journal.

7 thoughts on “Ask us anything: reviewing papers for small journals

  1. I don’t really adjust much my reviewing. I’d expect anything published in scientific journal to be set in sensible context, methodologically sound and well-discussed. Fit, importance and interest are then probably the things which make difference in publication decision between “small” and “bigger” journals, but I’ve always felt uncomfortable to assess them no matter the perceived quality or scope of journal.

  2. If “smaller” means “more specialized” then my experience is that the reviews I’ve received have often been more thorough in the small journal because the reviewers are more likely to be experts on the species or habitat being studied. If you submit a paper on, say, spider predation in deserts to Ecology, the reviewers are likely to be a general experts on predation and a general expert on desert ecology. Submit the same paper to a journal specializing in spider ecology, and you’ll get two reviewers who are experts on desert spiders, with one probably working on the same genus of spiders. They may not comment as much as the Ecology reviewers would have about general predation theory and general experimental design, but they will have question after question about the spiders: What sex did you use and why? How old and why? The brown morph or the sandy morph and why? Why didn’t you cite these 3 other papers on the genus published in some obscure journal? Is it possible that this behavior that only I have observed could explain your results? Given fact X about the study species, wouldn’t it have made more sense to do the experiment this other way? In short, they will go to great lengths to interrogate the natural history and whether the study design made sense given the natural history. Reviewers at Ecology are likely to overlook 90% of that because in many cases they won’t have the same level of natural history knowledge.

    In the same way, reviewers at a journal with a local focus will expect a high level of accuracy about the local ecosystems that reviewers at an international journal will not be competent to address.

    Bottom line: This doesn’t answer the question of what SHOULD reviewers do, but my experience is that reviewers at big versus small journals often focus their reviews on different issues.

    • This all sounds right to me.

      As to whether reviewers at leading general ecology journals *should* emphasize natural historical detail more, or whether reviewers at specialized taxon-specific journals *should* emphasize it less, I think it totally depends. Like anything, it should be emphasized when it matters and not when it doesn’t.

      To use an example from my own protist microcosm work: when I used protists as a model system to test resource competition (R*) theory (Fox 2002 Am Nat), I had to know something about the natural history of the candidate species to decide which ones to use in the experiment. Using a species that’s capable of inducibly shifting to a predatory macrophage (e.g., Blepharisma sp., Tetrahymena vorax) would’ve been totally inappropriate–you don’t want predatory species in a competition experiment! And using Tillina sp. would’ve been inappropriate too, because it encysts at the drop of a hat and so you can’t easily tell the difference between “Tillina got competitively excluded” and “all the Tillina encysted”. So, hypothetically, if I’d been silly enough to use Tetrahymena vorax or Tillina sp. in Fox 2002, I hope a referee who knew about the natural history of protists would’ve called me out on that even though I’d submitted to Am Nat rather than Journal of Eukaryotic Microbiology!

      But on the other hand, there are aspects of natural history that are irrelevant for the work I’ve done. Fox and Harder 2015 Evolution used lake bacteria as a model system to look for local adaptation to temporal as well as spatial environmental variation. To get Fox and Harder 2015 published, we had to get past a referee who insisted we ID the bacteria. The species identities of our bacterial isolates were totally irrelevant given the questions we asked (it didn’t even matter if they were all the same species), but we did it anyway because the referee insisted.

      Conversely, I wouldn’t be surprised if reviewers at specialized taxon- and system-specific journals are less likely than reviewers for leading general ecology journals to catch theoretical, conceptual, and statistical errors. Anecdotally, more than once I’ve happened across papers in taxon-specific journals that infer process from pattern in naive, old-fashioned ways that quite rightly would never fly at a leading general ecology journal. Or think back to Gould and Lewontin’s “Spandrels of San Marco”. Their examples of unsupported adaptationist “just so stories” didn’t come from leading evolutionary biology journals, but rather from specialized taxon-specific journals. Just anecdotes, though, I have no idea of the rate of theoretical, conceptual, or statistical mistakes in specialized taxon- or system-specific journals.

      Meg’s talked about the importance of knowing one’s own study system in various old posts:

      My experience is that ecologists publishing in leading general ecology journals on data they’ve collected themselves do know the natural history of their own study systems extremely well, and rarely or never make important technical mistakes that could be chalked up to their lack of natural history knowledge. In my experience, those sorts of mistakes only happen when somebody is just starting to learn a new system.

      These days, the concern you raise seems to get brought up most often in the context of ecologists running comparative/synthetic analyses of data collected by others, that were originally collected for some other purpose (example: How often do those comparative/synthetic analyses make *important* mistakes (as opposed to minor ones that don’t alter the conclusions) that wouldn’t have happened had the investigators talked more with people who really knew the original data and the systems those data came from? I have no idea.


  3. I review for both bigger and local journals. I review the papers in a similar way, but occasionally adjust the bar for recommendations whether to accept or reject depending on the journal. E.g. less novel or less well replicated papers probably deserve to be published in local journals even if their impact isn’t great enough to justify publication in a high impact international journal.

  4. Small “local” or “taxonomic” journals are often the types of papers used by others when (1) authors of novel/global/high impact papers pull parameters from the literature to test their theory and (2) these papers can form the basis for policy on important local and/or species specific conservation issues. It is absolutely essential that these papers are held to high standards. Their importance is often underplayed in ecology.

    • Well, held to high technical and reporting standards, certainly. But papers that are novel/global/high impact *collectively* might well be none of those things *individually*. Which isn’t a criticism of those papers or people who write them, it’s just the way it is. So I still take the view that as a reviewer you scale expectations of novelty/impact/importance relative to the journal.

      • Absolutely. The comment above was meant to highlight the importance of these papers being accurate [and held to high technical standards – not novelty/impact standards], since they are often the building blocks for regional and species specific policy decisions and also future high impact papers. I was echoing your position, but pointing out this added reason why its important to point out technical errors in all reviews regardless of journal.

Leave a Comment

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.