Here’s something I often struggle with when writing my papers: should I anticipate and address potential criticisms? And if so, where–the methods, the discussion, or the online supplements?
In general, your paper should explain, motivate, and justify what you did, not merely describe what you did. That often means explaining why you didn’t do things some other way. On the other hand, you don’t want your paper to sound defensive. “The lady doth protest too much” and all that. Nor do you want to interrupt the logical narrative flow of your paper by raising and addressing possible objections.*
I struggle with this in part because I work in a system–microcosms–to which some ecologists have blanket objections. These objections are groundless**, but they crop up sufficiently often that I can anticipate them. And if you can anticipate how readers are likely to object to (or misunderstand) your work, shouldn’t you head off those objections and misunderstandings? If for no other reason than to prevent your paper from being rejected? Worst case scenario, the reviewers tell you to cut those passages as unnecessary.
If possible, frame your defense of your approach positively rather than negatively. For instance, I’m currently writing a paper on a microcosm experiment from which, unusually, we only collected presence/absence data rather than abundance data. I could defend this choice by explaining how presence/absence data are adequate for our purposes, even though they contain less information than abundance data. But I plan to use a more positive framing: collecting presence/absence data allowed us to run a larger experiment, with the full range of treatments needed to rigorously test our hypothesis. Both framings are equally correct, but the latter will sound much better to reviewers and readers (I hope!)
I also struggle with whether to justify my approach in the methods or discussion. Some people routinely include a subsection near the end of their discussion called “caveats” or something like that, in which they discuss the limitations of their approach. But others do it in the methods. And these days many people bury discussion of alternative approaches in the online supplementary material. In particular, alternative ways of doing one’s statistical analyses commonly get relegated to online supplements. And I routinely use online supplements myself when I’m writing about the Price equation, to address common misunderstandings of, and objections to, the approach. But I don’t think the trend towards increasingly-voluminous online supplements is a good thing on balance (see here, here, and here). Plus, it’s not always obvious what to put in your online supplements.
Of course, one way to avoid the entire issue is just to cite a paper that addresses any potential objections to your approach. I think this works best if there’s a recent review paper or other “standard reference” you can just refer readers to. Which is a problem, since you’re most likely to run into objections when you’re doing something new or otherwise non-standard. If it was already well-established that your approach was The Right Way, you wouldn’t need to justify it. More subtly, as a reviewer I tend to be a bit suspicious of authors who justify non-obvious methods this way, because sometimes I follow up their citations and find that those citations don’t actually address my objections (or sometimes, even say what the citing authors claimed!)
The other reason I struggle with this issue is perhaps specific to me. When reading a paper, I often find myself wondering about how the same broad topic could’ve been addressed in some totally different way. For instance, if you tell me that you used local-regional richness relationships or a phylogenetic approach to try to infer if interspecific competition affects local species composition, my first question will be “Why didn’t you just do the sort of manipulative experiment Shurin (2000) did to answer that question directly, rather than making an inference based on a bunch of dodgy assumptions?” As another example, if you tell me that approach X wasn’t tractable in your system, my first question will be “Why didn’t you work in a model system instead, so that you could use approach X?” Not because I think there’s a single “right” or “best” way to address any given question–I don’t. But just because I attach a lot of importance to having good reasons for what one does, and to spelling those reasons out (see here and here). That’s the way my mind works, and so I tend to assume that other people’s minds work the same way. Perhaps causing me to worry more than I should about justifying my own methodological choices.
Do you try to anticipate and address potential objections to your work? If so, how do you do it?
*Unless of course the narrative for your paper is something like “here’s a problem that other approaches either have, or cannot solve, that my approach solves.”
**The next even semi-defensible blanket objection to microcosms that I encounter will be the first.