As scientists, we often judge research on two criteria: how good was the question (interesting? important? etc.), and how convincing was the answer?
But often, other criteria creep in. For instance (and this is just one example), the cleverness, elegance, or creativity of the methods.
We all have our favorite examples. Meghan highlighted Jasmine Crumsey and others before her who’ve used medical CT scanners to reconstruct earthworm burrow systems. As Meghan wrote, how cool is that? 🙂 She’s also highlighted the amusing example of using vibrators to mimic buzz pollination. I like the recent example of researchers who figured out that dung beetles navigate by the Milky Way by putting dung beetles in a planetarium. It strikes me as not just an effective test of the hypothesis, but a very creative test. It never would’ve occurred to me to do that! And when I talk to undergrads about my research to recruit them to work in my lab, they’re often really intrigued by the idea of growing protists in jars to test general ecological principles. “How did anyone ever think of doing that?! That’s really neat! ” is a common reaction.
There’s usually no harm, and plenty of enjoyment, in appreciating the cleverness, elegance, and creativity of somebody’s methods–as long as it doesn’t color our judgements about their effectiveness.
For instance, in economics instrumental variables have become a standard method for inferring causality from observational data. The basic idea of instrumental variables is to exploit what ecologists would call “natural experiments”: naturally occurring exogenous variation in some driver variable that perturbs the causal relationship of interest. The method of instrumental variables is popular approach in part because it’s seen as rigorous–but also in part because it’s seen as clever. I only know what I read on economics blogs, so take what I’m about to say with a whole salt shaker’s worth of salt. But anecdotally it’s common to see economists praised for particularly ingenious choices of instrumental variable. Using the Colombian colonial royal road network to estimate the causal effect of government on economic development, for instance. How did anyone ever think of doing that? That’s really clever! But an idea doesn’t necessarily work just because it’s clever. In fact, empirical economics is having a mini-crisis right now because of a major review paper showing quite convincingly that, in practice, instrumental variables in economics are usually worse than less clever, less “rigorous” approaches. It looks to me like many economists cared too much about cleverness of instrumental variable choice, and not enough about the quality of the resulting inferences.
What do you think? How should we value the creativity, elegance, or cleverness of a paper’s methods, particularly in relation to other desiderata?