Should we care about the cleverness, elegance, or creativity of a paper’s methods?

As scientists, we often judge research on two criteria: how good was the question (interesting? important? etc.), and how convincing was the answer?

But often, other criteria creep in. For instance (and this is just one example), the cleverness, elegance, or creativity of the methods.

We all have our favorite examples. Meghan highlighted Jasmine Crumsey and others before her who’ve used medical CT scanners to reconstruct earthworm burrow systems. As Meghan wrote, how cool is that? 🙂 She’s also highlighted the amusing example of using vibrators to mimic buzz pollination. I like the recent example of researchers who figured out that dung beetles navigate by the Milky Way by putting dung beetles in a planetarium. It strikes me as not just an effective test of the hypothesis, but a very creative test. It never would’ve occurred to me to do that! And when I talk to undergrads about my research to recruit them to work in my lab, they’re often really intrigued by the idea of growing protists in jars to test general ecological principles. “How did anyone ever think of doing that?! That’s really neat! ” is a common reaction.

There’s usually no harm, and plenty of enjoyment, in appreciating the cleverness, elegance, and creativity of somebody’s methods–as long as it doesn’t color our judgements about their effectiveness.

For instance, in economics instrumental variables have become a standard method for inferring causality from observational data. The basic idea of instrumental variables is to exploit what ecologists would call “natural experiments”: naturally occurring exogenous variation in some driver variable that perturbs the causal relationship of interest. The method of instrumental variables is popular approach in part because it’s seen as rigorous–but also in part because it’s seen as clever. I only know what I read on economics blogs, so take what I’m about to say with a whole salt shaker’s worth of salt. But anecdotally it’s common to see economists praised for particularly ingenious choices of instrumental variable. Using the Colombian colonial royal road network to estimate the causal effect of government on economic development, for instance. How did anyone ever think of doing that? That’s really clever! But an idea doesn’t necessarily work just because it’s clever. In fact, empirical economics is having a mini-crisis right now because of a major review paper showing quite convincingly that, in practice, instrumental variables in economics are usually worse than less clever, less “rigorous” approaches. It looks to me like many economists cared too much about cleverness of instrumental variable choice, and not enough about the quality of the resulting inferences.

What do you think? How should we value the creativity, elegance, or cleverness of a paper’s methods, particularly in relation to other desiderata?

21 thoughts on “Should we care about the cleverness, elegance, or creativity of a paper’s methods?

  1. I always evaluate methods on their ability to provide data that address the question or test the hypothesis well. One of the key characteristics I look for is how well the things that were measured capture the underlying concept and sometimes it does take a lot of creativity to figure out what you can measure that is a good index of the concept you care about. But when the connection between the thing(s) you measure and the concept is so non-intuitive that it takes an extremely creative mind to see it, I would worry about how close that connection is. So, it’s clever to think of using the development of road networks as an index of how governments affect economic development but is the connection between the thing you measure (road networks) and the concept (effect of government on economic development) a tight one?
    I value creativity and cleverness in methods only to the extent that they result in data that will allow us to answer the question/test the hypothesis better than the ‘uncreative’ methods did.


    • “I value creativity and cleverness in methods only to the extent that they result in data that will allow us to answer the question/test the hypothesis better than the ‘uncreative’ methods did.”

      Same. In the post I talk about creative methods that also work. But I also can think of examples of creative methods that don’t work, and I’m not impressed with their creativity.

      Which raises the question of whether there is ever such a thing as a “noble failure” in science. A creative or clever method or approach that we admire for its cleverness or creativity even though it didn’t work.

    • A further thought: often, when I’m impressed with the cleverness of a paper’s methods, it’s because it’s a clever way to *directly* measure or test whatever concept we’d like to measure or test. For instance, want to know if dung beetles navigate by the Milky Way? Put them in a planetarium and turn off everything in the night sky but the Milky Way (and also do appropriate controls like turning off the Milky Way too, of course). When reading ecology papers, I’m rarely impressed (or convinced) by “creativity” in the form of long, convoluted chains of logic connecting what was actually done or measured to the concept being quantified or tested.

    • “One of the key characteristics I look for is how well the things that were measured capture the underlying concept”

      Right on, Jeff! This is so important and underappreciated. Choosing quantitative representatives of ‘linguistic’ concepts, and properly handling the analysis relating them is a current soft-spot in our field (IMHO). The translation from verbal model to quantitative model should be one of the first things we teach in any quantitative methods class.

      • This gets back to a couple of old posts here:

        I wish I could say those posts had been clarifying for me, but they weren’t. The discussions were good, but yet I didn’t come away from them feeling like I had a good understanding of the problem. I still can’t tell you “here’s what it takes to operationalize a vague verbal concept.” (Aside: Can *anybody* say what it takes? Honest question. For instance, I only know about psychology what I read on blogs, so maybe what I’m about to say is way off. But FWIW, it seems like psychologists have a huge literature on measurement of latent variables, operationalizing verbal concepts, etc. And yet the field is still in a mess because of disconnects between what people actually measure and the verbal theoretical concepts they’re trying to measure.)

        Nor do I feel like I have a good understanding of how common and important failures of operationalization are in ecology, as compared to other sorts of failures. For instance, I’ve seen the IDH discussed as if its difficulties were mainly a matter of failed operationalization–failure to define *exactly* what counts as “disturbance”, failure to distinguish disturbance frequency, intensity, and extent, etc. But I’m not so sure that that’s really the root problem with the IDH.

        One thing those old posts and discussions did reinforce for me was the idea that operationalization of vague verbal concepts is hard, and so ought to be avoided as far as possible. That’s why I argued in that first old post that one ought to focus on developing and testing process-based mathematical models, because if you’re doing that questions of operationalization either don’t arise, or arise in a tractable form.

  2. Seems like a there’s a risk-reward trade-off involved here. I take “clever” to also imply “novel”, on which journals place a huge premium. Something clever brings the potential for offering a new way to think about things or to tackle big questions, and if that promise bears out, there’s a big reward for the field. But it also risks being a one-hit-wonder – noticed because it’s clever, but not ultimately pushing the field forward. The risk is deepened if the allure of following a high-profile paper with more high-profile papers attracts people to adopt a clever method that isn’t even as good as the tried-and-true. Overall, it seems to me there should be some incentives for people to try new, clever things – who knows when one of them will be a gem? – while at the same time having disincentives for pursuing cleverness too far for its own sake. Probably relates to earlier posts on bandwagons too.

    • Hmm. I’m not sure risk-reward tradeoff is the best way to think about this. Because in many cases, the downside “risks” associated with a creative new method aren’t stochastic, and so aren’t best described as “risks”. For instance, there’s a paper applying a method of mine (the tripartite partition) in a context for which it’s not intended. It’s creative to take a method designed for one context and apply it in a different context! But this new context violates one of the assumptions of the methods, rendering the results uninterpretable. I wouldn’t call that the manifestation of a downside risk, I’d just call it a mistake on the part of the investigator.

      • Fair enough. A challenge of discussions like this is that we each have different ideas about what kinds of papers we’re talking about. If you take the instrumental variables example (about which I know nothing), presumably one couldn’t tell from the outset how useful it would be, but some people saw big potential. So it was an idea worth trying out in a bunch of contexts. Your case is a specific instance in which you know from the outset that the seemingly clever idea isn’t valid, in which case it can be snuffed out before it gets going. Kids seem really good at the latter – you don’t need to try mustard on your chocolate cake to know it’s a bad idea – but I still encourage the pursuit of new combination ideas (cheese curds and gravy on fries anyone?)! What’s the next ecological poutine…

  3. Though I mostly agree with earlier comments, I value innovative methods because they help to spark my own creativity. I have questions, but figuring out how to answer them often leaves me stumped.

    New methods in place of existing ones do not interest me as much (though they may be valuable and of real help) as innovative ways to answer questions that did not have existing methods. Perhaps this is because I work mostly on applied problems. Though as Jeremy pointed out, there is a good bit of simple enjoyment in appreciating the cleverness and creativity of others. The planetarium example made me smile.

  4. There is a bit of danger in conflating the novelty and cleverness of the method with the outcome of the test. The dung beetle-planetarium example is an interesting one, as it is clever but not novel; this experiment was done 40+ years ago by Steve Emlen to test the idea that birds use elements of the stars to navigate during migration. As a teenager in 1976, I read a Scientific American article Emlen wrote summarizing his experiments, and the cleverness of the idea contributed to my latent desire to be a scientist. A problem though is that putting birds or beetles into a planetarium, while extremely creative, could have produced the result that the stars were not how they oriented. The method is still creative, and still just as adequately tests the hypothesis, but if the result is that it rejects it but does not provide positive evidence for an alternative, should that change our view of how clever it was in the first place? I haven’t seen the dung beetle paper yet, but Emlen did a number of other experiments that also helped eliminate alternatives (e.g., magnetic orientation), so the overall outcome was a tour de force of creative thought and methodology. My guess is that if indigo buntings had used something other than the rotation of the stars to orient, Emlen would have been clever enough to eliminate that idea but nail the correct one as well. Are there cases of really creative methods that failed to provide positive support for a hypothesis, but then were adopted by others for other purposes and became revolutionary?

  5. I value cleverness of methods [in their own right] because they can spark new avenues of research, and open the door to answering new questions. Many new methods [even clever ones] are probably fads that will die with time. But some are gems.

    Perhaps by clever you mean something beyond just the novelty of the method. Cleverness is hard to define, but perhaps easier to spot when you see it.

  6. of course we should… creativity is one of those things that lead to genuine advancement of science. We shouldn’t leave our other faculties behind in the process, but we really need to at least try new things, and creative failures are just as important – if you choose to take inspiration from them and not try and hold them up as something they are not.

  7. Interesting question, Jeremy. I think that “cleverness” is in the eye of the beholder and sometimes what appears to be a clever method or source of data can seem to be banal or uninteresting to outside observers. I’ve personally experienced that a couple of times. One example (which links ecology and economics) was to use sale price data from an annual holly and mistletoe auction to look at how insect pollinators affect the value of a non-food crop (something that’s hardly been done previously). Both holly and mistletoe are dioecious so any plants with berries have to be the result of pollinators. We found that pollinators boost the price of those crops by a factor of two or three. Thinking we’d been very clever in finding such data I hawked it around to some big journals, just in time for some Christmas media interest, but none of them were interested. It was eventually published, and I’m proud of the paper, but others clearly didn’t think that it was as clever a study as I did!

  8. I think it is important to recognize that clever experiments have value aside from their truth value. People, including scientists often think in terms of stories. A good methodological story is more likely to be remembered and hence pass passed on. Even if clever experiments were no better or no worse than less clever designs, a clever design may be easier for other scientists to digest. This is important because as the discussion above indicates a clever experiment might or might not be more reliable than a less clever experiment. It sounds like the milky way navigation work is clever and reliable, instrumental variable regression seems to be clever and unreliable.

  9. One more thought, i think that there is a real danger of science being haunted by what i consider the mother of all clever ideas, Relativity. Einstin’s explanations of relativity in terms of trains, lights, clocks and elevators are to my eye profoundly clever. I think many of us (myself included) dream of obtaining deep insights from such thought experiments. If there really are few such clever discoveries to be made we might be giving up on more profitable opportunities.

  10. I think one of the main reasons people value cleverness is because of what it signals about the author(s). For instance, if I see a paper with a really clever approach–especially if it’s aimed at addressing an important question–then even if the answer the study finds turns out later to be wrong, it is a good bet that the author(s) will continue on to do great things in other areas. This is probably also why people often draw a distinction between methods that are actually clever (given the currently available information) and ones that are trying too hard to be clever. Ultimately, I think we all value work that asks important questions and uses creativity and pragmatism to get the best possible answer. My guess is that most people’s favorite example of a ‘clever’ study scores highly on all of these attributes.

Leave a Comment

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s