Friday links: Andy Warhol on citations, and (a bit) more

From Jeremy:

Philosopher of ecology Jay Odenbaugh argues that ecological theory doesn’t need to make accurate predictions to be successful science, and that you’re misunderstanding the purpose of theory if you think otherwise. The paper’s a few years old, but you probably missed it at the time and it’s still relevant. Speaking of predictions, I predict a counterargument from Jeff Houlahan in the comments in 3..2…1…🙂 (ht Ben Kerr)

In 2003 Wellesley College implemented a policy to combat grade inflation. Here’s a rundown of the effects it’s had since then. It’s an interesting study in part because the policy wasn’t campus-wide–it only affected 2/3 of the departments (most of the science departments, and economics, weren’t affected by the new policy because grades in those departments ran lower). Briefly, the policy worked–the marks that were supposed to drop, dropped, and by just enough to comply with the new policy. Students’ choices of major shifted, at least within the social sciences, in such a way as to suggest that previously students were choosing their majors in part out of a desire for high marks. Students in the affected departments gave their profs poorer course evaluations. And there were other effects.

In the future, everyone’s papers will be cited for 15 minutes. (ht Retraction Watch)

To make up for the paucity of reading material this week, here are some funny pictures I found by googling “ecology meme”:


10 thoughts on “Friday links: Andy Warhol on citations, and (a bit) more

  1. I think there is some ambiguity in terms of prediction. I think most people see prediction as testing the the prediction derived from their hypothesis (i.e. model fit), while others would see it as testing a model(s) on out-of-sample data. While accurate prediction is undoubtably desireable, I think the point should be that we are testing models on out-of-sample data to see if what we ‘described’ in our models still holds true at different scales. This is not something I see done often in ecology and possibly not at all outside of species distribution models.

    • There’s more to prediction than just in-sample fit vs. out-of-sample testing as well–it sounds like you’re restricting attention to statistical model evaluation? For instance, Darwin’s famous prediction of the existence of a long-tongued moth to pollinate a long-spurred flower isn’t best thought of as either an in-sample or out-of-sample prediction, I don’t think.

  2. Hi Jeremy, far be it from me to avoid rising to the bait…but, I suspect my position on this is not as extreme as you might think. I don’t really disagree with Jay Odenbagh. So, there are two reasons why I don’t need ecological models to make very good predictions. First, there are so few models in ecology that make good predictions (on independent data) that my expectations aren’t very high – if you can do better than flipping a coin I’m already a little bit impressed. Second, the ultimate goal is to demonstrate understanding by making good predictions but that doesn’t mean that every intermediate step has to achieve the objective. If we don’t know what could happen that will likely slow down progress towards discovering what does happen. But, we have to keep our eye on the ultimate objective – understanding how the world works. And we can only demonstrate we understand the world better if, yada yada yada. Best, Jeff H

  3. While it’s not surprising that course evaluations went down in the departments that lowered grades, it’s still an interesting piece of evidence about how evaluations are influenced by things other than instructor quality.

      • The effect on black students was most surprising to me. I need to think about that one more.

      • “The effect on black students was most surprising to me. I need to think about that one more.”

        I’d need to know more about exactly how the data were analyzed before commenting on that one. The linked piece shouts that result from the rooftops in the headline, but then in the piece itself doesn’t say much about it.

  4. Hi Jeremy, I’m with Bob on this – when I say prediction I mean out-of-sample (and maybe even out-of-population)…and ecology doesn’t do it much except, as Bob mentioned, for species distribution models. And Darwin’s prediction is absolutely an out-of-sample prediction. He based his hypothesis on a bunch of interactions he had seen between orchids and moths and what natural selection implied about co-evolution and he made a prediction to an orchid that he hadn’t used to develop his hypothesis. Similarly, Einstein’s prediction that light from Hyades star cluster would bend as it passed the sun was an out-of-sample prediction – he hadn’t used that fact to develop his hypothesis. The same is true of Halley’s prediction for when his comet would return – it;s an out-of-sample prediction. Best, Jeff H

    • Hmmm…I see what you mean, but I think you’re stretching the notion of an “out of sample” prediction by applying it to situations in which there aren’t any statistical populations, much less samples from them. I’m not sure I want to stretch the statistical notion of “out of sample prediction” to cover all cases in which someone makes an impressive prediction. It’s not clear to me that the statistical notion of “out of sample prediction” captures everything that makes a prediction impressive.

      As for whether ecologists make lots of out-of-sample predictions, however defined…not sure. Just off the top of my head, I can think of lots of ecology papers that use a theoretical model to make a prediction, then go out and test it. Which seems like “out of sample” prediction on your definition. For instance, Vasseur & Fox 2009–we used a theoretical model to derive a prediction about the spatial synchrony of predator-prey cycles, and then we tested that prediction in protist microcosms. That sort of thing isn’t all *that* rare in ecology, is it?

      • I think I’m stretching it a little too – you’re right that when we use a term as precise as out-of-sample it certainly implies a statistical model. Usually I talk about ‘independent’ data to imply data that weren’t used to inform the theory. And certainly some predictions on independent data are riskier than others.
        And you’ve identified the place where I think ecologist’s do a somewhat better job of predicting on independent data – when they have a strictly theoretical model. Where we almost never see tests on independent data are when ecologists develop statistical models.


Leave a Comment

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s