A key problem in interpreting observational data (UPDATEDx2)

I’m a bit late on this, but I just found this very nice blog post from Bob O’Hara on a little-recognized (at least in ecology) problem in interpreting observational data: “errors in variables”. That is, for whatever reason, there’s random error in the values of your independent variables.

This is a really important problem, and lurking just under the surface of Bob’s post are larger issues, from the utility of popular statistical methods like path analysis, to what we mean by “causality” and how best to infer it. Will try to post on some of these issues in future.

Click through and read the whole thing. The comment thread is good too. Also, um, interesting. It’s particularly interesting to see the original Science paper Bob criticizes defended by a commenter who argues that the authors of the Science paper are famous ecologists like Nick Gotelli who know a lot about statistics. Because famous ecologists never make mistakes. Nope, never.* Also interesting to see the same commenter imply that, by publishing his criticisms in a blog post on which anyone is free to comment, Bob somehow wasn’t giving the authors a chance to respond. And to see the same commenter imply that Bob’s criticisms aren’t worthy of notice because they weren’t peer-reviewed.  I can only assume that the commenter is opposed to people criticizing the work of others in face-to-face conversations, since such criticisms are even less “public” and also aren’t peer-reviewed. I also assume the commenter also believes that Science‘s peer review process always weeds out weak comments. (UPDATE: On further thought the snark here goes too far. It’s possible that the commenter in question didn’t intend these implications. But I stand by my other criticisms) Finally, interesting to see the same commenter defend the Science paper by…noting that completely different ways of studying the same topic could also be criticized on completely different grounds. Which isn’t something Bob, or anyone else, ever denied. Not to mention it’s also a blanket defense against any and all criticism of anything (since every approach has its own limitations). And it’s also the same argumentative strategy used by intelligent design advocates (who see any criticism of evolution as support for their own position).

At some point I’m going to do a post highlighting some cases of productive, useful debates in ecology (feel free to suggest examples in the comments). Because I’m getting depressed seeing so many very prominent people who should know better setting such poor examples.**

*In practice, I’m sure that the commenter** who defended the original Science paper with this argument didn’t mean to claim that famous ecologists never make mistakes. But if that wasn’t the intended claim, why raise the authors’ identity at all? Why not just stick to arguing about whether or not the putative mistake was actually a mistake? What possible legitimate purpose can be served by pointing out the authors’ fame and experience? I mean, heck, Nick Gotelli, one of the authors of the Science article Bob criticizes (and whose name is invoked by the commenter in defense of the paper), is perfectly happy to address criticisms of his work on their merits, even when they’re published on blogs (see, e.g., his comments on this post). He doesn’t just respond to criticisms by invoking the authority of his own name.

**No, I won’t reveal the commenter’s identity here–think of it as incentive to click through to Bob’s post.

UPDATE #2: On still further reflection, and after correspondence with some colleagues (some initiated by me, some initiated by them), I can see how much of this post could reasonably be viewed as making a mountain out of a mole hill, treating a legitimate but minor disagreement about the appropriate response to Bob’s post with more seriousness or intensity than was called for. I decided to post because it seemed to follow on naturally from my recent post on the debate over Adler et al. Which may just be another way of saying that I was, unconsciously, looking a bit too hard for an excuse to re-emphasize some of the points made in that post. I do think that it’s important that ecologists debate one another in productive ways, and I do think that one way to help ensure that happens is to point out when others are debating in unproductive ways. I don’t plan to stop doing that, but based on the feedback I’ve received I probably need to choose my battles a bit more carefully.

This seems like a good place to say again how much I appreciate reader feedback, whether in correspondence or in the comments. Without that feedback, the blog would be much worse, I’d make even more missteps than I do, and I’d recognize fewer of them.

3 thoughts on “A key problem in interpreting observational data (UPDATEDx2)

  1. Wished-for seminars in all science graduate programs: 1. “It’s not personal: how to take and deliver criticism in a constructive manner” 2. “Getting a job with a science degree” 3. “Scientists are Human: Confirmation bias and other very human tendencies that undermine science and that we are all guilty of”

  2. I think the problem at hand is that Bob called their statement “utter rubbish”. That sort of castigation obfuscates his criticisms and just gets people’s hackles up. I know Nick is thick skinned, but Bob’s remark is just rude. The tone of your own comments towards David and John read as condescending and similarly undermine your argument. You could have achieved the same end by simply saying that you don’t think appealing to authority is a valid form of argument and left it at that. Don’t get me wrong, I totally agree with what you said. However I think maintaining civility in blogs, even the comments, helps maintain them as valid places to discuss science.

  3. Pingback: Why I love a good argument (and what makes an argument good) | Dynamic Ecology

Leave a Comment

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.