A key problem in interpreting observational data (UPDATEDx2)

I’m a bit late on this, but I just found this very nice blog post from Bob O’Hara on a little-recognized (at least in ecology) problem in interpreting observational data: “errors in variables”. That is, for whatever reason, there’s random error in the values of your independent variables.

This is a really important problem, and lurking just under the surface of Bob’s post are larger issues, from the utility of popular statistical methods like path analysis, to what we mean by “causality” and how best to infer it. Will try to post on some of these issues in future.

Click through and read the whole thing. The comment thread is good too. Also, um, interesting. It’s particularly interesting to see the original Science paper Bob criticizes defended by a commenter who argues that the authors of the Science paper are famous ecologists like Nick Gotelli who know a lot about statistics. Because famous ecologists never make mistakes. Nope, never.* Also interesting to see the same commenter imply that, by publishing his criticisms in a blog post on which anyone is free to comment, Bob somehow wasn’t giving the authors a chance to respond. And to see the same commenter imply that Bob’s criticisms aren’t worthy of notice because they weren’t peer-reviewed.  I can only assume that the commenter is opposed to people criticizing the work of others in face-to-face conversations, since such criticisms are even less “public” and also aren’t peer-reviewed. I also assume the commenter also believes that Science‘s peer review process always weeds out weak comments. (UPDATE: On further thought the snark here goes too far. It’s possible that the commenter in question didn’t intend these implications. But I stand by my other criticisms) Finally, interesting to see the same commenter defend the Science paper by…noting that completely different ways of studying the same topic could also be criticized on completely different grounds. Which isn’t something Bob, or anyone else, ever denied. Not to mention it’s also a blanket defense against any and all criticism of anything (since every approach has its own limitations). And it’s also the same argumentative strategy used by intelligent design advocates (who see any criticism of evolution as support for their own position).

At some point I’m going to do a post highlighting some cases of productive, useful debates in ecology (feel free to suggest examples in the comments). Because I’m getting depressed seeing so many very prominent people who should know better setting such poor examples.**

*In practice, I’m sure that the commenter** who defended the original Science paper with this argument didn’t mean to claim that famous ecologists never make mistakes. But if that wasn’t the intended claim, why raise the authors’ identity at all? Why not just stick to arguing about whether or not the putative mistake was actually a mistake? What possible legitimate purpose can be served by pointing out the authors’ fame and experience? I mean, heck, Nick Gotelli, one of the authors of the Science article Bob criticizes (and whose name is invoked by the commenter in defense of the paper), is perfectly happy to address criticisms of his work on their merits, even when they’re published on blogs (see, e.g., his comments on this post). He doesn’t just respond to criticisms by invoking the authority of his own name.

**No, I won’t reveal the commenter’s identity here–think of it as incentive to click through to Bob’s post.

UPDATE #2: On still further reflection, and after correspondence with some colleagues (some initiated by me, some initiated by them), I can see how much of this post could reasonably be viewed as making a mountain out of a mole hill, treating a legitimate but minor disagreement about the appropriate response to Bob’s post with more seriousness or intensity than was called for. I decided to post because it seemed to follow on naturally from my recent post on the debate over Adler et al. Which may just be another way of saying that I was, unconsciously, looking a bit too hard for an excuse to re-emphasize some of the points made in that post. I do think that it’s important that ecologists debate one another in productive ways, and I do think that one way to help ensure that happens is to point out when others are debating in unproductive ways. I don’t plan to stop doing that, but based on the feedback I’ve received I probably need to choose my battles a bit more carefully.

This seems like a good place to say again how much I appreciate reader feedback, whether in correspondence or in the comments. Without that feedback, the blog would be much worse, I’d make even more missteps than I do, and I’d recognize fewer of them.