Instrumental variables: the key to analyzing "natural experiments"?

Inferring causality is hard. Especially in a world where lots of factors, some of them unknown, causally affect the response variable of interest (and each other), and where there are causal feedbacks (mutual causation) between variables. It’s even harder when, for whatever reason, you can’t do a properly controlled, replicated experiment. What do you do then?

One standard answer is to rely on what Jared Diamond (and probably others) have called “natural experiments”.  The basic idea is as follows. If you think that variation in variable A causes variation in variable B, compare the level of B across systems that vary in their level of A. So instead of manipulating A yourself, you’re relying on the “manipulations” (variations) in the level of A that nature happens to provide.

Unfortunately, natural experiments are infamously unreliable, not just compared to “real” experiments but in an absolute sense. As my PhD supervisor Peter Morin liked to say, “The problem with natural experiments is that there’s no such thing as a natural control.” That is, systems that vary in their level of A often vary in lots of other ways as well, some of which probably also affect the level of B. You can of course try to address this by statistically controlling for the levels of those other variables, assuming you can identify them. And you can try to simply collect lots of data from a large range of systems in the hopes that surely some of the among-system variation in variable A will be independent of all confounding variables. And you can try to get rid of any causal feedbacks from B to A by praying to the god of your choice…

Or maybe there’s a better way. Economists have to deal with all the same challenges in inferring causality that ecologists do. If anything, economists have it even worse because doing relevant experiments often is harder in economics than it is in ecology. In response, economists have come up with an interesting and potentially-powerful approach to inferring causality from natural experiments, the method of “instrumental variables” (IV).

Here’s the basic idea (for details, click the link above, which goes to the very good Wikipedia page on IV). An instrumental variable, call it X, is a variable that causally affects B only via its effect on A, and that is not itself causally affected (directly or indirectly) by B or A. Economists summarize the latter assumption by saying that X is “exogenous”. So you can estimate the causal effect of A on B by using, not just any natural variation in A, but only that natural variation in A that can be attributed to natural variation in X. Changes in X are perturbations that propagate to B via only one causal path, that running from A to B, so variation in the instrumental variable X allows you to estimate that strength of that causal path. The approach can be generalized to multiple causal paths, as long as you have multiple instrumental variables.

One thing I find interesting about IV is that they highlight how “more data” is not always helpful. Tempting as it is to think that, if only you had enough data on A from enough different systems, you could reliably infer the causal effect of A on B, it’s not true. What you need is not more data on the variability of A, you need the right sort of data on the variability of A (namely, that generated by an instrumental variable). Indeed, more of the wrong sort of data on variability in A can actually be harmful to inferring the effect of A on B.

The nice thing about the IV method is that it doesn’t require you to know anything about the rest of the system, such as other variables that might affect B while also covarying with A. All you have to know (and this is the hard part) is that X is what economists call a “good instrument”–that it satisfies the assumptions that make it an instrumental variable.

Which may limit the applicability of IV in ecology. In economics, IV are often policy changes. For instance, an increase in cigarette taxes should affect health only via its effect on how much people smoke. So you can use changes in cigarette taxes to estimate the effect of smoking on health, thereby getting around the fact that lots of factors may affect both health and smoking, and that people’s health may affect their inclination to smoke. Weather events like droughts also tend to make good instruments in economics.

I’m unsure whether ecologists will often have good instruments available to them. Weather is exogenous to ecological systems as well as to economic systems. But the problem is that weather changes typically affect any variable of interest via multiple causal pathways. And many policy changes certainly have ecological as well as economic effects. But the problem with many policy changes affecting ecological variables is that they’re not exogenous–the policy changes are made in response to observed changes in the variable which the policy change is intended to affect. So if ecologists want to use policy changes as instrumental variables, they may want to focus on policies with unintended ecological consequences. And even there you still might have the problem of unintended consequences propagated via multiple causal paths.  But we won’t know if IV can be useful in ecology if we don’t try them out.

And if you do try out IV and get them to work, I hope you’ll submit the paper to Oikos. 😉

18 thoughts on “Instrumental variables: the key to analyzing "natural experiments"?

  1. Perhaps you are aware that there’s already an interesting ecological application of regression with IV, published a few years ago:

    Creel S. & Creel M (2009). Density dependence and climate effects in Rocky Mountain elk: an application of regression with instrumental variables for population time series with sampling error. J Anim. Ecol., 78, 1291-1297.

    This papers exploit the fact that an instrumental variable is correlated with the regressors, but not with the errors, to compare the performance of IV regression with respect to state space modeling; obviously, there’s a lot of subtleties in both the IV and state space modeling approaches that are no dealt with in the above paper (for example, the Kalman filter can indeed be easily extended to accommodate strongly non-linear processes and non-Gaussian errors although the authors intriguingly suggest this is not possible, and it would be interesting to compare IV vs. state space approaches in these settings); but it is definitely a good starting point!

    • I was not aware of this, thank you for the link. Indeed, the main reason I wrote the post is so that readers could alert me, and each other, to applications of this approach in ecology.

  2. Great write-up of IV! And from IV-regression, it’s just a hop, skip, and a jump to full Structural Equation Modeling with observed variables. What I enjoy about the comparison between the two is that for each, you need to think carefully about 1) causality and 2) the whole system, not just a single bivariate relationship. Something I would love to see more of in Ecology!

    • Thanks Jarrett. But don’t get too excited–there’s going to be a follow-up post on structural equation models at some point, that’s going to be rather more skeptical. Maybe I’ll try to time it to coincide with the SEM course that I hear you teach, so that you can assign all your students to comment on my post. 😉

      • I look forward to it! FYI, I can’t remember if I have mentioned this before, but, I just started in on reading Judea Pearl’s Causality. You may find it of interest. In particular, it covers a few issues that SEM folk have been not as solid on until now.

      • I’d say if you have reason to be skeptical of SEM approaches you really should be skeptical of IV approaches*. I say this because IV approaches make all the assumptions of SEM, and a boatload more. Essentially, from what I understand, IV approaches heavily rely on linearity of the relationship between the instrument and the predictor that your interested in. In addition to that, if the strength of the instrument is weak (which a lot of instruments are), then even a very small relationship between the suggested IV and the outcome variable can lead to wildly inaccurate estimates of causal effects. The final nail in the coffin for me for IV approaches is that there is no way to actually test for that second problem; If it turns out your instrument actually does influence your final outcome through a path not through the treatment, you won’t be able to tell.

        *I am not an expert on IV methods, but I’ve got this from people who have far more experience with IV, and have seen them massively misused. The one post I was trying to find was by Daniel Davies on D-Square Digest, but I see he’s set his blog to a friend-only view, so I’ll link to Cosma Shalizi’s a href=”http://www.stat.cmu.edu/~cshalizi/402/lectures/23-causal-estimation/lecture-23.pdf”> course notes talking about this issue .

      • Thanks Eric. As I said in response to another comment, it was my hope that readers who know more than I do would chime in. I agree that the assumptions required to make IV analysis work are pretty strong, and in particular having an instrument with a strong effect is crucial.

        Whether IV or SEM merits more skepticism, I’m not sure. It’s not just a matter of what assumptions each approach makes, but our ability to assess whether those assumptions are satisfied. But it’s not an issue I’ve thought about a lot yet.

        I suspect IV proponents might respond to these concerns by admitting to them, but would also argue that the naive comparative approach–just ask how variable B varies as a function of variable A–is even worse.

        Frankly, neither IV nor SEM is an approach that I myself would be inclined to try, for reasons I’ll be elaborating on in a future post (briefly, my biggest concerns are actually more philosophical than statistical). But that’s mostly because I choose my questions, and the systems in which I address those questions, so that I can use much more powerful and reliable approaches. In writing posts on IV and (soon) SEM, I’m consciously taking a break from arguing for my own approach to science and trying to put myself into somebody else’s shoes.

  3. Interesting, Eric. And this is where SEM does come in – the test of mediation is fairly straightforward and can easily test that second assumption. And/or a good examination of semi-partial correlations. As for linearity, one can conduct an IV analysis with nonlinear responses. The underlying idea is the same, but the statistical machinery is just a wee bit different. Same for SEM, although my current understanding is that this is easier to deal with in IV analysis. And, honestly, if you’re assuming linearity everywhere you look, that’s going to be a problem in any analysis. Linearity is something we all too often blithely assume, even when the problem of interest cries out that linearity is the incorrect assumption.

    Thanks for the fantastic link!

    • One of the things I like about Judea Pearl and collaborators approach to causal modelling is that it not only lets us make statements about what causal relationships the data can support, it also lets us say that “given a set of predictors, no amount of evidence will allow us to distinguish between some sets of causal models”. I think that’s my biggest problem with IV… it seems very easy to disregard alternate supported models, since it doesn’t force you to write out your causal model explicitly. This might very well be my personal prejudices speaking though. 🙂

  4. Being a relative newbie to SEMs, I’m glad to see I’m not the only one who was thinking “this all sounds a bit like path analysis” as I was reading the post.

    The introductory stuff I’ve read about SEMs in Ecology has been very clear about its assumptions and limitations. However, I’ve also seen the method thoroughly abused in some manuscripts – e.g., basically as a data trawling tool with too few a priori hypotheses for the proposed directional, causal links.

    Looking forward to your actual SEM post, Jeremy!

  5. Pingback: Questions and resources about structural equation models | Dynamic Ecology

  6. I would present instrumental variables differently (but I am not an ecologist, I usually put an “applied statistician” in my resume; on some days, I go as a “quantitative social scientist”). Instrumental variables do not have to be causal. X does not have to cause A, it only needs to be correlated with it (let’s call it “condition R”, for relevance). More importantly, X has to be uncorrelated with the error term in the regression of B on A (let’s call it condition “U”, uncorrelatedness), and that’s a more important requirement than X having a strong association with A. If “R” is broken down, you have weak instruments, and the distribution of the t-statistic from the main regression is screwed up, but fixable if the right distribution for that t-statistic is used. If “U” is broken down, you have a biased estimate in the main regression, you are screwed up beyond repair, as you have a wrong estimate to deal with, in the first place. In econometrics, there are test for both valid instruments (“U”; the Sargan test for overidentifying restrictions) and for the weak instruments (“R”, see the paper I linked). The current publication standards, if only implicit ones, require you to present both to argue that none is an issue with your data. Angrist and Pischke’s Mostly Harmless Econometrics is a relatively non-technical (no matrix algebra), yet fully rigorous discussion of how economists view causality these days. Angrist, incidentally, is guilty of misleading everybody at some point with his own use of a very weak instrument.

    More on the point that the instrumental variables don’t have to be causal: Bollen (1996) used instrumental variables to estimate factor analysis models, which is a very plain model with causality going from the latent factors to the observed variables, but no causal relations of the factors themselves. However, Bollen used the “neighboring” indicators of a latent variable as instruments: if you have a factor (sub)model with a factor f and its indicators x1 = f + e1 (with the loading set to 1), x2 = b2 f + e2, x3 = b3 f + e3 (and these regressions can be interpreted in a causal way, if the application permits), then b2 is estimated from an instrumental variables regression of x2 on x1 using x3 as an instrument, and b3 is estimated from an instrumental variables regression of x3 on x1 using x2 as instrument. This is mind-boggling for economists who are used to searching for their instrumental variables far and wide outside of their models. And this may be mind-boggling to you if you restrict the meaning of the instrumental variables to the causal effects: obviously, there is no causal effect of x2 on x1, or x1 on x3, in the first regression to estimate b2. However, this use of the variables x1-x3 fully satisfies the complete definition of instrumental variables given by econometricians.

    Given that you guys in ecology have a good idea about both SEM and instrumental variables, I would expect the quant ecology to grasp the concept of instrumental variables in SEM faster than psychologists who have little clue about them even 15+ years after Bollen’s original publication.

    • Good to have comments from a proper statistician and econometrician; thanks!

      I see what you mean about instrumental variables not having to be causal. But in practice, isn’t that how many folks try to use them–as a way to help infer causality, or as one line of evidence about causality? That’s an honest question, to which I don’t know the answer, since instrumental variables aren’t much used in ecology (though I’m aware that the answer might be “it depends how you define ‘causality'”).

      Flattered to hear that ecologists understand this stuff better than psychologists. 😉

  7. Pingback: Friday links: how ecological research has (and hasn’t) changed in the last 30 years, and more | Dynamic Ecology

  8. Pingback: Friday links: your sample size is too small, tips for the academic nomad, and more | Dynamic Ecology

Leave a Comment

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s