Questions and resources about structural equation models

There’s an Aesop’s fable called “The Mountain in Labour“, about a mountain or volcano that rumbles and groans impressively but then “gives birth” to a mere mouse. It’s a parable about promising much but delivering little.

I’m like that mountain, and this post is like that mouse. For months (since back when I was still writing for the Oikos Blog), I’ve been promising to do a big post critiquing structural equation models (SEMs). But I decided that writing that post would require too much work on my part to update my knowledge of SEMs. So instead I’m going to do something sort of like what I did with frequentist vs. Bayesian statistics: provide an annotated bibliography of some stuff about SEMs, including material critiquing SEMs.

I won’t venture to say if SEMs are used in a “macho” way, but it is a complex, sophisticated approach involving a lot of judgment calls. Very briefly and roughly, structural equation modeling is a way to try to estimate the direct and indirect “causal” connections among a set of variables. The simplest structural equation models can be thought of as a bunch of pieced-together linear regressions and/or multiple regressions, in which some of the predictor variables for one regression simultaneously serve as the dependent variables for another regression. The idea is basically the same as the idea behind instrumental variables: if, after controlling for all other sources of variation, you can show that dependent variable Y has a significant (partial) regression on X, then you infer that X has a causal effect on Y. You piece together several regressions in order to describe indirect causal pathways like X affects Y only via its effect on Z. For details, read the stuff I’ve linked to below.

As with my previous list about Bayesian vs. frequentist stats, the list below is far from comprehensive. It’s merely a list of some things I’ve encountered pretty much at random, and found useful. Commenters are encouraged to suggest other useful readings.

I will throw in a few comments of my own, as I do know something about SEMs. Back in grad school I did a big review paper on SEMs in ecology and evolution for one of my courses, and papers I read sometimes include SEMs. Me being me, my comments on SEMs are mostly conceptual-philosophical-whatever contrarianism, the sort of thing that I don’t think can be dismissed as just one man’s idiosyncratic personal opinion or ignorance, but with which others might reasonably disagree. I freely grant that SEMs can be a quite useful tool, but I’m skeptical that SEMs, even when done well, can deliver everything their strongest proponents promise. I also think that SEMs are (like other sophisticated approaches) quite hard to do well, and that (unlike many other sophisticated approaches) they have certain features that strongly tempt ecologists to do them badly. But like I said, reasonable disagreement with my views certainly is possible, and I’m looking forward to feedback in the comments.

An entry point into the ecological literature on SEMs

Grace et al. 2012, just published in Ecosphere, is an open-access review of the history of the development and application of SEMs, within and outside ecology, with an emphasis in recent methodological developments due to the work of Judea Pearl and others. Jim Grace is basically “Mr. SEM” in ecology, so this paper is a must-read for any ecologist interested in SEMs. Having said that, if you’re not already very familiar with SEMs, you’re likely to find this paper rough sledding (I did). It uses lots of jargon, and while the glossary is helpful, no glossary can make jargon-heavy writing equivalent to jargon-free writing. The paper also tries to cover an awful lot of ground in a small amount of space (which perhaps explains why it’s jargon-heavy; jargon saves space). Very important points can easily whiz past the unsuspecting reader. It’s not a bad paper by any means; there’s absolutely a lot of value in this sort of “one stop shopping” paper. But it’s written more for people who already know about SEMs and want to learn the latest wrinkles than for people just starting out. Maybe that wasn’t Grace et al.’s intent, but that’s how it came across to me. Anyway, my recommendation if you’re new to the subject: use the reference list as an entry point into the SEM literature. Read a bunch of the older stuff that Grace et al. cite, and only then read their paper.

A blog about statistically inferring causality

Causal Analysis in Theory and Practice is mostly aimed at SEM experts. If you want to stay in touch with the latest news and thinking from the SEM community, this seems like a good place to do it. I haven’t dug into it much (much of it is beyond me), but I’ll note in passing my impression (admittedly based on cursory reading) that it’s written by folks whom one might call “true believers”. These are people who seem to believe not just that SEMs are the proper way to infer causality, but that SEMs define the concept of “causality”. I do not pretend to have the expertise to arbitrate such strong and deep philosophical claims. I’ll merely note that “causality” is an infamously difficult concept to pin down in philosophy, and so I’m skeptical that SEM guru Judea Pearl, brilliant though he is, has completely figured it out. It also worries me that the authors of this blog seem so confident that SEMs are The Right Way to think about “causality” that they’re quite impatient with pushback. See, e.g., this post, which is quite dismissive of an article criticizing SEMs that Andrew Gelman for one agrees with. Not that Gelman is infallible–no one is–but I’m pretty sure anything he agrees with can’t just be brushed aside. I’m not sure if these deep conceptual disputes matter for the ordinary application of SEMs in ecology. Often philosophy (explicit or implicit) has implications for practice, but sometimes it doesn’t. But as you read up on SEMs, it would probably behoove you to read stuff criticizing the approach as well as stuff trying to “sell” the approach, and then make up your own mind. Fortunately, Grace et al. aren’t “true believers”. They explicitly state that you cannot reliably infer causality from observational data alone, and that SEMs are merely one tool for helping you make reliable causal inferences.

A recent paper critiquing some aspects of SEMs

Lindquist and Sobel is a nice little paper, part of an exchange between the authors and SEM proponents.

Andrew Gelman’s thoughts on SEMs

Ace applied statistician and blogger Andrew Gelman has written a fair bit about causal inference in general and SEMs specifically. See here, here, here, and here. In particular, I agree with Gelman that it’s often better to try to estimate or predict the effects of particular interventions or manipulations than to try to estimate or predict “the” causal effect of variable X on variable Y. How you actually manipulate X generally affects the response by Y, but SEMs assume it doesn’t. I also think Gelman’s point that path strengths are rarely if ever literally zero, and that this creates problems for SEMs, applies in ecology as well as in social science (the area in which SEMs are most popular and have been best developed). Indeed, it’s worth keeping in mind that while SEMs grew out of an invention of evolutionary biologist Sewell Wright’s (path analysis), their “native habitat” is social science. Social science datasets often have much larger sample sizes than ecological datasets. Conversely, social scientists often cannot perform the sorts of manipulative experiments that ecologists can perform. So if you don’t have a massive dataset, and/or can do manipulative experiments, you may want to consider whether SEMs are the right tool for whatever job you’re trying to do.

A reading list on causality and causal inference

Here is the brilliant Cosma Shalizi’s reading list on causality and causal inference. A lot of it is pretty advanced and perhaps not strictly essential for a practicing ecologist. But if you’re serious about using SEMs I think you ought to at least dip your toe into this stuff. As Grace et al. emphasize, best practice in structural equation modeling is highly non-routine. There’s no recipe to follow, and lots of judgment calls are involved. Indeed, recent developments in SEMs have only increased the number of judgment calls to be made, which kind of worries me since back when I reviewed this literature in grad school I found that ecologists and evolutionary biologists almost universally failed to follow what was at the time regarded as best practice in path analysis and SEMs. Anyway, you’ll make better judgments if you are at least passingly familiar with the underlying philosophical foundations of SEMs. It’s not enough for you just to know how to drive the car–you need to know something about how the car was designed to work.

A classic application of SEMs in ecology that people should emulate

Wootton 1994 used path analysis (a form of SEM) to estimate the direct and indirect effects of different rocky intertidal species on one another. And instead of stopping there, he used the model to make eleven non-obvious predictions about the outcomes of experimental manipulations of species’ densities and bird predation. All eleven predictions were supported. I read this paper in class as an undergrad and was hugely impressed; I still am. So how come I hardly ever see anyone following Wootton’s example and using manipulative experiments to validate their SEMs? I freely admit my reading is far from comprehensive–do people do this routinely and I just miss it? I have the impression that many ecologists see SEMs as a stopping point rather than a starting point, which I don’t think is the most effective way to use SEMs in most cases.

A book about SEMs by an ecologist

Ecologist Bill Shipley is best known for his MaxEnt work these days, but he’s also written a book on SEMs. You should definitely pick up a copy if you’re serious about using SEMs in your own ecological work.

An SEM short course

Ecologist Jarrett Byrnes teaches a short course on SEMs. Much of the course material is here. Perusing this material is another way to get up to speed a bit before you try to tackle Grace et al.

My own questions/concerns about SEMs in ecology

Just some food for thought, on which I’d welcome feedback. As I say, I know enough about SEMs to have opinions–but not super-strongly-held ones.

• Ecologists mostly do SEMs by specifying one or perhaps a couple of alternative “path diagrams”, specifying which variables directly causally affect which others. Those path diagrams often do a pretty rubbish job of fitting the observed data, which is worrisome if (as is often the case in ecological applications of SEMs) your conclusions depend on your diagram being at least close to the “true” one. For instance, if you want an accurate, precise estimate of the strength of a particular causal link, your entire path diagram needs to be correct or sufficiently close to correct. I believe there are ways to try to estimate the correct diagram from the data rather than specifying it a priori, although my understanding is that rather restrictive conditions are required for this to work. Still, I’d be curious to see someone compare the ability of pre-specified path diagrams, vs. those estimated from the data, to predict the results of follow-up manipulations or the properties of datasets not used in the original fitting/estimation.
• SEMs often are used to fit a verbal or graphical causal model (“path diagram”) directly to the data. That is, you just draw boxes representing your variables, and connect them with arrows representing direct causal effects of one variable on another, and you’ve got most of what you need to fit an SEM. This is often touted as a virtue of SEMs, but in many cases I think it’s a vice. Ecologists’ verbal intuitions about how dynamical systems work are not only often wrong, they can even be illogical. By giving you the ability to just draw a sketch of how you think your system might work and then fit that sketch directly to data, I think SEMs tempt you to skip over the hard work of actually writing down and analyzing the underlying process-based model that you think describes your system. Putting in that hard work often changes, clarifies and makes more precise both your assumptions and predictions. And while I’m sure Grace et al. would advise SEM users not to be so cavalier, I worry that the temptation to be cavalier is built in to the approach and so difficult to resist. Yes, you should think hard about how exactly it is that X affects Y in your SEM–but writing down a dynamical model forces you to do just that.
• Because you can estimate the statistical relationship between any two variables, the causal “arrows” in your path diagram often are very difficult to interpret. I’ve struggled to articulate exactly what my concern is here (it’s not just “correlation doesn’t equal causation, though that’s part of it), but let me give it a shot. For instance, a simple SEM might ask how some environmental variable (say, soil N) affects total plant biomass both “directly”, and “indirectly” via its “direct” effect on plant species richness. Indeed, I wouldn’t be at all surprised if there’s a paper in the literature presenting precisely that SEM! Now maybe it’s just me, but I have no idea how to interpret this. Each causal arrow in our hypothetical SEM is somehow aggregating and summarizing an awful lot of what I would call the underlying causal linkages between those variables. Worse, what’s being summarized by each “arrow” in our causal diagram is totally different. The underlying mechanisms that directly link soil N to plant biomass aren’t anything like the underlying mechanisms linking soil N to plant species richness, and neither is anything like the underlying mechanisms linking species richness to total plant biomass. It seems to me that what this little hypothetical SEM is doing isn’t much different than just describing the correlations (and partial correlations) among different variables, and then declaring some of those correlations and partial correlations to be “causal”.
• SEMs aren’t great at dealing with feedbacks and nonlinearities, both of which are absolutely ubiquitous in ecology and evolution. Put another way, SEMs aren’t dynamical systems models. SEMs seem to be to be a tool for thinking about “billiard ball causality“, not a tool for thinking about state variables with simultaneous inflows and outflows. When I draw boxes and arrows, I’m thinking of stocks (state variables) and flows (e.g., births are an inflow into a population, and deaths are an outflow). In the dynamical systems models I think about, like standard competition and predator-prey models (and models of lots of other things in areas outside of population and community ecology), the causes operate on rates of change (i.e. rates of inflow and outflow), not on the state variables themselves. For instance, increasing parameter r might cause prey growth rate to increase; to find the resulting change in prey abundance at some future time you need to integrate the dynamical equations. Now, advocates of SEMs are in my (admittedly-limited) experience very quick to claim that my views on this are out of date. They say that SEMs can totally handle dynamical systems. In particular, they say that old-fashioned path analyses can’t handle feedbacks (which SEM folks call “causal loops” or “cyclic graphs”) and nonlinearities, but modern SEMs can easily handle any feedback structure or nonlinear system you care to specify. Maybe I’m just ignorant here (and I could well be), but I don’t think that’s true. Yes, you can try to deal with feedbacks by including time-lagged variables in your SEM (e.g., last year’s value of X affects this year’s value of Y, and this year’s value of Y affects this year’s value of X). But that’s simply a misdescription of many feedback relationships in ecology. Yes, you can deal with feedbacks by having causal loops in your SEM–but only certain sorts of loops, and only if embedded in the right sort of larger causal network, if I recall correctly. For instance, I don’t think you could model predator-prey dynamics as an SEM by saying that predator abundance affects prey abundance, and prey abundance affects predator abundance. Not unless you also had some other variable in there that just affects prey, or just affects predators. Nor do I think you could fit, say, a Lotka-Volterra competition model as an SEM, not if every possible direct effect is non-zero (i.e. every species has a non-zero competition coefficient with every other, including itself). And while you can deal with nonlinearities by trying to transform variables so as to linearize the relationship between them, that’s a quite limited and often ineffective way to deal with the sort of nonlinearities ecologists routinely encounter. Bottom line: if you can write down a dynamical systems model (a system of ODEs, PDEs, difference equations, or whatever) for your system, or if (as is more commonly the case) your predictions about how you expect your system to behave derive from some dynamical model, I don’t really see why you’d bother trying to force the analysis into an SEM framework. That’s trying to force a square peg into a round hole. Just fit the dynamical model to your data, or parameterize it in other ways, or (most commonly) do some experiments to test the predictions of that dynamical model, or etc. Even if I can’t write down a dynamical model for whatever reason, I still find it much more helpful to think in terms of state variables and the inflows and outflows that change the values of those state variables.
• SEM advocates often say that a virtue of SEMs is that they reflect how we intuitively think about causality. And they probably do–but I think that’s a vice, not a virtue (or at best irrelevant). The world is often non-intuitive. “Folk physics” famously differs from real physics in all sorts of ways. Similarly, “folk ecology” is very different from real ecology. In my experience, people’s intuitions (including mine) about how anything of any complexity works, and about how to find out how anything of any complexity works, are mostly terrible. Science should be based on methods that reliably separate truth and falsehood; whether those methods are “intuitive” is totally irrelevant. If reliable scientific methods are unintuitive or otherwise hard to understand, well, nobody said science was easy.

And finally, an old post that isn’t strictly about SEMs but is sort of relevant

Before you get too excited about any single technique or approach, remember: techniques aren’t powerful, scientists are.

Rebuttal from Jarrett Byrnes in 3, 2,… 😉