Inspired by, and related to, Meg’s recent poll on how you interpret interaction terms and main effects in ANOVA, I thought I’d ask my own ANOVA-related questions:

Part of the reason I’m asking these two questions is that different stats programs default to different calculation methods. Base R defaults to type I SS (though some packages have different defaults), while SAS defaults to type III. All of these methods give the same answer when the design is balanced (which is one good reason to balance your designs!), but they give different answers for unbalanced designs because they test different hypotheses. So really, what I’m asking is, exactly what hypotheses do you prefer to test when doing ANOVA with >1 factor?

I’m also curious whether people’s preferences match up with the defaults in their preferred stats packages. And if the match is because stats package defaults shape people’s preferences, rather than people’s preferences shaping their choice of stats package.

At the risk of biasing your answers, here’s an accessible discussion of the issue, here’s another, and here is a third (ht ucfagls, in a comment, for that third link).

10 thoughts on “Poll: how do you calculate sums of squares in an unbalanced ANOVA?”

Its depends on whether I expect the interaction to be significant and/or biologically meaningful. I will also throw Andy Hector et al.’s review paper out there as another excellent treatment of this topic: Analysis of variance with unbalanced data: an update for ecology & evolution (http://dx.doi.org/10.1111/j.1365-2656.2009.01634.x)

Yes, Andy Hector and Bernhard Schmid have long advocated doing type I SS with different orderings of model terms to help the investigator understand how much variation can be uniquely attributed to each factor vs. how much is shared between two or more factors. That’s always seemed sensible to me as an exploratory approach, though of course you wouldn’t want to then turn around and only report whichever ordering of terms gave you the answer you liked best (and Andy & Bernhard would never suggest that).

And as their experience with BDEF work illustrates, there are times when it’s literally impossible to have a balanced experimental design. So these issues come up even if you do your best to balance your designs.

Well, I did not expect this. So far, the most popular stats package by far is R (no surprise there)–but type I SS is easily the least-popular choice, with the most popular being type III SS. The influence of SAS lingers on long after people have stopped using it?

I leave it to others to decide if it’s worrying that so far 15% of voters weren’t aware that there is more than one way to calculate SS for an unbalanced design…

Thanks for following up with this Jeremy, even if it doesn’t get a lot of traffic.

As of my viewing now, 42 use type III SS, but only 16 do not use R. Retrieving type III SS’s in R was not obvious to me (I think you still have to probe the net to find how to do so). Either folks are well versed in R and stats here, or perhaps there is a misconcenption about what SS are given default via R? I’m baffled.

“15% of voters weren’t aware that there is more than one way to calculate SS for an unbalanced design”
Often not a big deal, but when it matters, it really matters. Doesn’t worry me too much, as I’m an optimistic skeptic who likes to read details.

“perhaps there is a misconcenption about what SS are given default via R?”

I’m now wondering that as well. Even though our readers probably do include a decent fraction of very R-savvy and stats-savvy people.

In terms of getting R to spit out type III SS, there are add on packages that do it by default–I know lmPerm is one. But in base R, I don’t think there’s any easy way to do it, though my R-fu is weak and perhaps there’s an easy way I don’t know about.

I’ve always enjoyed Langsgrud‘s paper on it. It explained the issue in good detail, and convinced me that often what ecologists really want is type II when you press them on the real meaning of the test. Is it meaningful to have A + A*B? Is that a model you can conceive of? When pressed, the answer is often no. This is also a nice explanation.

Its depends on whether I expect the interaction to be significant and/or biologically meaningful. I will also throw Andy Hector et al.’s review paper out there as another excellent treatment of this topic: Analysis of variance with unbalanced data: an update for ecology & evolution (http://dx.doi.org/10.1111/j.1365-2656.2009.01634.x)

Yes, Andy Hector and Bernhard Schmid have long advocated doing type I SS with different orderings of model terms to help the investigator understand how much variation can be uniquely attributed to each factor vs. how much is shared between two or more factors. That’s always seemed sensible to me as an exploratory approach, though of course you wouldn’t want to then turn around and only report whichever ordering of terms gave you the answer you liked best (and Andy & Bernhard would never suggest that).

And as their experience with BDEF work illustrates, there are times when it’s literally impossible to have a balanced experimental design. So these issues come up even if you do your best to balance your designs.

Well, I did not expect this. So far, the most popular stats package by far is R (no surprise there)–but type I SS is easily the least-popular choice, with the most popular being type III SS. The influence of SAS lingers on long after people have stopped using it?

I leave it to others to decide if it’s worrying that so far 15% of voters weren’t aware that there is more than one way to calculate SS for an unbalanced design…

Thanks for following up with this Jeremy, even if it doesn’t get a lot of traffic.

As of my viewing now, 42 use type III SS, but only 16 do not use R. Retrieving type III SS’s in R was not obvious to me (I think you still have to probe the net to find how to do so). Either folks are well versed in R and stats here, or perhaps there is a misconcenption about what SS are given default via R? I’m baffled.

“15% of voters weren’t aware that there is more than one way to calculate SS for an unbalanced design”

Often not a big deal, but when it matters, it really matters. Doesn’t worry me too much, as I’m an optimistic skeptic who likes to read details.

“perhaps there is a misconcenption about what SS are given default via R?”

I’m now wondering that as well. Even though our readers probably do include a decent fraction of very R-savvy and stats-savvy people.

In terms of getting R to spit out type III SS, there are add on packages that do it by default–I know lmPerm is one. But in base R, I don’t think there’s any easy way to do it, though my R-fu is weak and perhaps there’s an easy way I don’t know about.

I’ve always enjoyed Langsgrud‘s paper on it. It explained the issue in good detail, and convinced me that often what ecologists really want is type II when you press them on the real meaning of the test. Is it meaningful to have A + A*B? Is that a model you can conceive of? When pressed, the answer is often no. This is also a nice explanation.

Pingback: What we’re reading resurrected rodent teeth, the genetic origin of sex, and what’s in your ANOVA? | The Molecular Ecologist

Here’s what I think is a good explanation on what hypotheses different SS test and how to coerce R to give you what you the type of SS you want.

http://mcfromnz.wordpress.com/2011/03/02/anova-type-iiiiii-ss-explained/

Thanks!

Pingback: O que são graus de liberdade? | Laboratório de Limnologia/UFRJ