Also this week: tell me again when to use fixed vs. random effects, why everything higher education costs so much these days, why you should distrust conclusions based on many small studies, and more.
From Meg:
Science might save my daughter. Don’t kill it. This is such a powerful, important piece by Alan Townsend. Read it, with tissues handy. (ht: Terry McGlynn)
From Jeremy:
Nobel Prize-winning psychologist Dan Kahneman now thinks that the entire field of social priming research went off the rails and that he himself was far too quick to accept and publicize its conclusions. Here’s his mea culpa, and diagnosis of what went wrong. tl;dr: you should be skeptical of a conclusion to which lots of small studies point, even if they come from different labs. That is, the high number of studies together with their unanimity is a reason to be suspicious of their conclusions. That’s because unanimity of published underpowered studies points to some combination of a severe file-drawer problem and severe p-hacking. This may have some implications for stereotype threat, a form of social priming on which we’ve posted in the past. Click through to Kahneman’s comments even if you don’t care about social psychology, because the issue is much more general than that. In many fields, including ecology, lots of conventional wisdom is based on many low-powered studies that all seem to point in the same general direction. Kahneman also provides a model example of a scientist saying “I was wrong”. He’s basically retracting an entire chapter of his influential bestseller Thinking, Fast and Slow–not an easy thing to do.
Long but interesting and very accessible read on “cost disease“: why the cost of certain things grows much faster than inflation for long periods. US examples of such things include higher education (which is why I’m linking to this), health care, and public transportation infrastructure. And here’s the thing: obvious candidate explanations like “those things are rising in quality”, the Baumol effect, and the sorts of explanations favored by political partisans of any stripe, don’t really fit the data. No doubt one could quibble with the discussion of this or that particular case, but the overall picture made me stop and think. (ht @noahpinion)
Unlearning descriptive statistics. Cogent argument that your choice of summary statistic should depend on whether you’re planning to do statistical inference about population parameters, vs. just trying to summarize the sample in an easily-interpretable way. (ht Small Pond Science. )
Two pieces (here and here) on the importance of not fighting political tribalism with tribalism. (ht Small Pond Science, Economist’s View)
Semi-relatedly, Margaret Kosmala on dipping her toes into political activism.
An econometrician on fixed vs. random effects. Apparently, econometricians teach their students that:
You should use random effects when your variable of interest is orthogonal to the error term; if there is any doubt and you think your variable of interest is not orthogonal to the error term, use fixed effects…Random effects should really only be used when the variable of interest is (as good as) randomly assigned.
Brian’s contrasting view on fixed vs. random effects is here. Discuss.
I’m from rural Pennsylvania, so trust me when I tell you that this is the most rural Pennsylvania story ever. 🙂 What seems like a highly-improbable confluence of events actually is a highly-probable confluence of events once you condition on the fact that the confluence of events happened in rural Pennsylvania. I say this fondly, by the way.
And finally, I am going to stop taking writing advice from Brian and start taking it from the kid who made this poster:
(ht @kjhealy) 🙂
Dynamic Ecology 
I enjoyed the Pennsylvania deer story. But in New Brunswick, when this happens, we just open the doors to the store and deer walk out again. I guess we raise them more polite? http://www.cbc.ca/news/canada/new-brunswick/deer-grand-falls-canadian-tire-video-1.3958208
Concerning the random effect post. I have no idea where this orthogonal story comes from, searching my mixed effect bible on this concept (Gelman and Hill 2007), I could not find any reference to orthogonality … In the case where the variable of interest (as the author name it) is not randomly assigned one can still include covariates in the model to account for potential interactions. I also did not get his second point: “the fact that the FE and RE results look a lot alike […] is confirmation of the fact that the variables on the RHS are orthogonal to the error term more than anything else, and that this says absolutely nothing about external validity. Thus, to claim that this makes it possible to make inference about the whole population is also wrong.”, well I thought that statistics is always about making inference for some population based on representative samples, this has nothing to do with the random – fixed effect artificial divide.
I find Brian’s decision tree way more helpful. Also I lean more and more towards Gelman’s arguments on this topic, that it is way more helpful to envision random effect as parameters varying depending on some higher-level processes.
“you should be skeptical of a conclusion to which lots of small studies point, even if they come from different labs. That is, the high number of studies together with their unanimity is a reason to be suspicious of their conclusions. That’s because unanimity of published underpowered studies points to some combination of a severe file-drawer problem and severe p-hacking.” => This describes my skepticism about biodiversity-ecosystem functioning research in Ecology. Very few labs have been willing or able to publish BEF experiments that find no or weak richness effects. We’ve published a few, but it was tough going, e.g. the editor et Ecology rejected this (http://onlinelibrary.wiley.com/doi/10.1111/j.1365-2656.2009.01533.x/abstract) paper saying it was a “failed experiment”. And of course its poorly cited (and sometimes not included in meta-analyses) compared to our experiments that DO find richness effects. We ran dozens of these experiments in all kinds of systems and locations and rarely did richness really matter (composition was FAR more important). I suspect this is common but that there is strong self-selection to publish results that fit the paradigm. How likely is it that 99 out of 100 experiments performed in ecology find evidence for an effect being tested for? More broadly, what proportion of ecologists ever publish null results? I wouldn’t be surprised if Jeremy has estimated this.
I haven’t estimated this but thanks for the suggestion!
More broadly, psychologists have come up with various indices to estimate the seriousness of file drawer problems and p-hacking. It’d be interesting to apply these in ecology. At a guess, I suspect these issues aren’t as serious in ecology as in social psychology, but that they are issues.
Ah, the ever-present (pseudo-)philosophical debate over when to use random effects. I am currently revising a manuscript where reviewers questioned the utility of accounting for repeated observations on individuals using a random intercept, and it is sort of troubling how widely advice varies. And with the proliferation of mixed models using R package lme4 (which are very useful – don’t get me wrong!) it is becoming increasingly difficult to discriminate between proper and improper use of random intercepts and/or slopes.
I’m a baffled by the claims in the “Don’t Kill Science” linked story about science spending. Wikipedia shows the latest R&D spending data available for most countries. The US is the leading world spender on R&D ($473B, 2013) , with China next ($409B 2015, 85% of US), then the EU ($334B; 70% of US) and Japan ($170B 36% of US). (“List of countries by research and development spending”).
And according to the time-series data linked to in the article, characterizing US spending on non-defense research as “flat” over any time period is wrong (the link “Federal support for research has been flat”…using the third series data set “By Function: Defense and Nondefense R&D, 1953-2017”). US Non-Defense R&D (NDRD) spending rose by 25% from 2000-2005, then by another 15% 2005-2011 – an overall 50% jump in 11 years – then fell by about 10% through 2015 and was up in 2016. From 1999-2015 the annualized growth in non-defense R&D spending was 1.9%, *despite* two extreme economic downturns. Going back to 1983, the low point in NDRD during the early 1980s recession, NDRD spending has risen at an annual clip of 2.6%.
Both the rise since 2000 and the rise since 1983 are all the more remarkable in that they occurred during a period of long-term decline in US GDP growth. So even as the growth in our economic output shrinks, we spend more of that output R&D. That’s remarkable, and not in bad way.
Last but not least, health, and surely cancer, has received the bulk of the increase in Federal R&D dollars over the last four decades. The health research budget has increased at 3.6% annually, nearly 4x the rate of the “general science” budget since 1976 – and that’s after the big pullback in spending from 2010-2015.
It’s worth pulling down the data and plotting it up to verify for yourself. It’s quite contrary to what’s often portrayed in the media.
With respect, it seems to me that you are jumping around from one data set and comparison to another, cherry picking comparisons and time periods as needed to support your views. For instance:
-You sometimes quote data on total R&D, not federal government R&D. Total R&D in the US is dominated by private sector spending unrelated to science.
-Federal government R&D in the US also includes a lot of defense spending. Again, not related to science.
-Comparisons of total spending among countries are meaningless unless you adjust for the size of the country’s economies.
Thanks for the links. The piece by Kahneman is very thought provoking. If we keep measuring the same things again and again, or measure a lot of things and search for significance, we will eventually find significant relationship or effects almost for sure. Part of this is publication bias and part of this is the limitation of using hypothesis falsification fashion of doing science. If bias in what is reported in literature exist, the fact that many studies points to the same direction may not be evidence for it. It could actually be evidence of such bias. Maybe examining the data with more mechanistic modeling/understanding, as you have discussed in this blog before, is a useful way to tackle this problem. If multiple aspects of the model are supported by the data, it seems we will be more certain that it is real, not the results of significance searching.