Friday links: rejected classic papers, great interview with Peter Kareiva, crowdfunding=bake sale, and more

From Jeremy:

Think you’re the only one who gets rejected? Think again, it happens to everyone. As illustrated by this paper on how even now-classic articles by Nobel Prize-winning economists initially were rejected. Similar incidents have occurred in ecology, as Meg has attested. In evolution, both of George Price’s hugely important Nature papers initially were rejected. And as The EEB and Flow notes, Joe Felsenstein’s hugely popular PHYLIP software has been repeatedly rejected for funding, both before and after it was first developed, a fact Joe memorializes in PHYLIP’s “No thanks to” list. Anyone know of any other really famous ecology & evolution work that was rejected initially? Which isn’t to say that you should always keep doggedly trying to publish an idea that’s been rejected. But deciding if/when to give up can be a difficult, and even heartbreaking, decision. (HT Paul Krugman)

BioDiverse Perspectives has a great interview with population ecologist and Nature Conservancy Chief Scientist Peter Kareiva. Much of it is on how and why to do policy-relevant science, a topic Brian hit on earlier this week. Much of what Kareiva has to say resonates with Brian’s post. Here’s an extended quote from Kareiva to give you the flavor:

Ecology matters to the general public because ecology is about water, pests and pestilence, recreation, food, resilience and so forth…focusing so narrowly on producing graphs that on the horizontal axis display number of species and on the vertical axis report some dependent ecological function (that is distantly related to human well-being) strikes me as not worth so much research…Our mistake has been to focus too much only on the one narrow dimension of nature that systematic biologists, natural historians, and a portion of ecologists care about: biodiversity.  Understand nature in a way that serves the public, not yourself. And remember, biodiversity as a label didn’t come into fashion until the late 1980s. There was a tremendous amount of conservation ecology that produced a wealth of understanding and useful insight before the biodiversity meme. My prediction is that in 2030, we will not be talking about biodiversity anywhere near as much as we do now—instead we will be asking how nature can make humans more resilient to climate disruptions, and what are the limits we should avoid crossing if we want to maintain a reliable supply of food and water.

Apparently this interview is the first in what will be an ongoing series of interviews with visiting speakers at the University of Washington. That’s a great blogging idea, it’s been done a bit before and I think it could be done a lot more. And kudos to Hillary Burgess and Halley Froelich, the grad students who conducted the interview, for having the guts to interview someone who told them that the entire BioDiverse Perspectives website was dedicated to the wrong thing!

Mike the Mad Biologist finds that there’s nothing new under the sun when it comes to scientific fraud.

Here are data on the length of the average dissertation. I leave it to you to decide if “above average” is a good or bad thing in this context.

Quote of the week, Twitter edition: Terry McGlynn says that “Crowdfunding science is as sad as a bake sale for education.” Discuss.

There may be tribes in science. But not only are there tribes in economics, they’ve been studied by an anthropologist! Just kidding of course, the linked article is a joke–a very funny one, even if you don’t know much about economics (though it’s funnier if you do know some economics). Someone should write something similar for ecology. (HT Worthwhile Canadian Initiative)

I didn’t see this until it was too late to participate, but earlier this week E. O. Wilson did an “ask me anything” on Reddit. Sadly, I don’t think anyone asked him if he reads Dynamic Ecology. And that’s probably for the best. I only skimmed it, it mostly looks like questions from admiring fans. There are a few interesting nuggets, such as Wilson’s answer to a question on the most promising fields of ecology and entomology:

I believe that the greatest leaps will be in ecology. The systems are so complex, depending on mostly little known interactions of many species that we have not begun to understand how the entirety of it works. This is a great subject for young scientists to go into, both to explore the ecosystems and define new ways to analyze them.

(Although arguably, whether ecology is “complex” depends on how you look at it) And I smiled at the anonymous evolutionary biologist who prefaced a question about the Nowak et al. kerfuffle with “You have taken more criticism in the last few years than you did over many previous decades.” To which Wilson surprisingly did not respond, “Umm, you’re aware that I wrote this, right? And that protestors at the AAAS meeting dumped a bucket of water on my head because of it?” (HT Terry McGlynn, via Twitter, and reader Artem, via email)

Species distribution modelers often choose their software based on ease of use, its use in previous publications, or on the recommendation of friends. Which the authors claim is a serious problem, but honestly, their arguments for this claim seem pretty weak. They say that trusting one’s colleagues when it comes to software choice is a risky thing to do–but peer review comes down to trust too! I mean, sure, if there’s a bug in someone’s code then that’s a problem. But there are lots of ways for a scientific study to go wrong, and bugs in code is only one of them. And this study has no data on how often it’s a problem. The authors just take survey data on how people choose their software and then leap to conclusions about the purported negative consequences of their choices. And they just assume that this problem needs solving via big changes to how students are trained and how peer review is conducted, with no attempt to balance the proposed changes against their costs, or against the benefits of the status quo.

The CEO and CFO of Plos left the company on the same day?! That’s really unusual for any organization of any size, for-profit or non-profit. I have no idea what this means. I suppose it might not mean anything, at least not anything important–or it could mean something really bad, or somewhere in between. I’m curious: are readers who admire Plos and support its goals worried by this news? Again, I have no idea if you should be worried; I’m just wondering. I mean, if, say, the Ecological Society of America announced that both the President and Treasurer of the Society were leaving their posts, effective in 10 days, I’d be a little worried. But maybe that’s a bad analogy? (HT Scholarly Kitchen)

Hoisted from the comments:

Brian, Jeremy, and ace commenter Margaret Kosmala discuss how to choose the right postdoc. Starts here.

Zombie ideas are losing the war for the intertubes

Readers of this blog know that the intermediate disturbance hypothesis (IDH) is a zombie idea that deserves to die. And now users of Google and Wikipedia are increasingly likely to learn the same thing. If you search Google for “intermediate disturbance hypothesis” here’s what you see on the first page of hits (click for larger version):

That’s right, my TREE paper attacking the IDH is the first scholarly article to come up in the main list of search results! And a second link to the same paper comes up a bit further down on the first page of results. Thanks, internet! Sadly, my paper is not among the top Google Scholar hits–yet!

And when you go to the Wikipedia page on the IDH, you find this:

Under “support and critiques”, there’s an entire paragraph devoted almost entirely to my TREE paper. I didn’t write it and don’t know who did. So thanks, anonymous Wikipedia contributor!

The notion of local-regional richness relationships is another zombie idea. And look what comes up as the very first hit if you google “local-regional richness”:

Clearly, the defenders of zombie ideas need to start blogging. Because slowly but surely, zombie ideas are losing the war for the intertubes.

p.s. I really hope these results don’t reflect Google personalizing its search results for me. That would be a bummer. I don’t think that’s the case, because my post on the zombie local-regional richness relationship is the top hit on Bing and Duck Duck Go as well, despite my never having used either of those search engines before. But personalization of search results may in part explain why my TREE article is one of Google’s top hits for “intermediate disturbance hypothesis”, since on Bing and Duck Duck Go it doesn’t show up until a bit further down in the results.

Project MOSAIC: tools for teaching mathematical modeling and statistics

Via a commenter over at Small Pond Science, just discovered Project MOSAIC, an NSF-funded project to help teach introductory undergraduate mathematical modelling, statistics, and computation. Among other things, they’ve developed a very handy-looking R package called mosaic, which simplifies the use of R for basic statistical and modeling task, and alters the output in a way designed to be friendly and people new to both statistics and to R.

I’m very interested in this as I’m currently revising our intro biostats course. Have you used the mosaic package or other Project MOSAIC resources in your own teaching? If so, please chime in with advice in the comments, or drop me a line (jefox@ucalgary.ca).

Who’s asked me to review recently, and how I’ve responded

In case anyone’s interested, here’s a list of the journals and funding agencies that have asked me for reviews since July 2011, and how I’ve responded. I don’t count reviews of invited revisions, because that’s really just a matter of completing the review of the initial ms. Nor do I count papers I handled as an editor before I resigned from the editorial board of Oikos. In agreeing to serve on an editorial board, you’re agreeing to handle any mss you’re assigned to handle; you can’t pick and choose the way you can as a peer reviewer.

I’m starting in July 2011 because I track these data for my university’s biennial performance review; the next one is this summer. I’m putting the data up now because I’m short on time to write real posts at the moment.

Journal or funding agency, # of reviews requested, # agreed to

Ecology Letters, 11, 3

American Naturalist, 7, 2

Plos One, 6, 0

Nature Communications, 5, 3

Ecology, 3, 2

Ecography, 2, 0

Theoretical Population Biology, 2, 0

Journal of Theoretical Biology, 2, 0

Methods in Ecology and Evolution, 2, 0

Proc. Roy. Soc. Lond. B, 2, 0

Journal of Ecology, 1, 1

Oikos, 1, 1

Ideas in Ecology and Evolution, 1, 1

Frontiers in Microbiology, 1, 1

Journal of Applied Phycology, 1, 0

Applied Vegetation Science, 1, 0

Global Ecology and Biogeography, 1, 0

BMC Biology, 1, 0

Plant Ecology, 1, 0

Functional Ecology, 1, 0

Microbiology and Molecular Biology Reviews, 1, 0

Landscape Ecology, 1, 0

Ecosphere, 1, 0

Oecologia, 1, 0

Frontiers in Biogeography, 1, 0

Trends in Ecology and Evolution (book review request), 1, 0

National Science Foundation (USA), 2, 1

Natural Science and Engineering Research Council (Canada), 2, 2

Marsden Fund (New Zealand), 1, 0

Some comments:

• If you’re scoring at home, that’s a total of 60 requests to review, of which I accepted 17.
• I’ve done over twice as many reviews as papers I’ve submitted or co-authored since July 2011, and that’s without counting the editorial duties I had during that time. In the past, I used to do reviewing at something like four times the rate I submit papers. My PubCred balance is very positive and growing, albeit growing slower than it once was. I’m lucky to get as many requests to review as I do, as it means I can pick and choose without worrying about whether I’m doing my fair share of reviewing.
• In only a few cases have I turned down reviews that I really wanted to do but just didn’t feel like I had the time to do. More often, I decline because the paper sounds boring, bad, or too far from my area of expertise. Of course, being busy and thinking the paper sounds boring or bad aren’t mutually exclusive. Again, I’m always busy, and I get lots of requests to review, so a paper needs to sound interesting for me to agree to review it. Note that I don’t just assume that anything submitted to a leading ecology journal like Ecology Letters or Am Nat will be interesting. I judge by reading the abstract. More than once (including just the other week…), I’ve declined reviews because it’s pretty obvious from the abstract that the authors should’ve submitted to a highly-specialized or unselective journal, but for whatever reason decided to take a shot at a journal like Am Nat.
• Speaking of far from my area of expertise…several of these review requests came from journals I don’t read, including some that came from journals I’d never heard of before receiving the request to review. I’ll let you guess which ones those were.  I declined them all, because none of those papers were anywhere near my area of expertise. This illustrates why you should always suggest reviewers in the cover letter accompanying your submission. Editors often are obliged to handle mss far from their area of expertise, and so may have a hard time identifying potential reviewers. Which sometimes leads them to send papers to reviewers who aren’t really appropriate. This also illustrates why you need to put a bit of thought into which names you suggest. I’m far from the only established ecologist who gets far more requests to review than I will ever agree to. So before you suggest me, or any established ecologist, as a reviewer, try to honestly consider whether I’m likely to do it. In general, people are very reluctant to review papers that they wouldn’t be interested in reading. So in suggesting referees for your paper, think about the audience of the journal to which you’re submitting, and whether the referees you’re suggesting are likely to be part of that audience. Someone who’s never published in the journal to which you’re submitting, or in any closely-allied journal, is probably not part of the audience for your paper, and so isn’t a good person to suggest as a reviewer.
• Yes, I do sometimes do reviews for granting agencies to which I am ineligible to apply. Basically, if I think the grant sounds interesting, I’ll review it.
• No, I don’t have anything against open access journals. I decide whether to review for Plos One, Ecosphere, and other open access journals on the same basis I decide whether to review for any journal. If Plos One ever sends me a paper that sounds sufficiently interesting, I’ll review it.
• A few of these review requests came to me at least in part because of my blogging, specifically concerning the intermediate disturbance hypothesis. Judging from the abstracts of these papers, let’s just say that there are clearly many ecologists out there who don’t read this blog.
• In the past, I’ve reviewed for a number of journals and funding bodies not on this list, and for various journals on this list for which I haven’t reviewed since July 2011.

What it takes to do policy-relevant science

I just spent last week at SESYNC which some ecologists might still know better as “the new NCEAS”. It is however different from NCEAS in at least two ways. First the first “S” in SESYNC is “Socio” – not that NCEAS didn’t do plenty of applied ecology with a human dimension, but it is right up front and part of every single project at SESYNC. The second is that the focus is on what SESYNC calls “actionable” science. As some of you know, I have a joint appointment in the Sustainability Solutions Initiative (hereafter SSI) at the University of Maine which has a very similar orientation. Of if you want another hook, the NSF SEES granting program has a very similar orientation. Loyalty demands that I point out that University of Maine’s SSI was the first of these 3 programs!

I should warn you this is a long post. This represents 3+ years of active thinking. Feel free to skim or skip if this is not a topic of interest to you.

I don’t say you need to do the kind of science these three institutions/programs are aiming for. Nor do I say that all science should be of this kind. I still have a healthy component of basic research in my own work and think this needs to keep going. But many, many scientists, especially I find earlier career scientists, want to deliver solutions. And I think I could (if I weren’t lazy) build a case using data from NSF that funding for this kind of science is on the upswing. There is an increasing discomfort with the “loading dock” model of science (do the science, put it on the loading dock, and wait for the adoring users to back up their trucks and do all the work to load it up and use it)

The rest of this post reflect my musings, hard won experience, and education by my colleagues on what it takes to deliver policy-relevant (or actionable science or solutions) .A good part of what I am writing here was learned from my colleagues in SSI*.

So without further justification, let me present my version of the 7 P’s of policy-relevant science (yes I am a sucker for alliterative memory gimmicks). . Also just to be clear I am trained and experienced as an ecologist, so that is obviously my bias, but in this piece I am saying things I believe to apply to science generically (and broadly to include all empirical quantitative researchers and maybe even all academic researchers, certainly not just biophysical science, but remember I can only really claim expertise in ecology).

1. Presentation – one common claim is that science would be used for policy decisions more if we only presented it better. This is the idea that scientists are terrible communicators. It is epitomized in Randy Olson’s movie Sizzle. I might even go so far as to say this has been the biggest effort by NSF to make work more useful – they have been conducting extensive media training under the title “Becoming the messenger” conducting workshops all over the country. Personally, I think this one is a big cop out  Are there some terribly poor science communicators? Yes. Would all scientists benefit from additional training in communication? Yes. Will the world change if all science were well communicated? NO! There are already plenty of great scientist communicators. There are also plenty of NGO organizations functioning on the boundary with professional communicators repackaging the work of scientists. Improving presentation is relatively cheap and requires little effort and change on the part of scientists – it would be nice if it were the solution. But its not.  You can put me down for this being 1% of the problem.
2. People – scientists should talk to people who are going to be affected by or care about the problems they addressing. This more commonly goes under the label “stakeholder engagement” and has been a movement building since the social activism of the 1970s and become a US government mandated part of land management under Clinton in the 1990s under the label Community Based Management or CBM. I 100% think policy-relevant science requires engagement with stake-holders (lots of engagement in most cases). However, I don’t think there is much training for scientists in how to do this. And even more importantly, I don’t think there has been much thought in how this process should occur in a way that leaves the scientist doing research instead of just turning into another activist voice at the table. Many social scientists have written research arguing that scientists are just another voice at the table. And of course many politicians are also framing scientists in this just-another-voice box to pursue their own agendas. Thus, while I think stakeholder engagement is critical, I think scientists need to take some real ownership and leadership in figuring out what this looks like and in training our peers. Iin my experience most stakeholders (and most politicians who aren’t pushing an anti-science agenda and many social scientists like Cash) also think science should not just be another voice at the table, but they don’t yet have a well-formed idea of what science-stakeholder engagement should look like either.  In my opinion David Cash has written some of the most thoughtful work on this topic (e.g. this piece which coined the idea of “loading dock” research). So to repeat myself, scientists need to be thought leaders in what stakeholder-engaged research looks like. The next 5 P’s are some of my thoughts of what this should look like.
3. Problem co-defined – this is probably the most radical but most important of the 7 P’s. The vast majority of scientific questions that get asked come from the scientists themselves. The proportion is a bit lower in natural resource departments  but even there a high proportion of the questions are scientist driven. If one steps back though, it is blindingly obvious that if one wants to deliver useful, policy-relevant science, one ought to ask potential stakeholder constituencies and policy makers what science might be useful to them! This is no different than a business finding out what their customers want. This is not to say that scientists are then obligated to do whatever the stakeholders ask. Often they ask impossible questions, questions more expensive than what they or society are willing to pay for, questions outside the expertise of the scientists talking to them, and yes, questions uninteresting to the scientists talking to them. Question definition is best done as a joint negotiation between scientists and stakeholders, and this has been called in the literature as “problem co-definition”. Despite the obviousness of the necessity for this approach, there is an equally obvious reason why it is rarely used – it involves loss of control for the scientists! it is frightening. It is a radical break from existing practice. The experience of SSI with over 20 separate projects is that every time you start a project thinking you know what the stakeholders need but then go ask them before starting, you are usually only somewhere between 0-50% right. Every single SSI project has changed their research questions in fundamental ways in response to stakeholder engagement. This approach fully embraced will radically change the kind of science that is done. That said, in SSI every scientist is still happy with the questions they co-defined and even happier with the fact they are sure the work will be useful. This might be radical, but it might not be as painful as people think at first glance. It might even be good for science for us to be pushed to pursue some questions we avoid because they’re hard!
4. Place-based – One common difference between what the researchers want and the stakeholders want is that the stakeholders want much more specific research that is useful to them. In contrast, most scientists are trained to generalize, generalize. It can seem a come down to be asked “what is the population size of deer over the last 5 years in Orono, Maine, USA?” Asking what are the general drivers of deer populations is a more scientific seeming question. And again, natural resource department researchers are already much more used to this kind of research (for which biology/EEB departments sneer at their colleagues for not doing general research and for which the natural research department folks sneer back at their colleagues for building castles in the sky). This is a real issue and research really does fall on a spectrum from highly general to place-based (and organism based and time-specific). To some degree people wanting to do policy-relevant science might just need to sacrifice a bit and move past their comfort zone to do more place-based research. But I would like to argue this dichotomy is made out to be bigger than it really is. How often does a researcher aiming for general results really span more than a handful of sites in a small geographic area at a limited point in time? This is research that is place-based even if it might also be designed to be generalizable. Many scientists can do policy-relevant place-based research funded by policy making agencies (e.g. USDA, state wildlife departments) while still writing very general science papers from the data in addition to the detailed grey-literature place-based reports they deliver.
5. Poly-disciplinary – OK – I made this word up to fit my P fetish. Interdisciplinary would be the more common word. Or the hip word would be transdisciplinary (so fully merged the individual disciplines aren’t recognizable). But I think most ecologists recognize that if you want to make the world more sustainability the biggest challenge is humanity. So you better study humanity. The NSF program Coupled-Natural Human (CNH) systems gets at this, although I think it is fair to say that not all policy-relevant science need use a CNH approach (often times the human system completely dominates the natural system and the idea of delicately balanced back-and-forth feedbacks isn’t too useful) nor do all CNH studies lead to policy-relevant science. Instead it is often important to understand the psychology of what motivates people to change, the economics and policy of creating appropriate incentives, human dynamics of population and consumption growth, technology change, etc.
6. Post-research-engagement - There is some talk in the literature about co-definition of the problems (see #3 above) and some talk about improving mass-communication (See #1 above), but I think truly successful policy-relevant science requires more – namely stakeholder engagement during research to some degree, but especially after the research is done. As science transitions into policy, there is a very important period of interpreting research. Again scientists like to pretend we don’t do interpretation, but there is no denying it happens in a policy arena. Being actively engaged with stakeholders during this stage is critical. In part it is because stakeholders can help with the communication and help us strip out jargon and avoid obvious pitfalls (like communicating climate change in degrees Celsius instead of Fahrenheit to the American public). More subtle issues of presentation are often important too – for example there have been interesting studies of “boundary objects” – things that sit on the fence between science and policy like maps and what makes them successful or not. But things beyond communication like pulling out what is important to policy, the implications for policy etc also must occur. To my mind this aspect of stake-holder engagement in interpreting research after the research is done (or at least a round of research is done) is the most overlooked and ignored part of doing policy-relevant science. This is why I prefer the phrase co-production of science with its focus on engagement with stakeholders from start-to-end over the more common phrase co-definition of problems or the other common-phrase of stakeholder-engagement that doesn’t given an explicit relationship of stakeholders to research (opening the door to the scientists are just another voice at the table paradigm)
7. Personal relationships – stakeholder-engagement has already been discussed above. But it is important to note that successful stake-holder engagement will depend directly on personal relationships built up over time and through informal contacts over meals and what not as well as formal meetings. In a boundary-spanning class I co-taught (more about this below) we brought in 20 people with experience in spanning boundaries between science and policy and literally all 20 people said that in the end personal relationships were the most important thing. This is pretty different from science with its efforts (although never fully successful) to focus on objective progress independent of personal links. But if a legislator is going to take a vote based on science, you can bet they want to know the person who did the science and trust them.

So to do policy relevant science, all you have to do is spend a lot of time  with stakeholders, let them have 50% of the job of deciding what research to do, study complex social behaviors in addition biophysical sciences, become more place-based, and open up the research interpretation process to other people and invest time in building relationships with them. No problem, knock it off with a few extra hours of work, right? Of course not! Policy-relevant science involves a fundamental change to the way science is done (again assuming policy relevance is your goal – not all science should have this goal). I have argued above that some aspects like the place-based research and the stakeholder co-defined research need not be as frightening as they seem. But in total, there is no denying doing policy-relevant science is a lot of work! And a big change for many scientists.

So for those scientists who are making this change, or for those educators and administrators trying to facilitate this change, how should the change happen? As always, I am ready with a silly mnemonic. The four T-s of transition.

• Training – This should be obvious to professors, but in truth we have a blind spot – we have gotten so good at teaching ourselves things related to our discipline that we think we can teach ourselves anything. Or that science is the hard part, policy is easy to pick up by osmosis (this is obviously false). If we want scientists to do policy-relevant science we need to train them in skills related to the 7 P’s and this includes soft skills like facilitation  To my mind this can be best summarized as the task of boundary-spanning. A transformative event for me (and I think the 3 colleagues I co-taught the course with – David Hart, Laura Lindenfeld and Kathleen Bell) was teaching a 1 week intensive course in boundary spanning for graduate students. In addition to reading some of the literature and theory of boundary spanning (probably a whole other post in there), we brought in panels of people who successfully span boundaries and had them speak about what they saw as key success factors. But whatever you call it, there are a lot of skills that boundary-spanning scientists doing policy-relevant science need. Media training (P #1) is just the tip of the iceberg.
• Teams – the biggest error in thinking is that one person can or should do all 7 P’s. It is a rather laughable idea actually when you really stop to think about it. This means that policy-relevant science almost always happens in teams. In these teams everybody needs to have the rudiments of the 7P’s but different individuals with different strengths and expertise can complement each other to actually fulfill the full range of the 7 P’s.
• Tag teams- policy relevant science can also happen by one person doing one piece and making it available in a public manner and then somebody else picking it up and doing another piece, not unlike children’s tag-teams where when you tag somebody you go out and they step in to do something (or WWF wrestling if you prefer). People can do this without ever thinking of themselves as a team or ever even meeting. There is obviously a continuum from strong teams through weak coalitions to tag-team processes. And tag-teams are dangerously close to the loading dock model that was derided above and in the piece linked to above by Cash. Thus I expect the majority of successful policy-making will occur closer to the team than the tag end of the spectrum. But some good policy-relevant science happens through tag if people do it in a thoughtful way. And no doubt this approach can be used to channel more basic research in useful ways. As an extreme example you don’t need stakeholder engagement to know that cold-fusion or understanding the causes of extinction are important research topics. Just don’t use this as a cop-out to avoid teams.
• Thinking – I’m talking about big changes. Scientists need to apply their academic approach of discussion, analysis and data to the 7 P’s and success factors in doing policy-relevant science. Social scientists have been the ones studying and writing about this problem most to date. Many of them are doing a good job. A few of them in my opinion don’t get biophysical research at all. But it is kind of ironic that if the ideas which emerge are things like teams and inter-disciplinarity that the topic of how to policy-relevant science is not team-based and inter-disciplinarity. A full understanding of how best to do policy-relevant science needs to include the biophysical scientist’s expertise and perspectives too. This has not been happening enough.

You can call everything I have talked about boundary spanning or co-production of science (as Cash that I cited above does). I think both of these phrases are useful. But whatever you call it, it is a lot of work and the world needs more of it. And as a scientist trying to learn this material, I found very little published material (journal articles or otherwise  to help me learn. That needs to change.

I am very curious to hear what others think. Again I don’t want this to turn into a debate about whether policy-relevant science should happen and scientists should be part of it. They should but not everybody should be forced to do so – basic science is important to. For those who would describe themselves as already engaged in policy relevant science, what did I get right or wrong? For those just starting out, was this helpful? what are your impressions? For those who have never done policy relevant science, does this scare you away or appeal to you?

* Including amongst many others David Hart, Tim Waring, Kathleen Bell, Mac Hunter, Aram Calhoun, Laura Lindenfeld and Shaleen Jain

“Null” and “neutral” models are overrated

Recently I reviewed an interesting paper proposing a new model of X.* X is an ecological phenomenon that we’d like to understand and predict. X has been modeled before in various ways, with different models making different ecological assumptions about the factors that govern X, and different simplifying assumptions about other things. The main goal of this new paper was to develop a simple model of the effects on X of some factors ignored by previous models. All of which is fine (like I said, I found the new model quite interesting), and none of which is what prompted this post.

What prompted this post was that, in a couple of places, the authors referred to their new model of X as a “neutral” model. I admit that I wasn’t 100% clear on what they meant by this. But I got the impression that the authors felt their model had some sort of special status compared to previous models of X. That they viewed their model as a “limiting” or “baseline” case, perhaps–the factors included in their model are always at work, whereas the factors included in other models might or might not be at work. Or perhaps they felt that their model should be treated as a “null” model, to be tested and rejected before we are entitled to infer that some other process, not included in the model, matters? As I say, I’m not clear exactly what they meant, and the authors didn’t make a big deal of it so it wasn’t a huge concern for me

But this paper is just one example of what seems to me to be a growing trend, although its roots go way back. In the wake of Steve Hubbell’s very influential application of a neutral population genetics model to ecology, ecologists seem increasingly keen to develop “neutral” or “null” models for all sorts of ecological phenomena. In practice, this usually means a simple model which omits, or sets to zero, the effects of one or more ecological factors or processes, while explicitly or implicitly retaining the effects of other factors or processes. Just as neutral models in population genetics set selection to zero, but include (or can include) effects of other evolutionary forces, like mutation, migration, and drift. And then it’s claimed or implied that the resulting model has some sort of special status, that it’s somehow different than other models of the same phenomenon, and so should be treated differently.

This trend kind of bugs me. Developing ecological models that omit or set to zero the effects of some ecological processes often is very useful, I have no problem with that. But I really wish we’d quit calling the resulting models “neutral” or “null” models, and treating them differently than we treat other models on which we haven’t slapped those labels.

The issue here is one of which research strategies are effective in which contexts, or for which purposes. There absolutely are contexts in which it makes sense to treat some particular simple model as a “null” model, which ought to be rejected as a first step, before we are entitled to infer the operation of any processes or factors not included in that particular model. But there are many other contexts in which that research strategy is not only ineffective, but likely to be positively misleading.

To explain why, let’s consider a canonical case in which it really does make sense to start with a null model that you will try to reject before doing anything else. In simple statistical contexts, the null hypothesis describes how you’d expect the data to look if there was nothing going on except sampling error. Sampling error is of no scientific interest. It’s a nuisance, pure and simple. If we could completely and accurately census the statistical populations of interest, we would. But unfortunately, complete and accurate censuses ordinarily are impossible, so sampling error is ubiquitous. Further, its effects aren’t always obvious or easily recognized. So in order to avoid getting fooled into seeing patterns that aren’t really there, it makes sense to first rule out the possibility that any apparent patterns in the data arose from sampling error alone. And in order to do this we need to be as sure as we can be that our null hypothesis correctly describes the effects of sampling error, and doesn’t include the effects of anything else besides sampling error. Because otherwise we will be seriously misled.

Of course, sampling error isn’t the only possible “nuisance” in science. A “nuisance” could be any factor that, for whatever reason, is totally irrelevant to the question being asked. So in general, we can say that a “null” model is one that includes the effects of any “nuisance” processes or factors that are of no scientific interest, but just get in the way of detecting effects that are of scientific interest. Unfortunately, these “nuisances” are ubiquitous or nearly so (otherwise why would we worry about them?), and have non-obvious effects (otherwise why would we need to model them to detect them?) To be useful, the null model must correctly describe the effects of these “nuisances”, and must not include any effects of any non-nuisance factors. Indeed, insofar as the null model doesn’t correctly describe the effects of “nuisances”, or includes effects of non-nuisances, it can be worse than useless. It can be positively misleading. And of course, all of this assumes that we can all agree on what’s a “nuisance”, for purposes of the question asked.

In practice, I think “neutral” models in ecology often are intended to function as “null” models in the sense just described. Which is a big problem, I think. Because can you think of any ecological model (as opposed to a statistical model of sampling error) that actually fits the description I just gave? I can’t.

For instance, all neutral (in the sense of selection-free) models of which I’m aware include the effects of other processes of scientific interest–drift, migration, mutation, etc. These processes are of interest both in their own right, and due to their interactions with selection. And further, those other processes aren’t necessarily ubiquitous; there are real-world situations in which some or all of drift, mutation, and migration are negligible. And further still, different models omitting different processes often can produce similar-looking data. This is a really crucial point. For instance, there are models with selection but no drift, mutation, or migration that produce realistic species-abundance distributions. When the world is overdetermined, it is a very bad research strategy to default to assuming that certain processes matter while others might or might not. And in ecology, the world often is overdetermined, by which I mean simply that many different combinations of processes are sufficient to generate the observed data, with no one of them being necessary. So if you’re trying to understand the processes that generated your data, I don’t see why you’d ordinarily want to confer special “null” status on a model omitting any one of those processes. Not when that “null” model is simply one model among others that might have generated the data.

But at least neutral models in population genetics do in fact omit selection, while retaining drift, migrations, etc. Many other putatively “neutral” or “null” models in ecology don’t even manage that. For instance, randomization-based “null” models for detecting effects of interspecific competition are infamously problematic because it’s totally unclear what effects they actually eliminate and what effects they retain. As a second example, the “mid-domain effect” is a strange “null” model that admittedly nullifies only some of the effects of environmental gradients on species’ geographic ranges. I could keep going, but you get the idea.

I sometimes see ecologists argue that one always has to have a null model. You always have to rule out “noise” before you can claim that there’s a “signal” worth studying. One problem with this argument is that it gets deployed in contexts in which what counts as “noise” is highly debatable. If by “noise” you mean, not “sampling error”, but “ecological processes that I personally happen not to be interested in”, you really should not be deploying this argument. A second problem with this argument is that it’s deployed to defend null models that the users themselves admit are imperfect, e.g., because they include effects of “non-nuisance” processes. Again, having a bad null model often is worse than not having one at all, because it’s positively misleading. In such cases, your best bet is to find some other way of addressing the scientific question of interest. For instance, back in the 1980s community ecologists famously abandoned randomization-based null models and other observational approaches for inferring the operation of competition, in favor of field removal experiments to directly test for competition.

I also sometimes see ecologists giving special status to simple “null” models on grounds of parsimony. I don’t buy that. I wonder if people who make this argument have thought sufficiently carefully about precisely what “parsimony” means and why we might care about it. (There is an extensive philosophical literature on this) Personally, I generally don’t care about simplicity (parsimony) for its own sake. I care about the truth, or at least a good enough approximation to the truth for my purposes. And the truth, or a good enough approximation to it, might well be complicated! For instance, if the truth is that the world is not neutral, so that selection is among the processes that actually generated my data, why should I care if a simple model that omits selection can reproduce certain features of my data? Especially since, thanks to overdetermination, different “null” or “neutral” models that omit different factors often will all be able to reproduce those same features of my data. Which means you can’t argue that the factors omitted from any one of those models are irrelevant (too often, “parsimony” is invoked not as a substantive argument but simply as a way to shift the burden of proof) And if you say that simpler models are to be preferred only when all else is equal, you’ve just admitted that parsimony is irrelevant in practice, since in practice all else is never equal when it comes to comparison of substantive scientific models. Bottom line: the reasons for favoring simple models over complex ones, independent of how close they are to the truth, are extremely limited at best.**

None of the above is intended as an argument against statistical hypothesis testing in ecology. Even in an overdetermined world, it still often makes good scientific sense to start by ruling out the possibility that your data could’ve arisen from pure sampling error. Traditional statistical ideas about sampling error are pretty much always relevant.

Don’t get me wrong, I know as well as anyone that all models are false, are imperfect approximations to the unknown and unknowable truth. And there absolutely are good reasons why, when trying to learn about how the world works, we might want to start by developing and testing simple models rather than starting out with more complex ones. This post is emphatically not an argument that we should aim to develop literally-true models (that’s impossible), or models that are as complex as possible! But the whole point of having a false model, or a bunch of different false models, is to home in on the particular ways in which they’re false, and leverage those falsehoods to get closer to the truth. Too often, that’s not how purportedly “neutral” or “null” ecological models are used. It’s usually a bad research strategy to set up one particular model among others as a “null”, just because it happens to be simpler than the others or just because it omits some particular process that other models include. It’s often far more useful to start with a suite of alternative models, none of them privileged with the label “null”, in order to get a sense of the range of models that might have generated the data (e.g., the recent work of Storch et al., to pick one possible example among many).

*Obviously, I can’t go into any further detail without violating confidentiality.

**As illustrated by the fact that popular statistical methods for model selection, such as AIC, are not methods for choosing “parsimonious” models. They’re not methods for choosing “simple” models, independent of how close they are to the truth. They’re not even methods for choosing models that represent some sort of optimal “compromise” between simplicity and closeness to the truth, though they’re often described that way. Rather, they are methods for choosing the model that’s closest to the truth, period. A model can be false by being simpler than the truth, or by being more complex than the truth (as in cases of “overfitting” the observed data, also known as “fitting the noise”). That, and not “parsimony”, is why AIC includes a penalty term for the number of free parameters a model has. AIC scores for alternative models are estimates of the relative Kullback-Leibler divergence between the alternative models under consideration, and the unknown true model that generated the data.

Friday links: s**t students write, do big name scientists have too much money, and more

From Jeremy:

S**t my students write: a Tumblr compilation of hilariously-bad passages from student essays. My favorite line: “Scientists are well educated and don’t make mistakes because they have their degrees and what not.” Yup, that’s me: Jeremy Fox, Ph.D., A.W.N. I have to say, I’m glad my student days were almost behind me before the internet took off, so that all the amusingly-awful poetry I wrote in high school could only be mocked by my classmates rather than being immortalized on Tumblr.

Canadian government continues to slash and burn basic science: with no warning, world-famous Bamfield Marine Science Centre just had its research budget cut by 1/3. A bunch of other major research facilities in all fields of science are getting cut too, in favor of increased funding for “applied” research relevant to industry. Funny how a conservative government that professes to believe in the power of unfettered free markets also believes that the government ought to intervene in the market by subsidizing the sort of industrial R&D that you’d think private business would pay for itself if it was really so relevant to them. And how they don’t want to pay for the sort of basic research and other public goods that the free market has never provided. Although, as The Monkey Cage notes, they might change their tune if they could be convinced that basic research will save us from giant space rocks. Or if basic research could be pitched as creating jobs in the districts of key legislators. Wonder why so many basic researchers are hypocrites when it comes to justifying public funding for their work? This is why.

Pell Grants for low-income students are actually making private US colleges and universities less affordable for those students. That’s because of the way the majority of US colleges and universities structure their tuition and financial aid packages, with the linked goals of making money and attracting “elite” students. Effectively, many colleges and universities are using Pell Grant money to help subsidize merit scholarships that mostly go to students from high-income families, while pricing themselves so as to avoid admitting too many really needy students. There’s an amazing interactive chart if you want to check the numbers for your own private college or university. I was reassured that my own undergraduate college, Williams, is among the few that do it right. Williams offers sufficiently-generous need-based aid that the neediest students pay a low effective price, and has a relatively high proportion of highly-needy students in the student body. Although I was embarrassed to see that our biggest rival, Amherst, does better than we do. And I do wonder if both schools, and others like them, could be doing better still (further up and to the left in that interactive chart), and without even sacrificing student “merit”, by doing a better job of seeking out needy applicants. Many financially-needy students never even apply to places like Williams and Amherst, assuming (incorrectly) that they couldn’t afford them.

2-3% of NSF grant applications have “actionable plagiarism”, and the rate for applications from young investigators is 10-15%?! Is this true? Anyone have more info on this? (HT Retraction Watch, via Twitter)

Further to my post earlier this week on pseudoscience, here’s the text of physicist Richard Feynman’s classic speech on “cargo cult science”. It’s most famous for Feynman’s line “The first principle is that you must not fool yourself–and you are the easiest person to fool.” But the whole thing is well worth your time (HT to a commenter on Andrew Gelman’s blog).

The Lab and Field crunches the numbers and questions the value of Canada’s Excellence Research Chairs Program (a program to help Canadian universities attract big-name scientists). There are two problems with throwing huge amounts of money at small numbers of scientists. First, it’s risky–it amounts to putting all of your eggs in very few baskets. Second, after a certain point it’s surely inefficient, because those big-name scientists stop being money-limited and become time-limited. You’d get more bang for your buck if you gave that money to people who don’t have as much. And indeed, my former Oikos Blog colleague Chris Lortie compiled evidence that the most elite ecologists really do have more money than they can spend effectively, so that funding agencies would get more bang for their buck by reallocating funding away from the “ecological 1%”. For additional discussion of how funding agencies should allocate their funding, with a focus on the virtues of the Canadian Discovery Grant program, see here and here.

As reported in Nature this week, a Rutgers University report has backed famed evolutionary biologist Robert Trivers’ arguments that one of his co-authors falsified the data in a famous Nature paper of theirs. I’ve discussed this case before (see comment thread here). The guilty party, psychologist William Brown, continues to protest his innocence on grounds that manage to be convenient, vague, and implausible all at once (Despite the fact that you were corresponding author, you lost the original hard copies of the data and your electronic copy inexplicably got corrupted, you say? Uh huh. And how long did it take your dog to recover after eating your homework?) Personally, I find Trivers’ evidence (which he self-published as a free book after failing to convince Nature to issue a detailed retraction) and the Rutgers report overwhelming. Worth noting that this is another case in which a graduate student’s attempts to reproduce published analyses led to discovery of fatal (and in this case, clearly intentional) flaws in the published work. Finally, kudos to Robert Trivers for setting a model example of what to do when you suspect that there’s something wrong with your data (and wrong with your collaborators).

You should’ve gotten your PhD in economics instead of whatever it is you actually studied.

Of course you can use the Price equation to model the evolution and adaptation of the entire universe. Which brings to mind an old Weird Al Yankovic lyric, “You can even cut a tin can with it…but you wouldn’t want to!” And I say that fondly–I’m a fan of the Price equation, and of the lead author of this paper.

And finally, a large brood of 17 year cicadas is about to emerge on the US east coast, and The Onion is on it.

From Meg:

Terry McGlynn has a new post on teaching philosophies. He has written it with someone who will be on the job market in mind, but I think it’s also useful for those of us (myself included) working on the teaching statement portion of a tenure dossier. I am starting mine off with a section on my teaching philosophy. Right now, I sum up my teaching philosophy as “students learn best when they are actively engaged with the material”, followed by examples of how I encourage students to engage with the material in different types of classes.

Evolutionary biologist Jeremy Yoder is surveying LGBT folks working in STEM fields. He says the goal of the survey is “to answer the questions we have about queer folks in STEM: who we are, what we study, and how our identities have shaped our interest in science and our experiences of working in research.” You can find the link to the survey in his blog post.

Here’s a great resource on gendered words and letters of recommendation. Studies indicate that we (and, yes, “we” includes women!) are more likely to use words related to teaching or working hard when describing women, and words related to ability and research when describing men. This page gives examples of these different types of words, and includes this excellent advice: “When writing letters of recommendation for women, it is important to keep these associations in mind and purposefully use standout, ability and research words to describe qualified female candidates.”