Friday links: a morality tale about p-hacking, field notes from the future, NSF 101, and more

Also this week: Wikipedia vs. its own success, liberal activism vs. science, how to fund high risk, high reward research, Robert Trivers’ wild life, new ggplot2, and (much) more!

From Jeremy:

A bit late to this, but that’s ok, because it’s great: a Christmas morality tale about researcher degrees of freedom (or p-hacking; call it what you will). Read this for your next lab group meeting and discuss what you’d do in that situation–and more importantly, how you’d avoid ending up in that situation in the first place. I’m also mulling over the difficult collective action problem hinted at at the end of the piece:

To let this result enter the literature wouldn’t have a significant effect, it wouldn’t cause anybody to go down an experimental path that they weren’t going to anyway, and it didn’t rule out any life-saving approaches. There was no point being puritanical about it.

Multiply that completely understandable line of thought by the number of labs in the world to get a rough sense of the scale of the collective action problem here. Put bluntly: most individual bits of science have little or no impact, a fact of which most scientists are well aware. Does that select for a collective lowering of standards? I’ve asked that question before, but back then I hadn’t cottoned on to the collective action problem here. It’s true that one mistaken paper rarely matters much–but if lots of papers are mistaken, and if knowledge that no one paper matters increases the frequency and seriousness of mistakes in papers, then in aggregate mistaken papers might matter a lot.

Andrew Hendry vs. Alex Bond on the proposition that anyone with a decent record can get an academic job, if only they’re persistent and not too picky about their first job. I think both have some good points. Do yourself a favor and read the comments on both pieces too. I may have more on this in a future post (sadly, no time to post on this now, I’m swamped).

Stephen Heard reflects on his first year of blogging. Keep it up Stephen!

This is up my alley: Extinct is a new group blog on the philosophy of paleontology. For instance, here’s my Calgary colleague Adrian Currie arguing that we should resist the lure of simple narratives about why evolutionary history played out as it did (wonder if Robert Trivers, highlighted below, would agree?) Accessible and thoughtful.

Grumpy Geophysicist argues that NSF isn’t funding sufficiently risky work, because work that challenges current views doesn’t get funded. Do folks here agree? At the end, there’s an interesting suggestion about how to identify high risk, high reward work:

Long ago, GG was told that Eldridge Moores had a policy back when he edited the journal Geology; his preference (so it was told) was to publish papers producing one review recommending “accept without changes” and another recommending rejection.  Although this occasionally produced papers that, in retrospect, fully justified the “reject” recommendation, the basic idea had merit: it could be good to focus on the papers that so challenged part of the community that they felt those papers should be squashed. Perhaps this is apocryphal, but it seems worth considering.  Maybe program managers, instead of having panelists vote on a grade for a proposal, should simply time how long they want to argue about any given proposal and fund the ones producing the longest arguments.

One downside of funding the most controversial work is that, if a field is divided into opposing camps, you end up funding lots of work that just perpetuates the argument rather than resolving it. More broadly, I think “riskiness” isn’t unidimensional and can be tricky to judge; see for instance my interview with Rich Lenski on whether the LTEE was a low-risk or high-risk experiment. Related: my old post on “experiments so crazy they just might work”. (ht Retraction Watch)

Empiricism is replacing theory in economics. Includes time series data on the types of papers published by leading econ journals. Always interesting to read about trends in other fields that share some of ecology’s features. Stick with it to the end for some interesting comments on economics’ last fling with empiricism and why it was abandoned.

Next time you complain about Wikipedia getting something wrong and someone responds by saying “Well, why don’t you just edit it yourself?”, feel free to reply “Because Wikipedia doesn’t want me to” and link to this paper. Wikipedia is an amazing project and absolutely has its uses. But the days of Wikipedia as an open resource anyone could edit are long gone in practice, even if they’re still there in principle. Nowadays, there is no hypocrisy or selfishness in (i) making use of Wikipedia, (ii) complaining when it’s wrong or unhelpful, and (iii) not editing it yourself. (ht Marginal Revolution) Related: Meg’s cautionary tale of having students edit Wikipedia as an assignment.

Here’s Jesse Singal’s review of Alice Dreger’s Galileo’s Middle Finger: Heretics, Activists, and the Search for Justice in Science. Sounds interesting and important. It’s about cases in which collisions between science and liberal activism have driven both off the rails (in contrast to the anti-science conservatives scientists usually worry about). I don’t think it’s an exercise in “both sides do it” false balance to worry about such cases. Other reviews from the New York Times, the journal Human Nature, writer Clinton Peters, and science journalist Ellen Rupert Shell (those last two thoughtfully critical in places).

Paul Krugman comments on time series data on the political leanings of US academics, including scientists. US academics as a group identify more with the left than they used to. For reasons Krugman lists, I’m mostly not too worried about this issue in the context of science. Kevin Drum weighs in too. He says we shouldn’t need to argue about why the trend is happening, because the various hypotheses are easily testable. (p.s. my understanding is that the data discussed do not come from a random sample, so treat with caution)

NSF 101. From Margaret Kosmala. Useful for grad students and postdocs.

Prediction vs. causal inference in regression. Nice short practical overview of the differences between these two uses of regression. (ht Jim Grace, via the comments)

Summary and commentary on the Ontario provincial government’s final consultation report on a new funding formula for its universities. Short version: big changes are coming. Funding is likely to be decoupled from undergraduate enrollments and instead tied to some yet-to-be-determined formula based on outcomes and student satisfaction. And the provincial government is likely to get more directly involved in managing at least some universities.

A profile of Robert Trivers, based on his forthcoming memoir Wild Life: Adventures of an Evolutionary Biologist. If you only read one book this year, this should probably be it. Don’t believe me? Quoting from the blurb:

But unlike other renowned scientists, Trivers has spent time behind bars, drove a getaway car for Huey P. Newton, and founded an armed group in Jamaica to protect gay men from mob violence.

and the profile:

His dissertation was so strong that when he showed up before the evaluating committee, which included such luminaries as E. O. Wilson and Ernst Mayr, they skipped the charade of making him defend it and simply offered their congratulations.

and the profile again:

His college roommates once showed him pictures of a hippo and a rhino and asked him to identify which was which. He picked wrong.

Oh, and he decided to become an evolutionary biologist after failing to get into law school and taking a job writing children’s books about biology. (ht: Marginal Revolution)

And finally, field notes from the future. Very funny, and not a little thought provoking. I like the mega-apology for Megalodon. (ht Stephen Heard, via Twitter)

From Meg:
Jacquelyn Gill has a list of 10 straightforward things you can do to support diversity in academia in 2016. Lots of good ideas there!

There’s a new version of ggplot2, which sounds like it’s a pretty major update.

17 thoughts on “Friday links: a morality tale about p-hacking, field notes from the future, NSF 101, and more

  1. Hello Jeremy,

    I think that you’re right that if there’s a risk that a million slightly shaky but not fraudulent conclusions might create a weight of minor problems that adds up to the point where it affects the integrity of the literature.

    I think that at the heart of it, we have a series of perverse incentives in science. Particularly, the need to publish articles with high impact conclusions makes it impossible for scientists to be dispassionate about their hypotheses and it can even punish rigour.

    Researchers aren’t super-human, they have lives, aspirations and families to feed, just like everybody else. The idea that scientists might compromise just a little seems shocking until you look at the circumstances in which it happens and as you say, it becomes understandable. Under those conditions, there’s inevitably going to be a gentle downward pressure on standards.

  2. That piece on Wikipedia chimes with my own recent experience. I’ve constructed and edited pages on Wikipedia on-and-off now since about 2009 and last year finally got round to submitting a page about the history of the annual SCAPE conferences which have been running for almost 30 years now. Within weeks it had been rejected as “not notable” enough; I argued the point but the administrators wouldn’t budge. It was very frustrating. Eventually I decided just to host the page on my blog:

  3. The Trivers autobiography sounds amazing!

    Slightly related, there’s a new book: “Issues in Palaeobiology: a Global View”. It’s a series of five questions asked to paleontologists from over a dozen countries across five continents. Most of the questions are about where they see the field going and what they do, but the final question is about how they became a paleontologist at all. Some of them are really good! One chap ended up as a paleontologist only because he missed an interview to get into med school due road closure resulting from a battle in the Lebanese civil war….

  4. As a counterpoint, here is a compilation of posts from Julia Serano, a trans activist who was a developmental biologist for many years, criticizing Alice Dreger’s characterization of the Bailey book and the controversy around it, and chronicling Dreger’s history of attempts to discredit trans activism (she supports conversion “therapy” for trans kids!). It has links to numerous peer commentaries about that part of the book (including a very critical one from the respected neuroscientist Ben Barres), review articles and other peer-reviewed articles pertaining to the actual science related to Bailey’s claims.

    Here’s a link to the issue of Archives of Sexual Behavior that contains both Dreger’s article on the Bailey book that eventually became the relevant chapter of Galileo’s Middle Finger, and the set of peer commentaries, some supportive and some critical, that were published with it.

    I’m aware that most people don’t know any trans people (that they know of), and it bothers me that a bunch of people (including scientists) are going to get both their ideas of the state of the science and their ideas of trans women and trans rights activism from this book chapter. It would be like including a chapter about feminists being “out-of-control…activists” and Scary Science Deniers for harshly criticizing pop-evopsych or Freud, and picking a couple of critics who behaved inappropriately and trying to claim that they were representative of all feminists. Singal doesn’t have any particular knowledge on trans issues, either the politics or the science, AFAIK, and his reaction is an example of what I’m concerned about.

    • Thank you for the links. Yes, I’m aware that the trans activist community doesn’t like Dreger or her views. Whether that’s evidence for Dreger’s point, I’m not qualified to say. It comes down to an objective evaluation of the relevant science. And such an evaluation seems to be difficult to achieve (at least, that’s my impression as an outsider).

      Dreger of course covers numerous incidents in her book, the Bailey case is only one of them.

      • Right, that is why I linked to the peer commentaries (both pro and con), the Serano post that links to peer-reviewed papers on the science, and so on, to provide more work to look at.

        There’s a long history of “science” (e.g. phrenology) being used to reinforce oppression. It’s not hard to understand why people objected to someone saying that trans women “might be especially suited to sex work” on the basis of six or seven interviews that he had with trans women in one club.

  5. The “p-hacking” story is compelling and convoluted. I do not know how I would have responded to the circumstances as I’ve never encountered them. But I believe the decision should have been based upon the science and not the student’s independent constraints. While giving up the post-doc would have been painful, in the end the student might have found another that was even better. Alternatively, were everyone honest about the problem, perhaps the sponsor for the post doc could have delayed the appointment. Life often throws us a split-fingered fastball and we strikeout at the plate, but our next at bat could produce the game winner.

    • Here’s a real life story of researchers turning the lemon of a failure to replicate into the lemonade of a highly-publicized cautionary tale about the importance of not p-hacking:

      Of course, it isn’t always possible to make lemons out of lemonade. Easy to say that the decision in such cases should be based on the science. But hard to do if it’s your science, or your trainee’s science.

      • “Of course, it isn’t always possible to make lemons out of lemonade.”

        Suggests that it sometimes IS possible: I look forward to your first publication showing this reverse-engineering, Jeremy – Nature and Science will be scrapping over the rights to publish🙂

      • Wow- that’s one heckuva story. Although I confess I am completely ignorant about the methodologies of social science. I know just enough to understand their approaches often differ from what we see in the physical sciences. A couple of things really stunned me though.

        One was there did not appear to be any controls in their study- positive or negative. That would seem to invalidate the work before ever getting to the issue of p hacking. I was also stunned about their main conclusion.

        “Our conclusion: political extremists perceive the world in black-and-white, figuratively and literally.” Even had they properly replicated the experiment with controls, I don’t understand how one could ever reach such a starkly “black & white” conclusion.

      • It’s unclear to me how much of the story is made up: at the bottom it states “Almost all of the details in the the above story are made up. It wasn’t even snowing; it was June.” Does that mean the events never occurred? What does “details” refer to?

      • @Jeff: My interpretation of that “morality tale” is that all of the inessential details (time of year, weather, etc.) were changed to anonymize it. But I agree it’s a little odd.

      • @Eliot:
        “Even had they properly replicated the experiment with controls, I don’t understand how one could ever reach such a starkly “black & white” conclusion.”

        I don’t understand how anyone could see the initial idea for the “50 shades of gray” study as anything other than a silly play on words. So I don’t understand why they needed a failed replication to come to see the hypothesis as incorrect. But at least they got there in the end, I suppose.

Leave a Comment

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s