Friday links: how to handle bad reviews, zombie ideas in conservation, and more (UPDATED)

Also this week: nerd sniping Stephen Heard, what’s a better word for p-hacking, the latest modern aRt, worst zoo exhibit ever, and more.

From Meghan:

From the Onion: Zoo posts hourly updates on aphid about to give birth. Clearly I need to start posting updates on our Daphnia!

From Jeremy:

Zombie ideas in conservation biology: 3/4 of studies finding significant effects of habitat fragmentation per se (as opposed to habitat loss) find only positive significant effects–but only 40% of those studies describe their results that way in their abstracts. In contrast, the large majority of studies that find only negative significant effects of fragmentation describe their results that way in their abstracts. Studies finding positive effects of fragmentation also sometimes emphasize that their results shouldn’t be extrapolated to other systems–but studies finding negative effects of fragmentation never emphasize that. Apparently, authors tend to spin their abstracts to line up with what they expected (or hoped?) to find, rather than with what they actually found? It would be very interesting to go through the same exercise for papers on other topics. (ht Mark Vellend, via the comments)

Ecologist Joe Craine was fired from his tenured position at Kansas State University after the university concluded that his misconduct accusations against two KSU colleagues were “malicious, or at the very least frivolous”. NSF denied his application for federal whistleblower protection because “baseless accusations of fraud to the Ecology editors” are not eligible for federal whistleblower protection. Now the US Circuit Court of Appeals for the 10th Circuit has upheld that denial. Follow the link, and see here, for background on this unusual case (I know nothing besides what I’ve read in news reports). I would not take this case as illustrative or emblematic of any broader problem or issue. Perhaps the only broader lessons I would take away are that, if you want to accuse somebody of scientific misconduct (as opposed to merely wanting to correct published errors, without making any claims about the reasons for those errors), you should (i) have a valid basis for doing so, and (ii) speak to a lawyer and determine the proper channels to go through before making the accusation. (UPDATE: paragraph corrected)

Terry McGlynn on how he handles bad reviews, as both an author and editor. Very useful perspective, includes several points that aren’t made often enough.

Andrew Gelman doesn’t like the term “p-hacking” because it suggests conscious, nefarious intent on the part of the investigator that usually isn’t there. I agree with the suggestion in the comments that “researcher degrees of freedom” is the best term. Will try to remember to stick to that term in future (I’ve used “p-hacking” myself in the past without meaning to imply nefarious intent).

A brief potted history of public intellectuals, how they’ve made a living, and how their means of making a living has affected their work. Useful historical context for present discussions of the public role of scientists. Worth keeping in mind that what we all think of as the “normal” state of affairs–public intellectuals mostly are college and university faculty–only really goes back to WWII.

Economists like to think of themselves as scientists. So here’s an economist asking what’s so great about science. Long, wide-ranging post, covering a lot of philosophy of science and its applicability (or otherwise) to economics. You may want to skim for the interesting nuggets, for instance on how statistical methods of time series analysis struggled to gain a foothold in economics. Always enlightening to read about how different fields have handled the same issues ecologists have struggled with.

Accidental aRt remains very amusing.

I would love for Stephen Heard to write a blog post about the Latin name of one of these species. πŸ˜‰ (ht Matt Levine)

19 thoughts on “Friday links: how to handle bad reviews, zombie ideas in conservation, and more (UPDATED)

  1. Jeremy, you have hit on the perfect nerd phishing scheme. I would have clicked on *anything* linked from “nerd sniping Stephen Heard”.

    And my thought process at the cartoon:

    1. Ooh, I wonder if this problem has been solved for migration rate?
    2. And what about a migration/local selection balance?
    3. I bet you could get some cool spatial pattern out of that.
    4. I could run simulations.
    5. But they’d have to be on a torus.
    6. Wasn’t I supposed to be doing something?

  2. Public intellectuals and “thought leaders”, right. I guess there’s supposedly some kind of difference or something.

    So, now we have “TED” talks where academics can go and impart their knowledge, in a glitzy forum, to whoever thinks attending is worthwhile and can afford a ticket.

    I watched a YouTube video of a well known climate scientist, who shall here go un-named, giving one of these. It consisted of the standard sort of spiel to the public, which is to say a very predictable and glossy big-picture kind of thing without much detail or discussion of the uncertainties or disagreements or various other problems, but with a lot of smiling, “confident body language” and pauses for effect. There were a few shots of the audience during same, and my they looked impressed with this presentation, star-struck even, given the big graphics on the stage screen, a real live scientist up there, and just the whole cool thing of it all.

    Then I watched another video of this same scientist attempting a very similar talk to a group of–I think it was–philosophers or historians of science. There were again a few shots of the audience but the wide-eyed looks were replaced almost entirely by critical, attentive and dead-pan looks, like a jury listening carefully to trial arguments. The pauses for effect went nowhere, as if the audience was saying “keep going please”. When time for questions came, the very first one was along the lines of “Well, you’ve said a lot of nice this and that, but you didn’t really address these critical A, B and C issues which are the ones that really matter here”. An answer was given. They were not impressed, you could tell.

    So yeah, by all means, give lectures to the public, you can make some nice money doing so, more than some scientists have to conduct their research with if you play it right and keep it going.

  3. Habitat fragmentation reporting bias:

    I’ve seen a similar thing in the geological literature that has this general form: author discusses data, notes several aspects of results that are inconsistent to varying degrees with PPH (Principle Prevailing Hypothesis – whatever it happens to be for the particular line of research), then goes on in conclusion to ignore anything that’s inconsistent w/ PPH and happily pronounce that the data support PPH.

    Dunno why people do that bcz to me finding the unexpected is what’s fun about doing research. But perhaps its a reflection of the “Right Answer” mentality that afflicts many students: they’re not that interested in subtleties. They want to get the Right Answer and move on.

  4. And OMG, forehead slap, I just read that “What’s so Great about Science?” piece. Yikes.

    First off, many people involved in the public discussions now recognize that indiscriminately labeling critics as “climate change deniers” has done nothing but stoke the fires of personal animosity in these discussions, and indeed is **likely a principal original cause thereof**. If you want the history on that, I’ll be glad to give it to you with particulars, because I was, unfortunately and regrettably, associated for a time with one of the responsible parties before I realized by careful observation what was actually going on there. I can cite chapter and verse.

    The guy uses, as a “denier” example, people who supposedly deny global, observed T changes. Well, sorry man, that sells nicely among the unaware but it’s a very tiny fraction of climate science critics. Almost universally, T change trends over the 20th C are acknowledged; the focus of the debate is instead on the quantitative (1) attribution of cause and/or (2) subsequent effects, and (3) transparency and level of consensus. A lot of this gets into appropriate analytical methods, potential bias in models or data sources, and so forth. But this kind of red-herring diversion is utterly commonplace.

    Economics and climate science actually have a lot in common–they’re both almost entirely non-experimental, complex and heavily quantitative. The point he makes about economics being sort of an increasingly complex and self-sustaining metaphysics could just as well be applied to climate science, but instead of casting a critical eye there he just says he’s going to assume that that field of science is sound.

    Oh OK then.

  5. That paper on bias in abstracts of fragmentation papers highlights the value of systematic reviews and meta-analysis. Harder for a meta-analyst to “spin” results, I’d think.

  6. I have followed the Craine case with some interest.

    “I would not take this case as illustrative or emblematic of any broader problem or issue.”

    I believe there might be some other generalities to take from this case. K-state reacted to the matter really as any organization would. Specifically, an accusation made against a staff/ faculty person is the same as an accusation made against the institution. These institutions will defend the accused and not the accuser because they seek to defend themselves. In fact, in most cases, their internal policies and procedures are designed to protect the accused and, if necessary, punish or terminate the accuser.

    That does not mean Craine was right, wrong or otherwise- it just means that anyone leveling an accusation against a university employee can expect the university to vigorously defend the accused. Sometimes, the those defensive strategies are injurious and unfair. And really, fair or not, it is standard operating procedure even when a person has a legit gripe. For example- the University of Colorado embarked on a vicious smear campaign of Lisa Simpson when she alleged she was gang raped by the designs of the athletic department (denverpost.com/2007/12/05/cu-settles-case-stemming-from-recruit-scandal/). Eventually, the university accepted blame… but not before they had already done serious damage to Simpson’s reputation.

    The point I am making is that I do believe the Craine case is emblematic of this kind of behavior, and it really has little if anything to do with the validity of any given complaint.

    • Fair enough. Difficult to say in this case, because based on what I’ve read I do think Crane was in the wrong. Hard to say how the uni would’ve reacted had he been in the right. Perhaps they would’ve reacted similarly; I think it’s hard to say.

      • Your point about consulting an attorney (or several) is a good one I’d say. The reasons for getting that advice may not be apparent to lay persons- or, even to attorneys when these kinds of issues are involved. Terminology is really the key, because not all language used to formulate policy has corresponding legal definitions. Most universities, journals and funding agencies have defined “research misconduct,” but I think you would be hard pressed to find it defined in most states’ statutes. Fraud, on the other hand, will be clearly defined in the law. “Honest mistakes” is also a term defined by most universities, but I’d bet the ranch no statute has defined it. Flip that coin again, and you will find plagiarism is well defined in state and federal statutes.

        The other point to bear in mind is that university and funding agency definitions tend to be fluid, and decided on a case-by-case basis, and usually by a committee of faculty peers. In other words, one man’s fruit can become another man’s candy in a New York minute. It is a very risky business to allege things like plagiarism and misconduct, even when the “smoking gun” and the “blue dress” are there for all to see. Sometimes our sense of ethics and personal responsibility behoove us to take action, but everyone should know the blow-back will be intense and at the very least career-threatening.

        The safest way to proceed is to get that legal advice and short of that, to not use judgement-based terminology. So rather than saying, “he/she committed research misconduct,” it is much safer to allege, “the values reported in table 1 are not consistent with the measurements taken in the lab.” Also- it might be useful for folks to remember that concerns can be reported, anonymously, at PubPeer.

      • I forgot to add, I tend to give folks like Craine a significant benefit of the doubt (although I admit, maybe I shouldn’t). The reason for this is that any sane person among us understands the repercussions of coming forward with a complaint of this nature: ie., not only will the wagons get circled, but the proverbial institutional firing squad will lock, load and fire.

        It is difficult for me not to envision Craine experiencing enormous anxiety- fully understanding that by making these accusations, he likely imperils his career (even in the event the accusations are proven true). I think it takes tremendous amount of guts and courage to come forward as he has, likely knowing what was in store for him. So unless he is somehow mentally deranged, I am left with the feeling he very likely had a story to tell.

        Perhaps you, Dr. Fox, might give consideration to highlighting the experiences of those who have come forward with knowledge of wrongdoing and experienced the inevitable institutional backlash. Maybe a guest post along these lines would be not only informative, but inspirational.

      • @Elliot:

        I encourage you to follow up at the links re: the Craine story if you haven’t already. The roots of the story go back many years, apparently. Craine’s problems with some of his colleagues seem to have been longstanding, and at least fairly widely known within his department. To my mind, that makes this situation disanalogous to one in which a whistleblower works up the courage to blow the whistle on a colleague with whom he or she previously had good relations.

        We do have posts on finding the courage to come forward regarding errors in the literature–one’s own:
        https://dynamicecology.wordpress.com/2016/12/02/on-finding-errors-in-ones-published-analyses/
        https://dynamicecology.wordpress.com/2015/04/14/mistakes-happen-in-science/

      • Dayum- that was an excellent post about how you handled an error in a paper. I think it instructive on so many levels. Not the least of which is to accept mistakes can and will happen, and that the mistake is not what defines our character, but rather how we respond to it.

        There are those among us who behave very differently, and go to great lengths to bury the mistakes they have made… and bury those who dare expose them. I was especially intrigued by Dan’s experience- where he thought he revealed a slight trend and then realized there was none. I very recently discovered a former colleague did just the same concerning data in my possession, and I am lost on how to handle that one, because this person is well known for being a bull in a china shop…

        I too have an anxiety disorder, and believe you me, I now am kept awake at night over this issue. While I know for certain the person has published a false result, I am torn to say something v. nothing about it. Say something and it is almost certain grief will envelop me. Say nothing and I have to live with the fact I am complicit.

        Sheesh…

      • I acknowledge my error and stand corrected… that feels better!

  7. Pingback: On being nerdsniped by Dynamic Ecology | Scientist Sees Squirrel

Leave a Comment

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s