Friday links: Berkson’s paradox, writing review papers using Twitter, and more (UPDATED)

Also this week: you vs. your departmental seminar series, research experience for school teachers, nut grafs figures, Berkson’s paradox, the Zizek Maneuver, a link from Brian (!), and more. Also, Jeremy gives in to peer pressure and comments on Courchamp & Bradshaw (2017).

From Brian:

After an original “Ask Us Anything” question on the contributions to ecology of scientists in developing countries, we have had a series of first hand accounts (with more coming). Emilio Bruna just added a comment to the original post summarizing some papers he has been part of that actually measure the answer to the original question bibliometrically. Lots of nuance and interesting twists. Well worth a read if you’re interested in this topic.

From Jeremy:

A post arguing that the Canadian government isn’t convinced by the Naylor report urging big new investments in basic scientific research, because the report simply assumed those investments would lead to higher economic growth. Argues further that the government was looking to the Naylor committee for advice on how to balance funding of basic vs. applied research, not for an argument to increase overall research funding. Concludes that the way to make the case for Canadian government funding for basic research is to argue that it will produce better (not more, better) “knowledge workers”, thereby attracting capital investments from international companies employing such workers.

A post from middle school science teacher Andrew Doggett about NSF’s Research Experience For Teachers program, of which I was unaware. Sounds like a win-win for both teachers and profs. And another from teacher Tonia Tasneem.

Terry McGlynn with a nice post recommending that you plan your study by first imagining what your ideal key figure (“nut figure”) would look like.

Berkson’s paradox: when you select a multivariate sample on a combination of two (or more) variables, the relationship between those variables will be different than it was pre-selection. Very clear, accessible post. Suggests some possible examples from academia. I’m sure you can think of ecological and evolutionary examples, for instance regarding genetic and phenotypic correlations pre- vs. post-selection.

Courchamp & Bradshaw (2017) present the results of a multi-stage survey of ecology journal editors and ecologists, asking them to identify and then rank the most seminal ecology papers. They show the ranked top 100 list. The authors comment on their paper here and here. The paper predictably lit up the ecology intertubes, which is why I’m reluctant to chime in. I don’t feel like I have much to add. Plus, I’m generally reluctant to wade into Twitter debates because I’m wordy. 🙂 But in case you care what I think:

  • Respondents mostly seem to have interpreted the survey question as asking them to name foundational papers. I say that because the top 100 papers are mostly deservedly-famous old papers that kicked off important lines of research.
  • I think the paper would’ve worked great as a blog post or series of blog posts. But I would say that. 🙂 That’s not a criticism of the paper.
  • I’ve seen three distinct but related criticisms of the top 100 list (from Tuomas Aivelo, Terry McGlynn, Amy Freitag, and Markus Eichhorn). Some of these issues are noted by the authors themselves. The three issues, as I understand them: (i) Many of the top 100 papers are quite old. Some are outdated, and the non-outdated bits of the others are in the textbooks. Either way there’s arguably no reason for current students to read all these papers, except perhaps to learn the history of the field. As evidenced by the fact, noted by Courchamp & Bradshaw, that people often recommended and voted for papers they hadn’t actually read themselves. As someone who thinks that ecology needs new textbooks, and who thinks textbooks should stop teaching zombie ideas, I agree with this criticism, up to a point. But I also think that reading old foundational papers can be helpful to current students if they do it in the right way. And I do think there are some old papers on the list that reward repeated reading by today’s ecologists. I think it would be very interesting to discuss which old foundational ecology papers current ecology students should read (see also this related old post). I do think it’s vital for students to learn something about their history of their field, for instance so that they can recognize when “new” research is just old wine in new bottles. Reading lots of old foundational papers is one way to learn your field’s history, but far from the only way (e.g., you could read history books like Modelling Nature). I don’t know that any one way of learning history is inherently better than others. Different strokes for different folks, do what works for you. (UPDATE: A clarification of something that came up in the comments. In saying that students should learn the history of the field in whatever way works best for them, I meant to make a general remark about how students should learn the history of their scientific field. Read old papers, or read history books, or a combination of old papers and new papers on the same topics, or whatever works for you. I didn’t mean to imply that Courchamp & Bradshaw’s list specifically is as good as any other list of old papers for purposes of learning the history of your field. Sorry that wasn’t clear.) (ii) The authors of the top 100 papers are almost all white men. Obviously, that reflects the gender and ethnic composition of English-speaking ecology many decades ago. But arguably, the list is even more male-skewed than you’d expect given that it’s a list of mostly-old papers. That probably reflects subtle implicit biases to which everyone’s subject (Meghan has a good post on this in a different context). Courchamp & Bradshaw’s follow-up paper on the gender balance of their list is summarized here. (iii) The topics covered by the top 100 papers are those that many N. American and European English-speaking ecologists doing fundamental research traditionally think of as defining “ecology”. I’m very much part of that group, of course. Arguably, we should quit caring so much about some of those topics in favor of caring about others (e.g., newer topics, applied topics, place-specific topics…). Heck, maybe there is no single set of topics (and associated foundational papers) that all ecologists should know and care about (I have some sympathy for this view). To my mind, this criticism isn’t so much a criticism of the Bradshaw & Courchamp list per se; it’s more a criticism of how “ecology” has been defined in N. America and Europe, or of any attempt to define “ecology”. I think there’s a lot of room for interesting debate and reasonable disagreement on this one. This criticism raises super-difficult issues like whether “ecology” is even a single coherent scientific discipline, and whether fundamental research is worth doing in a world with pressing applied problems. Coincidentally, we have an upcoming guest post addressing some of this.
  • I’ve been disappointed to also see a few personal criticisms of the authors that are out of line, such as this (the subsequent apology is to her credit. UPDATE#2: and the original tweet has now been deleted). I think it’s unfortunate that Terry McGlynn felt the need to personally vouch for Franck Courchamp (a buddy of his). UPDATE: I think Terry made the right choice to take down his link to the now-deleted tweet.  Semi-related old post.

20 years ago, futurist Ray Kurzweil wrote a book in which he predicted technological progress for 2009 and 2019. How’d he do? Well, he was overoptimistic, especially regarding predictions about the more distant future. Many of his predictions for 2009 have come to pass, but only in the last few years. And many of his predictions for 2019 look like they won’t come to pass for many years yet. (ht Marginal Revolution)

The Zizek Maneuver: the most academic way to get on other people’s nerves. Pretend that you’ve never heard of someone that, by rights, you surely should have heard of. I’m now amusing myself imagining my own ecological equivalent. “David Tilman? Who’s he?” (ht @jtlevy)

And finally, how to use Twitter to write a comprehensive review paper on any topic without having to do any work yourself. 🙂 (ht @dandrezner)

From Meghan:

Andrew Hendry on why your department seminar needs you.

33 thoughts on “Friday links: Berkson’s paradox, writing review papers using Twitter, and more (UPDATED)

  1. Obviously, that reflects the gender and ethnic composition of English-speaking ecology many decades ago. But arguably, the list is even more male-skewed than you’d expect given that it’s a list of mostly-old papers. That probably reflects subtle implicit biases to which everyone’s subject

    I actually disagree with this. I saw a similar exercise last year at a conference meeting with the same results. It’s made me think a lot about these “top 100” kinds of lists. A platonic ideal of such a list is certainly useful from an educational standpoint. But how we build it is key. I’ve been thinking about what we mean by 100 must read pubs. We know that there’s going to be variation from person to person in that list, etc., and so we want to get the most representative sample we can.

    So, a sample. A representative sample. In a field that has skewed demographics that could lead to outsized influence of those demographic factors in choice of papers. This kind of survey problem is something the social sciences deal with all the time. If we’re going to try and do a real true version of this platonic ideal, then why not try and use the gold standard for reducing bias in our sample to make the list. It’s never going to be perfect, but there are methodologies – that take a good bit of work and thinking, but can be well argued for – that can and should help deal with at least some of the problem here.

    • To be clear, I don’t disagree that we all have implicit bias (think I copied a bit too much), but I do think the sample choice in this exercise was ripe for exacerbating this a huge amount – and often is without carefully controlling it.

      Further, with a properly demographically stratified design (and/or by discipline), such a list, I think, would not only be stunningly interesting, but the crosstabs would also be highly informative. But it requires careful thought as to what is a good survey and how to conduct such a thing – something outside of our disciplinary boundaries.

    • I think it’s interesting that people who think they have technical concerns with Courchamp & Bradshaw’s methods are all suggesting totally different alternatives.

      For instance, you suggest random sampling (of what population, exactly?). I saw somebody else suggest a “Delphic” process which (as I understand it) doesn’t involve sampling at all, but instead involves having people talk to one another and come to a consensus. And I saw somebody else say it was a good idea for Courchamp & Bradshaw to get leading experts to suggest candidate papers, but they should’ve supplemented that with other sources of suggestions. And I’m sure there are other suggested approaches I haven’t heard about.

      What I think people are actually disagreeing about is the purpose of the exercise, not whether it used the best study design given its purpose. You have to decide the purpose of the exercise to know how to conduct it. Are we trying to document the pre-existing opinions of all ecologists as to what constitutes the “core” of the field? Come to a field-wide consensus as to the “core” of the field? Develop a useful reading list for grad students? Or what?

      I don’t have any strong opinions myself on whether Courchamp & Bradshaw should’ve used a different study design because I don’t think I have a sufficiently-precise sense of their goals. (And also because I just don’t care that much about their study design. I’d rather talk about stuff like whether ecology has, or should have, a “core” of foundational ideas with which all ecologists have to be familiar, and if so what those ideas are. But I recognize that different people care about different stuff, which is fine.)

      • ” I’d rather talk about stuff like whether ecology has, or should have, a “core” of foundational ideas with which all ecologists have to be familiar, and if so what those ideas are.” I agree Jeremy.
        I have never met an ecosystem ecologist who believed ANYTHING in Behavioral ecology was worth paying attention to; That said I am pleasantly surprised at the choices in Behavioral/Evolutionary Ecology. They are very good!
        Realize [we] are not using this list to train Behavioral Ecology grad students;[ We] are using it to make all ecologists aware of the discipline. So to me the question becomes what few papers would I have ecologists read in Life History Theory, Foraging Theory, and so forth. I am a fan of history, and even though r/k selection has not worked out as originally intended, I think folks should read Pianka,1970 as a truly great paper: then they should read a modern update.

      • Very interesting suggestion that the core of your discipline is the small number of papers you’d want people working in other disciplines to read so that they are aware of and appreciate your discipline.

        I’ll need to mull over what papers I’d suggest that any ecologist read so that they appreciated and understood population or community ecology at its best. Probably not foundational old papers–not Hutchinson or MacArthur for instance.

        Which is interesting to me, because I mostly agree with your suggested readings for behavioral ecology. Why is that? Why is it that in one discipline, classic foundational papers are still great “advertisements” for the discipline, whereas in other disciplines that’s not the case?

      • The “essential readings in **********” is a widely used format, and a great variety of methods are employed in choice of paper/book-excerpts. The ones I know identify mostly old classic papers.
        Consider WILDLIFE MGT AND CONSERVATION: https://www.amazon.com/Essential-Readings-Wildlife-Management-Conservation/dp/142140818X

        I like the format of this book because it has sections written by the editors that introduce the papers, place them in historical context, and broadly discus/up-dates the fields they represent.

      • From Jeremy
        “…. Probably not foundational old papers–not Hutchinson or MacArthur for instance. Which is interesting to me, because I mostly agree with your suggested readings for behavioral ecology. Why is that? Why is it that in one discipline, classic foundational papers are still great “advertisements” for the discipline, whereas in other disciplines that’s not the case?……”

        Interesting way to put it Jeremy; maybe the answer is that foundational papers put new questions on the table,[ or settle important issues with data] AND provide answers and/or concepts about those questions. Or they find a new phenomenon to explain.
        many of the explanatory concepts in classic Behavioral ecology are the same ones we use today, while that is not so in pop/community ecology.Or maybe the field has gone on to quite different questions.
        Your guess?

  2. Not a criticism of Courchamp & Bradshaw (2017) as such, more of an observation: over 60% of the papers in the top 100 are single author, most of the rest have two authors (see my tweet showing the graph). Now clearly part of this is historical, but it also applies to papers published in the last decade or two: most of those are single or double authored. So what is this telling us about the way in which papers were chosen? Is it that “big ideas” papers tend to be associated with just one or two names that get remembered? Or that single or double authors are more likely to write “big ideas” papers?

  3. I just wanted to point to an underrated part of that story

    FWIW: At least they published the #preprint so everyone can now engage in #openpeerreview who knows, maybe then this 2nd paper gets rejected #openscience— Gregor Kalinkat (@gkalinkat) 17. November 2017

    https://platform.twitter.com/widgets.js
    as far as open science and open peer review have come over the last couple of years I see this as aclear sign of progress. Remember that 5 years ago many major ecology journals did not allow preprints.

  4. “I don’t know that any one way of learning history is inherently better than others. Different strokes for different folks, do what works for you.”
    Really? I can think of a huge number of cases in which I’d be disappointed if you thought this was true. I think it’s important to consider one’s privilege in these cases. We white males have not been subjected to terrible interpretations of history in the past that persist today. I see this list as another cog in the wheel that perpetuates gender/race imbalance in our field.

    • Hi Casey,

      You misunderstood my intent, sorry if I wasn’t clear. I didn’t mean that one can learn the history of the field just as well by reading a biased (or otherwise unfit-for-purpose) list of the top 100 papers vs. by reading an unbiased, fit-for-purpose list. My remark was meant more broadly. In general, one person might like learning history by reading old papers and get a lot out of doing so, another person might prefer to read history books and get a lot out of doing so, a third might prefer to read old papers paired with recent ones on the same topic, etc.

  5. As an ecologist, it’s hard for to imagine any other case in which authors and a journal could say, we’re well aware of biases in our study, but we’re just going to ignore them and publish the results anyway, knowing that there are dire consequences for where the field moves forward in the future.

  6. I agree with Casey here. Between ” I don’t know that any one way of learning history is inherently better than others. Different strokes for different folks, do what works for you.” which I think anyone verse in historiography or cultural theory would pretty thoroughly debunk with a broad citation list (not to mention the rewards we’re seeing of such a view being reaped in the current cultural moment), what strikes me more is

    “You have to decide the purpose of the exercise to know how to conduct it.”

    Hello intro to experimental design/stats/anything-to-do-with-analysis. Yes. Start with a question. If you’re putting together a list of 100 Articles Every Ecologist Should Read then you are not absolved from asking why they should be reading them, nor who are the people who should be making the choices about why they should be read. In the paper it was “list of seminal papers deemed to be of major importance in ecology” – ok, again, according to who? If it’s 100 papers ever editor at a major scientific journal thinks all other ecologists should read, great! That’s your population, and go forth and sample! But define that population. If it’s truly a platonic “list of seminal papers deemed to be of major importance in ecology” then we need to do better and think about all of the biases – explicit and implicit – that go into constructing such a list and design a sampling protocol accordingly.

    At the end of the day, the results are still going to be a sample, but my guess is that as scientists we’d seek to have as unbiased a sample as possible. And given that we *know* for social and historical reasons *we are biased* then let’s try and correct for that.

    Because if we don’t correct for the inherent biases of our field in asking what’s important, we will simply replicate the biases that are already there instead of actually learning how we should be thinking as a field. Unless you want to go on perpetuating those biases and see nothing wrong with them. But be honest about that. We are forced to do that even in the most trivial of meta-analyses all the time. Why should it not apply here as well to so much more weighty of a topic?

    • And to note, the authors do have a stated goal of objectivity: “Our aim was to collate a list of objectively chosen and ranked seminal papers deemed” – so, what does objectively chosen really mean and how can you design your sampling scheme to do just that. This is absolutely not “Different strokes for different folks.” Just the opposite.

      Again, social scientists have thought for a long time about this when there is bias in terms of demography and culture. We should be digging deep into their methodologies before we try and make such statements. There’s a lot more that can be learned – both about what is really valued and important and about how we value different contributions in our field beyond just reinforcing our cultural biases.

  7. On Courchamp & Bradshaw (2017).
    Only about 25% of the 600+ plus ecologists they contacted to nominate papers in the first round did so.
    maybe the ecology community, defined by their criteria [ which seems quite sensible to me…but then ecology is a big tent and i stay in one small part] did not think the exercise worth their time.
    I really doubt anyone could define a sample that represents all of ecology. Ecology is just too diverse, topic wise. And we are an arguementative tribe.[ Sorry for my spelling]
    About 12 Yrs ago GA Parker did a survey of ~25 sr behavioral ecologists to suggest and vote- for THE10 GREAT PAPERS OF BEHAVIORAL ECOLOGY. They were then ranked by # of votes, and Geoff wrote a whole historical paper around them; a very interesting exercise. Most surprising was that many papers got just one vote, suggesting that everyone has favorite papers that are not shared by many other people. Also surprising was that topics like LIFE HISTORY THEORY fared very poorly, not really being considered part of behavioral ecology by many people. agreement as to the what is the central core of behavioral ecology….is lacking.

    • “Only about 25% of the 600+ plus ecologists they contacted to nominate papers in the first round did so.
      maybe the ecology community, defined by their criteria [ which seems quite sensible to me…but then ecology is a big tent and i stay in one small part] did not think the exercise worth their time.”

      Or they don’t see any survey as worth their time. People are busy. Only a minority of invitees will respond to pretty much any survey on anything. So I don’t know that we can infer anything from the 25% response rate about how much importance invitees attached to this exercise. Besides that they mostly saw it as less important than their other obligations.

      • I too dont respond to most surveys, and when I do I usually grumble to buddies about some aspect of it..usually of the form that ……the questions dont get at the essence of the REAL issues.

  8. Running up against the nesting limit, so starting a new thread here.

    Franck Courchamp responding to the accusation that the survey didn’t randomly sample ecologists: (i) it wasn’t intended to randomly sample ecologists, it was intended to compile the opinions of a select group of ecologists, namely editors. (ii) They were surprised to get a much lower response rate to their survey from women than men, which is what prompted them to write a follow-up paper about gender issues in their survey. (iii) Both men and women respondents to the survey suggested quite male-skewed lists of papers. I’m summarizing his remarks here:

    https://smallpondscience.com/2017/11/14/what-are-the-top-100-must-read-papers-in-ecology/#comment-23624

  9. Conceptual Breakthroughs in Ethology and Animal Behavior (Paperback) (Michael D. Breed) 2017
    This little book [ ~250pgs] is quite nice. Its between a text and an ESSENTIAL READING format. Small essays, 2-4 pgs on concepts , complete with selected readings and Michael’s expert opinion as to importance of the concept [ 1-10], all arranged by date….old classics to modern bandwagons.
    And frankly I want a distinguished chap like Breed to share with me his opinion; he has done it all ..including a successful undrgrad textbook.

    • Breed discusses ~80 concepts, including ‘2 on life histories’,island biogeography, and other topics not usually included in animal behavior.

  10. This will be my last comment, just a reminder to the DE community;
    A 5 min google search will show that every field has its anthologies, with titles like SELECTED PAPERS IN***, ESSENTIAL READINGS IN*****, CLASSICS IN***, FOUNDATION PAPERS IN*****, BREAKTHTHROUGHS IN***, SEMMINAL PAPERS IN***, BEST READINGS IN***:. Or BEST READINGS FOR THE YEAR ____.
    Ecology has them for every major subfield I can think of, as well as THE WHOLE FIELD, whatever that really is.. They go back decades, and were used when I was an undergrad, although then they were on clay tablets in a strange script.
    The selection processes range from a couple of folks[ asking for input from their circle], to organized committees [ eg,The Brown/Real compilation], to real surveys with some kind of vote structure.
    [Remember that the contents of every ECOLOGY text book is the result of similar choice processes, excluding a vote; These also result from extensive feed back from ecology instructors].
    { Every sr/grad level ecology class has required readings, suggested readings, and makes them available, on line or in the library. }
    They are free, cheap [$] , expensive: one in Animal behavior has ~75 papers, defines itself as the best of the last century , comes in 4 big volumes, and retails for 1200$; last I looked it was discounted to 300-400$. Too rich for me.
    Finally, everyone of us has reading lists, and share them when asked; I have/had mine for the best 12 LIFE HSITORY PAPERS, BEST 10 SEX ALLOCATION PAPERS.. and so do you. Dozens of class lists too, although I threw them all out upon retirement .

    Sampling peoples lists[ all the above] and reading from them over many decades shows me the great variety of choices exist. I often disagree with maybe 1/3-1/2 of anyone choices, sometimes more if whole topics are omitted.you do too.
    I think it quite worthwhile to think carefully about reading old classics vs modern breakthroughs and cutting edge stuff; I realise DE has a post on this but am still unsure of the optimal balance [ I would put it that way, right].

    I invite everyone to make up their list, and share it widely; use any criteria you think appropriate to advance the science, as you see important.
    But dont attack others for their choices. Suggest YOURS as a substitute.

Leave a Comment

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

w

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.