Friday links: break your methods, scientific writing vs. the trolley problem, and more

Also this week: was ecology more combative in the 70s and 80s, grade inflation at British universities, citation inequality is declining, a peer review experiment at eLife, MC Intermediate Disturbance, and more. Grab a coffee and settle in, there’s some very interesting stuff this week, on which I’ve commented more substantively than usual.

From Jeremy:

A very interesting remark from Anna-Liisa Laine:

The subsequent convo includes this remark from Stuart Auld:

The whole thing is worth a look. I would only add that you shouldn’t over-generalize from “the bits of ecology from the ’70s and ’80s that are widely-remembered today” to “all of ecology in the ’70s and ’80s”. The null model wars were atypical even at the time. That doesn’t mean they were unimportant (atypical != unimportant), or that one should only ever characterize a field by averaging over a large random sample of equally-weighted papers. But maybe the right question to ask is “How come so much of the small fraction of ’70s and ’80s ecology that’s still widely-remembered today is distinctive and often combative in tone?” And maybe part of the answer is “Such papers are unusual, so they stick in people’s minds.” Decades from now, maybe what’ll be remembered from ’00s and ’10s ecology is battles over biodiversity trends and blog posts about “zombie ideas“, and ecology students in 2040 will be going “Boy, ecology in the 2000s and 2010s was so combative and personal!” Or maybe not, maybe these days things really are different; I dunno. Related: this old comment from Andy Gonzalez, on the upsides and downsides of how today’s ecology papers all conform to a single “martini glass” template:

We are conforming to an approach that makes it easier for busy editors to make a decision, but maybe diminishes the creativity or at least the variety of approaches to writing a paper. I am casting my mind back to papers you could find in Am Nat in the 60s and 70s. There was a much greater variety of styles of writing papers then (not all Martini glass papers!). I enjoy reading those too.

See also the comment thread of this old post for a related conversation in the context of graduate student training. How do you create a climate in which students feel comfortable asking and answering challenging questions about their work, while keeping the conversations professional? And see my review of The Silwood Circle, a history of some very successful British ecologists of the ’70s, ’80s, and ’90s (Bob May, Mike Hassell, Mick Crawley, John Lawton, et al.). The Silwood Circle speculates that those ecologists share(d) attributes (including ambition, competitiveness, heightened yet selective awareness of what others are doing and thinking, a greater than usual willingness to make evaluative judgments about other scientists and their work, and self-consciously seeking to pursue and promote a particular approach to ecology) that may have contributed to their scientific success. But one might wonder whether other attributes might lead to equal scientific success, and if the attributes that lead to scientific success might change over time as social and professional norms change.

“I am a firm believer that before you use a [statistical] method, you should know how to break it.” I would generalize this to any method.

Citation inequality is decreasing as fewer papers go uncited. The proportions of citations going to very old and very recent papers also are dropping. (ht Retraction Watch)

eLife is trialing a peer review experiment. An editor first decides if the work meets “the highest scientific standards and potential significance.” If it does, it gets reviewed as usual for eLife mss (which as an aside is a slightly different review process than at other journals); otherwise I presume it gets rejected without review. But here’s the twist: authors of mss that get sent out for review are free to respond to the reviews however they want, because eLife will publish the paper no matter what (well, unless the authors decide to withdraw it from consideration). In response to the authors’ revisions (if any), the editors and reviewers decide if the published ms addresses their concerns, or if minor or major concerns still remain. Their assessment of the final paper gets published prominently at the end of the paper’s abstract. Interesting experiment. How it works in practice I think will depend a lot on the editor’s judgments about what papers to send out for review. “Solve for the equilibrium,” as Tyler Cowen likes to say. For instance, if the editors only send out for review stuff that they really like, well, that probably means the reviewers will like it too (not because of any nefarious conduct on the part of editors, but just because if one expert likes a paper then odds are other experts will too). Which if so means that, in practice, eLife will basically be offering desk acceptance plus a short News & Views-type commentary to some small fraction of submissions and desk rejection to the others. Which, when you put it that way, doesn’t…sound…like…a…great…idea? But I’m speculating that that’s how it might work out, and the only way to find out is to try it. Maybe editors will be willing to take more risks than I’m giving them credit for, and most authors will be willing to make requested revisions even if they don’t have to. Also, I think the editor/reviewer assessments will need to be abstract length, or else have abstracts. I doubt many people will read an abstract-free assessment that goes on for a page or more, even if it’s prominently placed. (ht Dan Bolnick, via Twitter)

Stephen Heard on the choice to write scientific papers in a dry style as an example of the trolley problem from moral philosophy. My undergraduate philosophy professor won the subsequent comment thread. Sort of. 🙂

Grade inflation at British universities appears to have accelerated in the last five years. I don’t know enough to comment on the linked article’s speculation as to the reasons. Perhaps Jeff Ollerton or other British commenters can chime in. (ht @kjhealy)

When you pose a question to the class and only one student raises a hand paw to answer. 🙂 (ht @dandrezner)

And finally (ht @kjhealy):

I’m either MC Intermediate Disturbance or MC Neutral Theory. If we allow methods as well as explanations, I could be MC Randomized Species x Sites Matrix. In the comments, I’ll give +1000 Internet Points to anyone with a worse rap name than any of those. 🙂

20 thoughts on “Friday links: break your methods, scientific writing vs. the trolley problem, and more

  1. Yes, there’s been lots of discussion about “grade inflation” and academic standards at UK universities, but not just in recent years, it goes back to at least the 19th century! There seems to be an expectation in some quarters that UK HE has been until recently a constant, unchanging entity of high standards and traditional practices. Which is nonsense, British universities have always been in a state of change, they have never stood still.

    Here’s one example that may explain some of the rise in First Class degrees: in some departments at older established universities it was a common practise to limit the award of a First Class degree to one student, and academics argued over who was the “best” in class who deserved that honour. You can imagine the bitter arguments amongst “colleagues” that ensued… Hopefully that doesn’t happen now but it would be interesting to know how widespread the practice was and when it was abandoned.

    The author of that piece raises some good points but also focuses on some odd targets, e.g. the notion of “value added” i.e. taking students from disadvantaged backgrounds who obtained poorer grades at school, and giving them a good education. Why is that so wrong? Also the notion of using the full range of marks, e.g. not limiting it to a range of 35% to 75%. Which, incidentally, is one reason why my university went to a letter grading system.

  2. The “martini glass” template for papers is ubiquitous across all life sciences, not just ecology. The “Devil’s Advocates” are rare and tend to be older scientists from the days of when criticism and challenges were considered as constructive and not destructive, nor as personal attacks. I miss those days, but those ghosts occasionally appear in Letters or Opinions.

  3. Two related areas from 70s and 80s that were highly confrontational and highly personal both in print and in department hallways: phenetics vs. cladistics and sociobiology vs. non-pan-adaptationism. I think the difference between then and now is real. Good riddance.

  4. Thoughts on two topics:

    On the style in 70s’80s papes I find it interesting but confusing that people have focused in on the confrontational papers as the reason things were less boring. I’m much more interested in the papers that were laying out totally new ideas and did it in elegant compelling ways. You can pick from Janzen’s hypothesis papers (e.g. mountains higher in the tropics) or MacArthur’s various papers, or May or Rosenzweig or Warner & Mittelbach or Tilman etc. None of those were confrontational or rude or targeted at a group. And all way more interesting to read than today’s papers.

    As for grade inflation in the UK, I’d be interested to hear what people from the UK say. But about 10 years ago the UK switched to a model where education became a private good and thus demanded tuition to pay the bulk of the costs (my understanding is fees could be like 9000 pounds). Most other countries still see education as a public good and only require students to pay minimal fees (like 1000 pounds). The only other country I know that treats education like a private good is the US which also has grade inflation. To me it is a not surprising unintended consequence. When students are paying most of the money they become customers who can pick and choose and demand. Surely this is at least a piece of the grade inflation story?

    • Hi Brian – see my comments on this above. The timeline for fees in England (not the whole of the UK, it varies, e.g. no fees in Scotland for younger students) is that they started at £1000 in 1998; £3000 in 2006; £9000 in 2010; £9250 in 2015. These changes don’t seem to correlate with the figures in the graph that the writer presents, and certainly not with what’s been happening over the last 5 years. Charging fees may be part of the answer but it doesn’t explain the whole pattern.

      • Hi Jeff – I imagine there could be some pretty healthy time lags in there. Universities are not exactly famous for rapid change! It is entirely believable to me that fees could show the main increase in the late 2000s and the jump in grades could respond in the mid 2010s. Of course I realize this becomes an unfalsifiable hypothesis. But, I don’t think the timeseries correlation analysis you present entirely convinces me to give up on the pricing argument (although as I said before I wouldn’t claim it is the whole story).

    • Question: What’s the origin of the “martini glass” paper structure? How far back does it go?

      Second question: how much of what makes a paper interesting to read come from its structure vs. writing style vs. content? I bet it’s the latter two. For instance, there are plenty of ecology papers from the ’50s, ’60s, ’70s and ’80s that don’t have a martini glass structure, but that are completely forgotten today (as are their authors). If you back and read those papers, would they all be hidden gems that would have you going “Oh my gosh, ecologists in the ’70s were all just better writers and more interesting thinkers than ecologists today! How did the whole field go off the rails?” I doubt it–I bet most old papers would read pretty turgid today (maybe I’m wrong!). So I think that what we’ve lost isn’t field-wide ability to write and think. It’s that today’s *upper tail* of papers aren’t as unconventional in the style and structure, or interesting in their content, as the upper tail of decades ago.

      Third question (also asked by Macrobe above): these days, should we be looking to opinion and “perspectives” papers for examples of non-martini glass structure + non-bland style + interesting ideas? That is, maybe decades ago there were some ecologists (the ones we remember today) who wrote all their papers the way that opinion and perspectives papers are written today?

      • A bug bear of mine for papers from earlier periods is a lack of detail in the methods regarding basic facts like localities, sample dates and sizes, etc. I’m sure these aspects of papers have got better over time.

      • I agree. I definitely think you can write very interesting papers in the hourglass style. Several of those 80s author I referenced as writing great papers (Mittelbach, Tilman) often used the hourglass style. While I think some modelling and opinion papers suffer from the hourglass, I think most research papers benefit from it. Getting back to Steve’s blog, it is the borg-mind-meld style, not the convergence on structure that is deadening today’s papers.

  5. I think that article is probably about right on the main causes of grade inflation in UK universities. One of the fundamental causes, IMO, is the publication of university league tables by newspapers that ought to know better. These don’t actually assess the quality of teaching in any meaningful way, but instead rely on a pile of correlated proxies, one of which is the percentage of first and 2.1 class degrees awarded. This obviously gives the Men In Grey Suits something to get their managerial teeth into, since the whole thing can then be gamed simply by awarding more of these degrees. This is now exacerbated by the importance placed on the National Student Survey by those same Men In Grey Suits, the NSS suffering from all of the problems associated with student evaluation of teaching, and now the introduction of the “Teaching Excellence Framework” by the government. This latter leans heavily on the NSS, so it’s essentially a popularity contest.

    University management are desperate to improve the scores in these assorted measures of… something… but obviously the idea that they might invest some money so that we can actually have the resources we need to teach our students properly is anathema. That money is earmarked for paying for vanity buildings and people with job titles like “Deputy Dean for the Student Experience”. Since student satisfaction (and therefore NSS ranking and therefore TEF score) correlates with the grade they get, there’s an obvious, cheap way of making things better.

    In case anyone thinks this is over-egging the pudding, here is a genuine tweet from our Minister for Higher Education.

    We want students to be better informed about degree choices & the returns – today, we’re officially launching a competition for tech companies to take graduate data & create a MoneySuperMarket for students, giving them real power to make the right choice.— Sam Gyimah MP (@SamGyimah) June 26, 2018

    • I agree with a lot of what you say Rob but I’m not sure about the link between NSS and grade inflation. The NSS is completed long before students know their final 3rd year grades, and of course 3rd year grades have the most influence on degree classification. I imagine that the timing of the NSS was deliberate in order to avoid it being skewed by disgruntled students who didn’t get the degree they thought they deserved.

  6. Related to grade inflation: I went to a talk last year by a historian who noted that grade inflation really took off in the US during the Vietnam War, because bad grades could mean a student would go to war. Based on my quick googling, apparently this is relatively widely known, but I hadn’t heard it before and found it interesting.

    Here’s one article that mentions it:

  7. I really enjoyed Stephen’s post. This was the key part for me: “If we accept that lack of style has readership costs just as style does, we come to what I’ll call the trolley problem of writing. Imagine that I write a paper with an unusual bit of writing style – perhaps a colourful metaphor. Image that doing so leaves 10 readers confused or repelled; but imagine that the style gets my paper a little social-media play or word-of-mouth that recruits 11 new readers. I’ve saved 11 readers, but run down 10: writing with some style has increased my net readership, even if it’s turned some people off. This is a trolley-problem win, isn’t it?”

    I agree that we focus a lot on not turning away readers, and need to focus more on drawing them in. This is one of the reasons I loved Helen Sword’s Air & Light & Time & Space and now am reading her Stylish Academic Writing.

  8. From what I’ve seen in geology, when human interactions become the main focus of a scientific controversy, its usually because the evidence isn’t strong enough for either side to convince the larger body of the science.

Leave a Comment

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.