Scientific bets vs. scientific influence

Posted on by
5

Martin Rees and Steven Pinker have a new piece in New Statesman about their friendly bet from a number of years ago, on whether “bioterror or bioerror” would kill a million people in a single event within a 6-month period before 31 Dec. 2020. Rees bet yes, Pinker took the other side. The winner of the bet depends on whether the “lab leak” hypothesis for the origin of Covid-19 turns out to be true. (Here’s a brief overview of the evidence on that.)

I’m not so interested in Rees and Pinker’s piece; it’s mostly a generic popsci overview of tail risks in biomedical research. I presume the reason it’s in a fairly high-profile outlet like New Statesman is some combination of (i) Rees and Pinker are famous, and (ii) the betting angle.

I think the interplay of (i) and (ii) is kind of interesting. I’ve written a lot in the past about scientific bets, and their cousin, adversarial collaborations. The purported purpose of scientific bets, and adversarial collaborations, is to settle scientific disputes. The idea is that, by committing to one side of a claim, and to the evidence that would decide its truth, you tie your own hands. Making a public bet forces you to make a specific, unambiguous, checkable claim, and so forces you to admit if that claim is right or wrong. That’s the theory, anyway, but in practice it doesn’t work out that way. Based on the examples I know of, I don’t think scientific bets and adversarial collaborations actually settle disputes (case in point). I can’t think of a case in which the loser of a public scientific bet admitted to being in the wrong. People can always find some excuse to retroactively wriggle out of the bet, or out of the scientific implications of the bet.

But just because scientific bets don’t serve their purported purpose doesn’t mean they don’t serve some other purpose.

I think the main purpose of public scientific bets is to gain or maintain influence. The bettors might not care so much about winning or losing the bet. They mostly just want others to pay attention to them, and to whatever scientific issue they’re betting on. Which only works if you’ve already got some measure of influence, I think. I mean, if two scientists with no public profile had made the same bet as Rees and Pinker, would anyone else care?

But I wouldn’t criticize Rees and Pinker for making their bet, and then taking up an obvious opportunity to write about it in the New Statesman. Lots of scientists, from the most famous down to the most obscure, try to bring themselves and their ideas to the attention of others, in all sorts of ways. Different ways of doing that will be most effective for different people. As best I can tell, there’s not much agreement as to which ways of seeking attention and influence should be totally beyond the pale, if any.

So I think it’s fine for Rees and Pinker to make a public bet and then write about it. Just as I think it’s fine for me to write a quickie blog post about it. They’re doing what works for them, and I’m doing what works for me. 🙂

p.s. There is a meaty post from Brian in the queue for Monday! This post is like chips and dip, tiding you over until your next proper meal. 🙂

What conclusions should you draw from a dataset when different analysts reach different conclusions?

Posted on by
10

The latest many analysts, one dataset project is out as an unreviewed preprint, and this one has the most depressing conclusions of the ones I’ve seen. Breznau et al. gave 73 social science research teams the same dataset, asking them to estimate the effect of immigration on public support for social policies. The point estimates were centered on zero but varied massively. A substantial minority of research teams reported a statistically significant negative effect, and a different, almost equally-big substantial minority reported a statistically significant positive effect. All this is in contrast to previous many analysts, one dataset projects I’ve seen, in which most or all analysts at least agreed about the sign of the effect of interest.

The variation among analysts in the Breznau et al. study is because there were a lot of analytical choices that varied among research teams, no one of which has a big “main effect” on the outcome of the analysis. So it’s not that, say, omitting one specific influential observation always reduces the estimated effect of immigration by some massive amount, independent of your other analytical choices.

On the plus side, at least you couldn’t explain much of the among-analyst variation from knowledge of analysts’ prior beliefs about the topic. Because it would be really depressing to live in a world in which every analyst was consciously or subconsciously putting a thumb on the scale, to reach the conclusions they already “knew” to be true.

Nor could you explain much of the among-analyst variation from knowledge of analysts’ methodological expertise. I find that result both unsurprising and reassuring. But I’m curious if any of you are either surprised or bothered by it.

Question: how exactly would you phrase the take-home message here regarding effects of immigration on public support for social policies? Would you say that the dataset is “uninformative” about the effect of immigration? That there’s “no evidence” for an effect? “Mixed evidence”? “Conflicting evidence”? Strong evidence against an effect of immigration? Or what? I’m torn.

Another question: I wonder if at some point we’ll have enough of these many analysts, one dataset projects to do some comparative analyses of them? If we ever do, here’s my pet hypothesis as to what we’ll find: the among-analyst variability will be highest in cases when the mean estimate (averaging over all analysts) is close to zero. My other pet hypothesis is that among-analyst variance typically will be as large or larger than among-dataset variance. That is, if you gave the same analysts two different datasets addressing the same question, I bet there’d usually be more variation in results among analysts than among datasets. (The analyst x dataset interaction term might be big too.)

Nobody writes papers complaining about the state of ecology any more (do they?)

Posted on by
29

Earlier this week, I was interested to read Liam Bright arguing that his own field (analytic philosophy) is a degenerating research program, and that ongoing attempts to salvage it by making it “relevant” or “practical” aren’t likely to succeed.

Even if you don’t think your own field is degenerating, I think it can be good mental exercise to consider the possibility. I’ve done so in the past in a somewhat tongue in cheek way, designing an imaginary graduate course on whether ecology is f*cked or awesome. That imaginary graduate course mostly wasn’t based on my own concerns or complaints about the state of ecology. It was based on other people’s concerns and complaints.

I bring this up because, reading Liam Bright’s piece, it struck me that I haven’t read any similar critique from an ecologist in almost a decade, with the exception of pieces critiquing ecologists’ statistical methods. I mean, click that last link–most of the readings I proposed in the imaginary graduate course are old! Heck, most of them were old back when I wrote the post in 2016, and now they’re even older. And the few that aren’t old are either (i) about statistics, (ii) are pretty narrowly focused (e.g., critiques of NEON), or (iii) aren’t really about ecologists’ research practices at all (e.g., concerns about climate change).

Now, maybe I’ve missed a bunch of stuff (please tell me if I have!) And maybe I’m just wrong–maybe such pieces have always been rare events and they aren’t any rarer nowadays. But if I’m right that such pieces are rarer these days, why is that?

Maybe it’s because the field of ecology has grown more sophisticated in its statistical methods, and more focused on global change and conservation? So worries about our statistical methods loom large in our minds. Whereas worrying that global change or conservation research wasn’t worth doing would risk crossing the line from “good mental exercise” to “trolling”.

What do you think? Am I just off base here? (Quite possible!) And if I’m not off base, what do you think has changed?

p.s. If something’s changed, I don’t know that it’s necessarily a bad thing, or a good thing. It’d depend on the reasons for the change.

What the heck is up with the many ecological meta-analyses that have inverted funnel plots?

Posted on by
14

As regular readers will know, my able research assistant Laura Costello and I have compiled a database of over 460 ecological meta-analyses. The database includes all the effect sizes, their sampling variances, and various other bits of information.

I’ve been sharing the fruits of my explorations of this database on the blog. Here’s my latest tidbit–a very striking feature of many (though far from all) ecological meta-analyses that I can’t make heads or tails of. Why the heck would many ecological meta-analyses have inverted funnel plots? So that there’s more variation among the most precise effect sizes than among the least precise ones?

Intrigued? Read on!

Continue reading →