Bonus Friday link: Deborah Mayo on Nate Silver

The ESA isn’t the only big scientific society that just held its annual meeting. The American Statistical Association also met recently, and Nate Silver gave the Presidential address. Here’s ace philosopher of statistics Deborah Mayo’s commentary on Silver’s talk. She asks some penetrating questions about Silver’s motivation for being “Bayesian”. For instance, if people mostly have very strong pre-existing biases, as Silver says, are they really going to be willing to be explicit about their biases, as needed for a Bayesian analysis (and also willing to give up their biases in light of new data)? And if those biases are just people’s prejudices, as opposed to relevant background information, do we really want to build them into the data analysis via priors? Shouldn’t we be trying to keep prejudices out of the data analysis entirely? Her comments resonate with my own review of Silver’s (mostly excellent) book.

7 thoughts on “Bonus Friday link: Deborah Mayo on Nate Silver

  1. I don’t understand the criticism here. If you’re doing a Bayesian analysis you don’t have a choice about explicitly specifying your priors so it doesn’t matter whether you are “willing”. And that is a benefit – the priors are laid out for all to see, allowing for agreement, disagreement, or indifference. If someone uses a prejudice as a prior then others will be able to interpret inferences about the posterior conditional on that prejudice. Now, the prejudice may not be obvious but the prior certainly will be.

  2. Being a philosopher of statistics is fine, but Silver puts his money where his mouth is by making predictions. He stacks up pretty well on that front. Is there a better test?

    • Yes, Silver himself has been successful at making predictions using certain methods. But I guess I don’t see what that has to do with the points Deborah Mayo raises. She’s not questioning Silver’s methods or their predictive successes for him. She’s questioning his motivation for adopting those methods (one can adopt good methods for the wrong reasons). She’s also questioning whether it’s really plausible that his methods would work well if applied by others. Silver’s a smart, thoughtful, unprejudiced, self-critical guy. That’s a big reason why he’s able to make good predictions. It’s far from obvious that the prejudiced, biased journalists and pundits he quite rightly criticizes would be able to adopt his methods successfully. Consider someone whose “prior” consists, not of relevant background information (as it does in Silver’s case) but of strongly-held and purely-subjective biases. Is such a person really going to honestly update their beliefs in light of new data? Now in fairness to Silver, it’s not clear that there’s any way, statistical-methodological or otherwise, to get people with strongly-held biases to give them up.

    • You know of course after all my posts on prediction, I’m going to agree with you about prediction being a strong test. Although its hard to tell exactly how Silver’s models work (Silver never has written up his methods for presidential prediction as a formal reviewed article as best I can tell) everything I’ve read suggests to me he is not using any actual Bayesian methods (neither Bayes law nor priors) in those models. I find it odd that so many people disagree with his philosophical claims and argue with him at that level, but fail to point out that he doesn’t actually use these methods in any mathematically meaningful way in what made him famous – his highly accurate predictions.

      Or have I got it wrong – can anybody find anything Bayesian in this (his Senate forecasting methodology. I can’t find as concise a summary of his presidential methodology but my memory from reading all of his posts is that it is pretty much the same. So when his reputation is on the line, where is the Bayesian?!

      • Hmm, good question. I guess I always had the impression that the way his polling aggregation models work is basically that the posterior from one day then becomes the prior for the next day, so then if any polls come in the next day they’re used to update the prior. I mean, I always figured there was more too it than that (e.g., how his model gives greater weight to economic data far in advance of the election, but then tapers this off as the election approaches so that the day before the election the prediction is based solely on polling data). But I thought that was the basic idea.

        Drew Linzer, the political scientist behind the Votamatic site, just published a preprint with his methodology, which I believe is Bayesian, so even if Silver’s not actually taking a Bayesian approach, others are.

Leave a Comment

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s