Recently, I noted a major kerfuffle in economics which began when a graduate student discovered serious analytical errors in a paper by two famous Harvard economists. The student discovered the errors because he’d been assigned the task of reproducing the analyses in a published paper for a graduate course he was taking.
So here’s an idea: maybe we should do the same in ecology and evolution? It’s increasingly the case that authors are obliged to make their raw data and computer code freely available to anyone after publication. So this kind of assignment is increasingly feasible. It’s an assignment that would fit well with graduate courses in statistical methods, and maybe with other sorts of courses as well. It’s a great learning exercise for students. Teaches you to pay attention to subtle but crucial statistical and coding details. Gets you thinking about other ways one might have done the analysis. Etc. And, on the off chance that you discover a serious mistake (or fraud!) in a published paper, you might well get a paper of your own out of the exercise.
This seems like a nice way of getting around the fact that correctness and reproducibility of analyses are important, but nobody has much personal incentive to check them. Even pre-publication peer reviewers rarely go to the trouble of reproducing analyses. And even if they wanted to they mostly couldn’t, since raw data and code aren’t ordinarily made available until after the ms is accepted. But if grad students are going to be doing some sort of data-analytical assignment for a class anyway, there’s no cost (not even an opportunity cost) to having them try to reproduce the analyses in a published paper.
What do you think? Do you already do this, or know someone who does? Is there any downside to it?