Should we judge a scientific field by its classic papers, its typical current papers, or its best current papers?

How should we judge a scientific field? One way would be via typical current papers in the field. After all, that what most papers are, almost by definition: typical. So if the typical paper in a field is a good paper, however you define “good”–it addresses an interesting question, uses technically sound methods, etc.–then the field as a whole presumably is doing well.

But that’s not the only way to judge, and maybe not the best way. I was prompted to think about this because of a passage in an old Anthony Lane essay on bestsellers (boldface emphasis added):

It is easy to brush aside best-seller charts as the product of hype and habit, but they are a real presence in the land of letters, generating as much interest as they reflect. And if they do, to an extent, represent the lowest common denominator of the print culture, this only strengthens our need to pay attention, since where else is that culture common at all? ‘Twas ever thus: anyone who imagines that a hundred years ago Americans were rushing out to buy the newest Henry James is kidding himself…This is nothing to be ashamed of; it is a proper corrective to our historical arrogance, the conviction that the best writings of our time will, shored up by our plaudits, both outlive us and represent us in centuries to come. But they won’t; we may not even have clocked the real thing when it passed before our eyes. That is why the ideal literary diet consists of trash and classics: all that has survived, and all that has no reason to survive–books you can read without thinking, and books you have to read if you want to think at all. In between is the twilight zone, the marshes of the middlebrow…That is why we should turn to the Times [bestseller] list every Sunday morning. If the language is still alive down at this end of the market–if there is juice running through the art of basic narrative–then we have no cause to be downhearted. Conversely, if the list is crammed with John Grisham, then we can all go out to brunch and rue the decline of the West.

The suggestion here is that the best novels can only be identified in retrospect, by having repeatedly proven their value to new generations of readers. So if you’re not reading old stuff that we’ve long known is well worth reading, you might as well just read something fun. Trying to write a Classic Novel is likely to result in a novel that’s neither fun to read nor of lasting value. So if the bestseller list is dominated by books that don’t strive for Classic status, but merely get the basics right–readable sentences, engaging characters, compelling plots–then we should all be happy about the state of literature. It means that our novelists are writing, and the bulk of readers are reading, good “trash”. Which, in the moment, is as much as we can expect of either novelists or their readers.

Does an analogous argument apply to science? That is, rather than evaluating a field by looking at its typical papers, perhaps we should look at its classic papers, and the atypical recent papers that become the scientific equivalent of bestsellers–widely read and cited, even if they’re quickly forgotten. After all, the typical paper will be quickly forgotten without ever being much read or cited, so does it really matter all that much if it’s any good?

A related argument, due to economics columnist Noah Smith, is that “vast literatures” on a topic function as “mud moats”: an ocean of typical research papers that would take ages for anyone to read, but wouldn’t provide much actual knowledge or insight even if you somehow did read them all. Indeed, might actually be misinformative in some cases. Noah suggests a solution: the Two Paper Rule. If you can cite two outstanding papers that illustrate and exemplify the virtues of some larger body of literature, that larger body of literature might be worth reading. But if you can’t, then that larger body of literature is probably a mud moat.

One reason not to apply Anthony Lane’s argument to science is that scientific papers that go on to become classics often are widely-read and cited immediately upon publication (there are exceptions). That’s in contrast to literature, in which novels that go on to become classics often aren’t bestsellers when they first come out. Which if anything actually strengthens Anthony Lane’s conclusions in the scientific context. If future classic scientific papers are mostly among the “bestsellers”, then that’s all the more reason to read the bestsellers, and to judge scientific fields by their current bestsellers rather than by their typical papers. Isn’t it?

One further implication, which I’ve argued for in various old posts, is that it’s probably optimal for science as a whole for post-publication scrutiny to focus narrowly on a small fraction of high-profile, potentially-influential papers (recent example). Though I’m not a fan of going one step further and giving no or only cursory pre-publication peer review to most other papers…

What do you think? Looking forward to your comments.