Selective journals vs. social networks: alternative ways of filtering the literature, or po-tay-to, po-tah-to?

Attention conservation notice: I’ve basically done this same post before. But as a blogger, I’m never too proud to put old wine in new bottles. Plus, historians of science will tell you that scientists win scientific debates in part by repeating themselves over and over. And I like to win. 🙂

*******************************

Nobody can read more than a tiny fraction of all the scientific papers published every year. And that fraction is getting tinier as more and more papers are published while the length of the day and the speed at which people read remain fixed. So everybody needs filters to decide what to read (even what abstracts to read), and what to ignore. This post is about different ways of filtering the literature, and questioning whether they’re really so different.

I mostly use pretty traditional filters. My main filter is to scan the titles of new and forthcoming papers in selective journals in general science, ecology, evolution, and philosophy (especially philosophy of science). I keep very close track of what’s coming out in journals that have a good “batting average” for publishing papers I want or need to read (mostly highly-selective journals like Nature, Science, PNAS, Ecology Letters, Ecology, Am Nat, Evolution, etc.). I keep less close track of what’s coming out in journals that have a lower batting average for me (e.g., Journal of Theoretical Biology, Theoretical Population Biology, Genetics). If I see a title that looks interesting, I read the abstract, and based on the abstract I decide if I want to read the full paper in detail. So if something’s not published in a selective journal I keep an eye on, it’s quite possible that I’ll miss it. Which is a problem, but I don’t worry about it too much because any filter would have the same problem. No filter is guaranteed to sift out all and only those papers you want to read. I have other filters, but this is the main one.

As I’ve discussed in previous posts and comment threads, my filters work for me.* They work because there are many other people who filter the literature in roughly the same way I do. That is, there are many other people who read, review for, and submit to the same selective journals I do. So in using “what’s published in selective journals” as a filter, effectively what I’m doing is trusting the collective judgment of the many colleagues whose scientific interests and values broadly overlap with mine. I’m trusting them as authors to submit what they regard as their most novel, interesting, and important work to journals like Science, Nature, and Ecology. And I’m trusting them as reviewers to only recommend to journals like Science, Nature, and Ecology papers that seem to them to be especially novel, interesting, and important compared to the bulk of ecology and evolution papers.

An alternative perspective on filtering the literature starts from the observation that reviewers often disagree a fair bit about what’s “interesting” or “important”. It’s often further argued that what gets published in leading selective journals is determined by “salesmanship” on the part of authors, and that journals like Science and Nature (and in ecology, Ecology Letters) are showy, insubstantial “glamour mags” (see, e.g., the comment thread on this post, or this post and the comments). As an alternative, it’s often suggested that reviewers should just evaluate technical soundness, with post-publication filtering being done (either in part, or entirely) via social media. For instance, you might choose to read papers tweeted by people you follow on Twitter, papers linked to by your friends on Facebook, papers recommended by people in your Google+ circles, papers linked to by blogs you read, etc. (Or maybe you read papers that have been read or downloaded many times from the journal website, or you do Google searches and then read the top hits. Those filtering methods wouldn’t ordinarily be termed “social media” methods, but I’m going to lump them in with “social media” because in practice they have the same consequences, as I argue below.)

Traditional filters (“consider reading what’s in leading selective journals”) and newer social media filters (“consider reading what people are blogging about, tweeting, sharing, linking to, and downloading”) sound very different. But are they? After all, relying on social media filters still comes down to relying on the judgment of your colleagues. You’re still relying on the fact that, collectively, they can read a lot more than you can. You’re still relying on them to point out to you papers that are particularly interesting or important. After all, if someone just tweeted a link to literally every new paper about ecology and evolution, would you find that useful? Of course not. Same if someone just tweeted random papers. So it’s not as if newer filtering methods eliminate inevitably-somewhat-subjective judgments about what’s “interesting” or “important” so much as relocate them. And of course, reading what’s been read or downloaded most, or the top hits in a Google search, is just an indirect way of relying on the collective–but nonetheless still subjective–judgment of your colleagues as to what’s most worth reading.**

So isn’t this really just a case of po-tay-to, po-tah-to? A case of differences that really don’t make much of a difference?

Now, of course, for any given person, one filtering method might work better than another, in the sense of revealing more, and missing fewer, papers that the person in question wants to read. Traditional filtering methods work better for me than other methods would, but I’m sure the opposite is true for others. But which method will work best for a given person is surely an empirical question, and likely one without a general answer, at least at the moment (at some point in future, traditional filtering methods may well just fail, but that’s not an argument for giving them up now if they currently work for you).

And in aggregate, both ways of filtering have pretty much the same consequences for science as a whole. For instance, citations in the scientific literature are highly concentrated, and they’re becoming more concentrated. A small fraction of papers get a disproportionately large fraction of the citations. That’s an effect of traditional ways of filtering the literature. Lots of people agree on what the top journals are. And they read, submit to, and review for them in large part because they’re the top journals. So the small fraction of the literature that’s published in those top journals gets a large fraction of the attention, and thus the citations. But citation concentration wouldn’t go away if we all switched to filtering the literature via social media. The popularity of pretty much everything in social media, or online more generally (or even offline!), has a highly skewed frequency distribution. A small fraction of blogs have massive readerships while most blogs have small ones. On any blog (including this one), a small fraction of posts draw huge numbers of pageviews, while most draw small numbers. A small fraction of Twitter users have massive followings, while most have few. A small fraction of YouTube videos garner huge numbers of views, while most garner few. A small fraction of Plos One ecology papers garner huge numbers of views, while most garner few. A small fraction of books become bestsellers, while most sell very few copies. A small fraction of movies garner a large (and increasing) fraction of the box office. Etc. In my experience, advocates of post-publication social filtering often bemoan the fact that papers in “glamour mags” are widely-read and often-cited just by virtue of where they’ve been published. But if anything, post-publication filtering is likely to lead to greater, not lesser, citation concentration (or concentration of any other attention metric–views, downloads, shares, etc.). Since rather than acting independently the way authors and pre-publication reviewers do, people often link to, share, tweet, and blog about papers others have already linked to, shared, tweeted, and blogged about.

Note that, so far, I’m not arguing that social media filters necessarily tend to promote any particular sort of work–good work, poor work, flashy work, whatever. I’m just saying that they tend to promote citation concentration (or more broadly, “attention concentration”), without making any claims about the properties of the papers that end up garnering the bulk of everyone’s collective attention.***

So now let’s consider the issue of whether different filters promote different sorts of work. Maybe both ways of filtering do indeed promote “attention concentration”, but social media filtering does a better job of concentrating our collective attention either more accurately on “objectively better” papers, or at least more precisely (i.e. consistently concentrating our attention on papers with certain properties). Maybe–but probably not. Ace social scientist Duncan Watts and colleagues have done massive, well-designed, properly controlled experiments asking whether the collective, evaluative judgements of people sharing information via social media are more accurate or repeatable than the evaluative judgments of individuals acting independently, and the answer is that they’re not. If you think that what gets published in Nature or Ecology Letters is a crapshoot, well, it’s no more of a crapshoot than which papers would go viral in a world in which everything was published in one place and then filtered via social media. And please don’t try to push back against this claim by citing data on which altmetrics (retweets, shares, downloads, whatever) best predict future citations, because future citations themselves are just one more metric among others. There’s no reason to think of future citations as an “objective” measure of the “interest” or “importance” of a paper, and then judge filtering methods by their tendency to pick papers that will accumulate lots of citations in future. Unless you really want to insist that the best filtering methods are whichever ones best predict bandwagons.

And if you say, well, at least a world in which all filtering is done post-publication via social media is a fair world, in which everything gets a shot at going viral, I’d respond, how is submitting to pre-publication peer review at a selective journal not getting a fair shot? Anyone can submit anything they want to any journal. How is hoping that a widely-read selective journal will publish your paper any different from hoping that, say, a widely-read blogger will blog about your paper, or that your paper will become one of the few to go viral on Twitter, or whatever? My question here is not at all hypothetical. For instance, in economics (which has a much larger and more active online community than ecology), everyone pays attention to what Mark Thoma and Marginal Revolution link to. Drawing a link from one of those sites is sure to send massive numbers of readers to your economics paper or blog post. How is hoping from a link from Mark Thoma or Marginal Revolution any different–in particular, any more fair–than hoping that the editor at Science or Nature likes your paper?

Look, I get that it really bugs scientists, who work so hard to be objective, and to tease out subtle signals from random noise, that the readership of their papers–and thus, ultimately, their own careers–might inevitably depend in part on subjective criteria and random chance. But unfortunately, there’s no changing that. Here’s why:

• Nobody can read everything
• Everybody wants to read interesting and important papers
• Everybody thinks some papers are more interesting and important than others
• Everybody wants and needs to pay some heed to what everyone else thinks is interesting and important. (Science is a communal activity. You can’t remain willfully ignorant of what everyone else in your field is reading. Nor can you define “your field” so narrowly that it only includes you, or you and your friends. Not if you want a job or grants, anyway.)
• People don’t always agree on which papers are most interesting or important.

Those ingredients are, I think, sufficient to ensure that a small fraction of papers will always garner a large fraction of the attention (however measured–citations, downloads, times tweeted or shared, whatever), and that the identity of those papers will always be determined both somewhat subjectively, and somewhat randomly. And no alternative filtering method is going to change any of those ingredients one bit (and I’m not the only one who thinks so).

So just use filtering method works best for you, and don’t worry about the consequences of your choice for science as whole. Because all filtering methods have the same consequences for science as a whole.

*If they ever stop doing so, I’ll change them.

**There are differences in detail of course. For instance, in relying in part on the judgment of pre-publication peer reviewers, I’m relying on the judgment of people who have read papers in detail, which strikes me as a good idea. Collectively, those peer reviewers also represent a larger and more random sample of all ecologists than, say, the set of all ecologists on Twitter or Facebook (especially since anyone on Twitter or Facebook only follows or friends a non-random subsample of Twitter or Facebook users). So my way of filtering the literature may well be more effective at finding me stuff I didn’t even know I wanted to read, and better for helping me avoid groupthink and bandwagon-jumping. And conversely, I’m sure folks who filter the literature differently than I do can cite differences in detail that favor their way of filtering the literature. But I do think those details are just that–details–which I why I didn’t focus the post on them.

***Although I note with amusement that this is the most-viewed Plos One ecology paper of all time. Whatever the undoubted virtues of this paper, I think you’d be hard-pressed to argue that it’s been read 281,000 times because of the “objective” importance of its science, and not because of the first word in the title. 😉