Online supplements have ruined Nature and Science papers (UPDATED)

So, Back In My Day*, Nature and Science papers were a different beast from papers in other journals. A Nature or Science paper told a clean, simple, interesting, incisive story. They went to the heart of the matter and nailed it.** They were short because you didn’t need many words or pictures to fully describe what was done, why it was done, and what it implied. Kaunzinger and Morin 1998 is a good example.

I miss those days. Because nowadays, Nature and Science papers are mostly extended abstracts for dozens of pages of online supplementary material. Which is a totally different–and worse–thing. It’s not that people are still writing Nature and Science papers as they used to, except that now they’re also putting a bunch of additional inessential detail in the online supplements. I’d be fine with that. Rather, nowadays, you often can’t tell what was done, why it was done, or what the results were without turning to the online supplements. The paper itself only gives you the gist. Which is fine if the gist is all you want (and as long as the gist really is what the author says it is). But if you want more than the gist, you’re out of luck and you have to read what might as well be an Ecological Monographs paper.

And here’s the thing: it’s true that old-school Nature and Science papers didn’t tell you everything you might’ve wanted to know about the study. Which is why we used to have follow-up papers. It used to be that a Nature or Science paper often would be followed up by a longer, related paper in a leading specialized journal–Ecology or JAE, say. The follow-up paper would derive from the same study as the Nature or Science paper, and so would overlap partially with it, but usually would address some different questions. That doesn’t happen any more, because of online supplements. So one way to think of online supplements is as inferior, little-read versions of (or substitutes for) good ecology journal papers.

UPDATE: Just to clarify, I’m not against all online supplements. See the comments.

*I’m 43. I can use this phrase now, right?

**At least, the good ones did. And while some Nature and Science papers weren’t good, or good enough, even back in the day, overall I’d say Nature and Science’s batting average in ecology and evolution was mostly quite good.

18 thoughts on “Online supplements have ruined Nature and Science papers (UPDATED)

  1. Good point. I think this phenomenon is happening not only in Science and Nature, but also in many other journals. Online materials should only supplement the paper, but many times it’s impossible to fully understand or judge the validity of the central message without reading them.

    • “I think this phenomenon is happening not only in Science and Nature, but also in many other journals.”

      Agreed, though I think it’s made the biggest difference to how Science and Nature papers are written.

      • Are you saying that having supplements make Nature and Science papers less accessible (because so much goes into the often bloated supplements), or is it that you think all online supplements for papers (in any journal) are a problem? Personally, I think the supplement is a useful place to put details that are only likely to be of interest to a few readers of papers, that would previously have needed to be in the main part, but that aren’t of interest or use to most readers (say, a table with GPS coordinates of my study lakes).

      • @Meg (below):

        I agree that that sort of thing is the sort of thing that should go in supplements. But in practice, supplements often include other sorts of material.

        I think the problem is worst at Science and Nature papers because it’s really changed how people write them. Used to be that, if you didn’t have a clean result that could be fully described at Science/Nature length, you couldn’t go to Science or Nature. I think online supplements have changed how people write other sorts of papers too, but to a lesser extent.

  2. I’m so tempted to argue that most Nature and Science papers were always flashy but trivial anyway… but that could be sour grapes, since I’ve never published there🙂 But more generally, I agree. Online supplements are such a temptation: there’s no need to make serious decisions about what belongs in your paper and what doesn’t, because you can just put EVERYTHING in an online supplement. As Truman Capote supposedly said about Jack Kerouac: That’s not writing, that’s typing. [Here I should probably admit that I’m exaggerating slightly for effect… but only slightly.]

  3. The supplements also arguably make the paper less likely to be carefully reviewed, since even reviewers rarely look carefully at all the supplements. All that additional detail just ends up providing a false sense of security to readers.

    • Good question.

      Limiting the length of supplements is one idea. Might help, though for it to help I think you’d want a pretty short limit.

      A more radical idea is getting rid of online supplements. A leading neuroscience journal did just that years ago. In support of this radical approach, you could argue that the raw data are already being deposited on Data Dryad or whatever, so anybody who wants to see a bunch of alternative analyses or analyses of other response variables or whatever can just go do it themselves.

      You could also try writing detailed guidelines as to what online supplements are or aren’t for. But you’d have to have a strong editorial board to ensure that both authors and reviewers followed them. For instance, you’d have to have editors willing to ignore reviewers’ requests for additional inessential analyses/expts./etc.

      • But you’d have to have a strong editorial board to ensure that both authors and reviewers followed them. For instance, you’d have to have editors willing to ignore reviewers’ requests for additional inessential analyses/expts./etc.

        This sounds very much like you’re describing a fully professional editorial board, that may wish to make a final publication decision despite what reviewers may recommend. The sort of editorial boards that Nature and Science have…

      • @Mike:

        No, I don’t think so (EDIT: I mean, I don’t think it has to be a professional editorial board. I agree that it has to be a strong editorial board). When I was an editor at Oikos, I didn’t just blindly follow the recommendations of referees, or tell the authors to do so.

    • Someone recently made a similar comment to me, so I looked it up.
      “Science is also now accepting a few Research Articles for an online presentation (about 1 per issue). These are expected to present significant research results that cannot be fully presented in the print format and merit the extra length and attention provided in an enriched online format. These can be longer, up to 8000 words and include methods, additional figures and potentially videos, as part of the main presentation. Additional supplemental material that cannot be incorporated within the text is possible. Presentation will include an enhanced PDF version. The full text will be included in all digital versions of Science, and an enhanced abstract will be included in the print version. The cover letter should indicate why the additional length is merited.”

  4. I agree that lengthy online supplements make papers more cumbersome to review (and to read sometimes), but this info is often crucial for inclusion of data from these studies in a meta-analysis. In the past I often had to exclude Nature and Science papers from meta-analysis because methods were not described in sufficient detail and a lot of important info was missing, but nowadays things are better and these details often can be found in online supplements. So from the meta-analyst perspective I would not say OS have ruined Nature and Science papers – they can often be life savers, but I would say the authors need more guidelines in terms of what info to put in online material (as opposed to the main body of the papers) and reviewers need more guidance on the extent they are expected to scrutinize OS while reviewing the papers. Clearly, expecting a reviewer to read 100 additional pages per ms is not realistic, but on the other hand a danger with super-lengthy OS is that they provide ideal ‘burial ground’ for some methodological details that the authors would like to hide from the reviewers.

  5. The existence of unlimited online supplemental materials also allows reviews to ask for inordinate amount of extra analyses. For example, whenever I try to publish anything that includes a model that predicts how variable x will change under climate change, it is almost guaranteed that a reviewer will ask that I rerun all my analyses under all possible GCMs and all possible emissions scenarios, and just dump the results in the online supplemental materials. In many (most) cases, these extra analyses are entirely superfluous or irrelevant to the message of the paper, but I guess the reviewers figure that since online space is unlimited why not just run them all and see if anything interesting comes out. This is not the path to good science.

    • I work on theory and computational models and my experience has been very similar. If we show a result holds true under a fairly simple and general model, referees (and editors) invariably ask how general are these results under this modified assumptions, new type of stochasticity and of course, parameter sensitivity? (Some of these questions are asked even for analytical results!) If we don’t show results work with various versions modified models, then almost sure to be rejected on the grounds that there is no evidence for generality and robustness! So, to avoid such possibilities, most of my manuscripts have ended up with really long online materials even before it goes to submission! But we try a lot to make sure they well organized and readable.

    • I’ve had exactly the same experiences recently of journals and reviewers just asking for more and more and more. And my arguments that extra analyses are cumbersome or unnecessary are inevitably met with the response that I can just include them in online supplemental material. And I agree with Ken: the path to good science does not lead through voluminous online supplementary material.

  6. Lets be real here, Science and Nature have stopped being journals some years ago.

    They are profit making enterprises, nothing more. The mount of junk papers they now publish beggars belief.

    Springer actually published junk papers, as did IEEE, over 100 of them, reviewers read abstracts and if they like them boom published.

    Peer review is gatekeeping. If your reviewer is your competitor you don’t get published.


    Furthermore, peer review has been hijacked for theoretical fields.

    When you hear the Science editor saying “the debate is settled” on a subject, what chance do you have of getting reviewed if you disagree, science apart, valid or not, you have been prejudged not your science.

    Science in the 21 century is in dire trouble.

    • We clearly have different experiences and will have to agree to disagree. In my own field of ecology and evolution, the majority of Nature and Science papers are outstanding work in my view.

      With respect, I think and hope that even many who agreed with you would say you’re lumping apples and oranges, if you’re referring to the incidents to which I think you’re referring at IEEE and Springer. Nature and Science have never done anything like that. Not all peer review mistakes that happen at around the same time are symptoms of a single underlying problem.

Leave a Comment

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s