Poll: controversial ideas about peer review and scientific publishing (UPDATE: poll now closed)

Our poll on controversial ideas in ecology was a lot of fun and got a great conversation going. So here’s another poll, this one on (possibly!) controversial ideas about peer review and scientific publishing. These ideas are widely discussed online, but you can’t tell how widely supported they are just by looking at online discussions. People who participate in online discussions of any issue are a small and very non-random sample of all people, in terms of their opinions on the issue. I honestly have no idea how many of you favor double-blind review, or paying reviewers, or universal author-pays open access publishing, or etc.–really looking forward to finding out!

It’s only ten multiple choice questions, so it’s quick. I tried to include a range of ideas, including a few that would represent massive changes from the status quo.

28 thoughts on “Poll: controversial ideas about peer review and scientific publishing (UPDATE: poll now closed)

  1. Regarding the issue of paying reviewers, some journals already have some forms of compensation for reviewers, such as access to the journal for a limited amount of time, free color figures for future publications, free publication fee for future articles in that journal, etc. At its current form, the “compensation” is often restricted to the particular journal the reviewer reviews article for and therefore is often of very limited use. If these forms of compensation could extend to, for example, all journals or all journals for a particular subject by that publisher, it could be very helpful. This could be especially useful for junior scientists who may have limited resources to cover cost associated with publication.

    • It can fairly easily be pooled across journals published within a society. A society & its journals that I’m active with (SETAC) only publishes two journals, but reviewers get points that they can cash in to pay annual dues or save and use for an annual meeting fee. Administering it isn’t that hard. I no longer belong to multiple societies, but for those that publish journals (most of them) it seems a low hanging fruit.

      • I like this idea and it seems to be very doable. If reviewer points can be used more flexible for societal journals or conferences, it makes the bonus “points” more useful.

  2. Why not simply drop the publishers and each scientific society take over their respective journals and make them open access? We all know how to make pdf’s (require final formatted style submissions) and societies could elect editors who know who and how to peer review. We could fund it by charging a fee to authors outside the society and use society dues to buy the electronic storage space and website set up. Maybe even discount membership fees for peer review services. Given the low cost of epubs I also see no reason to not allow more replication and negative result type studies to be published.

    • I’m not sure if the finances work. See this old post for some discussion: https://dynamicecology.wordpress.com/2013/09/18/follow-the-money-what-really-matters-when-choosing-a-journal/.

      Also, presumably under that model you’d want to go to very relaxed and non-onerous formatting requirements, otherwise you’re downloading a lot of work onto authors.

      Another way to decouple publishing from filtering of the literature while also keeping publishing very cheap for authors is arXiv overlay journals: https://gowers.wordpress.com/2015/09/10/discrete-analysis-an-arxiv-overlay-journal/. Of course, that model is cheap for authors only because preprint servers are heavily subsidized by other funding sources.

    • There are several things that publishers do that aren’t really accounted for in your model:
      1) Managing review flow – its an enormous amount of work to move manauscripts from EiC to AE to reviewers, hound reviewers, hound AEs. Althouhg software helps that, the software is not free, and a fair amount of human intervention is still involved. I would do much less editor work if I was personally responsible for chasing reviewers. Tim Vines of the now defunct Axios always emphasized this “hidden” task. Journals spend several hundred dollars per paper on this function.
      2) Copy editing and typesetting. The quality of this varies these days. But dealing with a high quality production team is a wonderful experience as an author. And as more non-native English speakers contribute (a good trend!), the importance of this stage only increases. You can debate what level of standards we need to rise to in this area. But as Jeremy noted it is an enormous amount of time fiddling around to get figures & captions in a good place, equations looking good and verified, etc. And it is also a fair amount of time as a reader to parse poorly done work (especially equations, bibliographies, etc).
      3) Distribution. Even if you’re wiling to throw out paper printings (which I am), you have to have servers that organize and make the material findable. Most reputable publishers also pay into an archive service to guarantee that the articles are available online even if the publisher goes under and their website goes dark.. Just because you can serve a few pages to a few hundred readers per day with 95% uptime on a server in your office doesn’t mean you could or should serve 1000s of downloads per thousands of journals with that infrastructure.

      You can debate the level of #2 you require, but #1 and #3 are really non-negotiable. And on #2 I think it is easy to underestimate how much more time would be wasted if we would be wasted if references were accidentally omitted and sentences were ungrammatical on a regular basis (which make no mistake is the level we would get if we left it to authors).

      All told it costs a minimum of ~$500/article to do the above even with completely free editors and copy editing/typsetting on the low end. Not coincidentally the very cheapest society based online-only open-access journals charge about $500 (See the link in Jeremy’s comment for more details)

      Your current subsidy model is backwards for societies too. Membership fees don’t fund the journal, the journal subscriptions lower the membership fees and/or provide funding for scholarships, advocacy, etc. You can argue which is right, but there is no free lunch out there. Membership fees would go way up if they were supposed to cover journal publication.

      While I appreciate your bold thinking, and do think that the long term (10-20 year) model looks close-ish to what you describe, currently the trend is in the other direction. it is not a coincidence that in the last 5 years the 3 largest ecological societies (ESA, Nordic, British), not to mention Evolution and many conservation and allied journals, have gotten out of self-publishing and contracted publishing out to a commercial outfit (almost always Wiley). Each of the 3 tasks is surprisingly non-trivial. There are rumors that one of the societies own-built editorial tracking software was running on a 10+ year old PC and nobody knew how to resurrect it when the machine finally died. The economies of scale in all of these tasks are very real.

  3. For Ecology and Evolution, some European research institutes have coordinated to subsidize a system close to arXiv overlay journals, albeit open to submission from any pre-print server: https://evolbiol.peercommunityin.org/ and https://ecology.peercommunityin.org/
    They found that they would get better value for their money than through classical publishing systems, although the robustness of the system to changes in the research, economical and political environment remains difficult to predict.
    The community of “recommenders” (aka editors) is largely French right now, but anyone with good credentials could apply.
    They are actually proposing for other topics to set up a similar system (there is currently an additional one in Palaeontology): https://peercommunityin.org/

    • Because of the low cost of entry and leveraging of existing parts of the publishing ecosystem, I have long suspected this to be the most viable model for major change. I’ll be very interested to hear how this works out.

      This mostly addresses point #3 in my comment above to Joel (distribution). And I suppose it takes a stand on #2 as well that readers don’t care about production (copy editing, typesetting). How is #1 handled (managing the editorial flow of chasing reviewers, etc)? Is that now on the editors?

      • We have a poll coming up on Thursday that will speak to some of this.

        One possibility is that maybe a few (some?) readers don’t particularly care about who the reviews come from, or even about having reviews at all beyond what’s necessary to filter out serious technical errors. They just want fairly-reliable recommendations as to what to read. To give such readers what they want, you don’t necessarily need an editorial board with a well-managed workflow. You can just have some people who recommend stuff. Like our Friday linkfests, but for preprints. And I know in economics (and I think in some other fields) there are some experts who like to take to Twitter to share summaries of, and commentaries on, preprints they’ve read recently. Random example: https://twitter.com/lymanstoneky/status/1011272647727091712. So if your own subfield happens to have a critical mass of such experts, you can just follow them on Twitter and use their recommendations as one way of deciding what to read. Will be curious how many people in our upcoming survey say they do that.

        Of course, one problem with this way of doing things is that the people making the recommendations ideally do need to read carefully, and be skeptical and knowledgeable enough to catch errors. Because if your “filter” is just “whatever lots of random people, or lots of your friends, happen to be sharing on social media”, you’re going to end up reading a lot of stuff that went viral despite serious flaws. Often because it’s clickbaity or supports people’s confirmation biases. We saw an example recently with that shaky study claiming that grad students experience anxiety and depression at 6x the rate of the general population. The study went viral; the pushback from Meghan did not go viral. Pre-publication review at good selective journals isn’t infallible, but it’s much better at filtering out that sort of study than the “crowd” on social media, I think.

      • Well, I’ll wait for the poll. but I don’t consider tweets of papers to read because they’re important at all equivalent to peer review. – for many reasons. I hope I’m not in a minority on that. That’s one reason that I think Charlotte’s peer-review and arxiv bundling model is much more likely to takeoff than the simple arxiv & social media guidance model is.

      • The poll’s not that detailed, so it won’t get at some of this. In particular, it won’t get at whether readers use different filters for different purposes. Like, maybe you skim journal TOCs to find papers that you can (probably) rely on. And you also use Twitter recommendations to point you to interesting-sounding preprints that you shouldn’t take too seriously unless you carefully review them yourself or have some other reason to rely on them (e.g., “I know the authors have a great track record.”) Hopefully folks will comment about with more details about how they use various filters.

      • To answer Jeremy’s comment, I would definitely use different methods to search for papers and assess them: social media can be a good way to point at interesting papers, but it all too often favours catchy titles over sound studies. For the same reason, I am not convinced by Altmetrics as an assessment of a paper’s significance!

        To Brian: The reviewer search is handled by the editors (which is also the case for some journals), but the workload is diluted by the larger number of them. The main risk, IMO, is that cooptation does not guaranty the commitment and quality of editorial work, but open reviews should help.

  4. My wish:

    A journal that is low cost, open-access, with a latex template (or similar) that automates most of the process of getting the article in the right post-production style (with the authors doing most of this work). Reviewers are menially paid (e.g. $50 for a review of reasonable quality), **but they can wave pay for half off a publication in the journal (two reviews = 1 free publication)**. The goal here is to create a community of high quality reviewers and authors, who review as much as they publish. I’m not so sure how feasible this model is, probably depends mostly on how good the post-production automation is, so it doesn’t put too much work on the authors.

    Pre-minor-revisions the paper can be submitted in any reasonable format (but section structure and word counts should be followed at this point).

    The profit margin of big publishing companies is abhorrent, they should be undercut.

    • The idea of letting people “pay” for the reviews their ms submissions receive by doing reviews themselves was the core of the “PubCreds” idea Owen Petchey and I once pushed, the main difference being that we wanted to do it with a virtual currency rather than actual cash: https://dynamicecology.wordpress.com/2011/04/22/an-oikos-editor-and-a-former-editor-are-fixing-the-peer-review-system/. One can think of it as a mechanism to enforce the “golden rule of reviewing” (review two papers for every one you submit). Many others have had the same basic idea, before and since. I’m still fond of it. But I think it’s hard to get off the ground (lots of details to be worked out) and I don’t get the sense that there’s massive passionate support for it.

      Re: LaTeX templates, I’ll just leave this here:

      😉

      • I like this idea or something like it (see below). Love the infographic. I try not to collaborate (for reason that are all my fault) so I am always in that lower blue box. I stopped using Word and switched to LaTeX the day a journal’s manuscript team wrote to me and told me I had to re-write all my equations using Word’s native equation editor and not the plug-in equation editor that came with word.

        I don’t get the dissing of LaTeX. Especially given a good template. “There is even a scientific study that shows that latex users would be faster if they used word (but enjoy the process less)”. Probably. But If journals gave incentives to submit accepted manuscripts in journal format – and the journal provided a template to do this – LaTeX would be much faster, I would think.

  5. Curious how multiple choice can force one off the fence. I selected “single-blind” even though intellectually “double-blind” should be better. If double-blind reviewing just pulls the wool over the eyes of reviewers by withholding conflicts of interest declarations, or sources of funding and thus potential funding bias, it’s a questionable practice. (https://ieamblog.com/2018/06/19/peer-review-and-implicit-bias-is-double-blind-peer-review-better/). Maybe in “pure” ecology field where I think most DE readers swim, it’s not such an issue. But for example, in studies of ecological effects of economically important chemicals, thee who pays the piper, may be calling the tune.

    • My assumption had always been that it was the editor’s job to police conflicts of interest, not reviewers’ job. Is that not the case in environmental toxicology?

      • Hmm, not sure what’s going on. There are no comments in our trash bin that shouldn’t be there. And I haven’t gotten alerts about any new comments that later vanished. Can you try reposting one more time?

      • Note from Jeremy: Chris Mebane has run into technical difficulties posting comments, so he emailed me the following and I’m posting it for him.
        ********************
        Policing conflicts of interest isn’t a straightforward thing. Sure, whether disclosure statements are included can be policed, but its quite another matter to detect subtleties in the framing of questions, design of observations, interpretation, and which prior works are cited or not. Scientists like to do studies, studies often require sponsors, sponsors often have interests, scientists have pretty good ideas about what kind of findings the sponsors are interested in, and pleased sponsors are more likely to go back to the same scientists for further study than are displeased sponsors. Findings that are congruent with funding aren’t necessarily wrong, but funding disclosures at least tip off readers to be appropriately skeptical. Why handicap the reviewers, who are the first line of defense against bias? Community policing takes a village. This is why I argue for a variant on the Brian McGill GEB soft-mandatory double-blind reviews that would retain funding and conflict statements (https://ieamblog.com/2018/06/19/peer-review-and-implicit-bias-is-double-blind-peer-review-better/). I don’t think ecologists should assume that their field is immune from such concerns.

        Also, if editors are supposed to police conflicts, who polices the police? Peer editors are usually practicing scientists and have their own skin in the game. They can direct papers to allies or assassins. Editors seem to have been getting a pass in the discussions of peer review models, compared to other aspects. In niche topics, a single topical expert editor/reviewer can have great sway. See for example a case that recently came to light through litigation disclosures (https://www.smh.com.au/lifestyle/health-and-wellness/toxic-secrets-professor-bragged-about-burying-bad-science-on-3m-chemicals-20180615-p4zlsc.html; “Toxic Secrets: Professor ‘bragged about burying bad science’ on 3M chemicals”)

      • Maybe too many words 🙂 Thanks, and thanks again to the three of you for going to the ongoing trouble of maintaining this blog. I don’t know how you do it. It’s a neat sense of community that strikes a nice balance among topics. Discussions are always as interesting as the posts. Hope the administration/moderation of those is low maintenance, for I’ve never seen an intemperate or inappropriate reply. I have to say, your neighbors’ pond over at Small Pond Science got a lot smaller when they stopped allowing discussions.
        Cheers,
        Chris

      • Thank you for the kind words Chris, and for being one of our most active commenters.

        Comment moderation takes us almost literally zero time. WordPress’ spam filter is essentially perfect; it blocks all spam without blocking anything that’s not spam. And we hardly ever get comments that we need to block or commenters we need to ban, except sometimes on posts about gender and equity issues. As far as I know, that’s the experience of other ecology bloggers as well.

        We had a discussion here a while back about Terry’s decision to close comments at Small Pond Science, in which Terry was kind enough to participate. He’s since reopened them for some posts but not others. We’ve made a different choice than him on that front, but that’s absolutely fair enough. Different bloggers have different goals and priorities, which is totally fine. I don’t know how Terry feels these days about the decision to close comments on some posts at SPS–if he feels it’s working out well, or what.

        And reader priorities will differ too. As a reader of SPS, I’m like you–I wish the comments were still open on all posts (just as I wish Rapid Ecology allowed comments by default on all posts). But I’m sure there are other readers who don’t miss the comments at all, or are even glad they’re gone on some posts. Just as I know there are readers of this blog (and would-be readers, and former readers) who wish Brian, Meghan, and I would change some things about it. For instance, I know there are a few readers who feels strongly that some of the rhetoric Brian and I use (“zombie ideas”, “statistical machismo”, etc.) has no place in scientific communication and wish we’d drop it. As another example, we know from reader polls that there are a few readers who’d like to see us post more often about gender and equity issues–and another small set of readers who would prefer if we never posted about those issues. There’s no pleasing everyone all the time as a blogger, and deciding which people to displease over what issues isn’t always an easy call.

  6. Pingback: Controversial ideas about scientific publishing and peer review: poll results and commentary | Dynamic Ecology

  7. Chris, I just came upon this remark a few weeks after you left it — about how I am running an experiment at Small Pond where comments are opened up only for some posts, which has been going on for about half a year so far. I thought I’d add some context for you with a few pieces of information. To be clear, I am still “allowing” discussions — I can’t stop them! They’re all over the internet, on facebook, on twitter, and apparently, on other blogs. Second, it turns out that my traffic on posts where the comments are shut off has increased, if that’s the metric you have in mind with respect to the size of the pond. Third, I’ve received far more compliments and thanks than negative remarks about closing comments. The only negative remarks I’ve gotten are from a few folks who seem to feel entitled to real estate on my blog, who think it’s somehow their right to be able to publish their remarks on posts. This doesn’t really encourage me to open up comments on all posts again. But hey, we’ll see, I’m still running the experiment.

    • Thanks for taking the time to share this Terry. Glad to hear you feel your experiment is succeeding so far.

      “Second, it turns out that my traffic on posts where the comments are shut off has increased”

      Interesting. Can you clarify the comparison you’re making here? Are you comparing traffic on recent SPS posts that allow comments to traffic on recent SPS posts that don’t allow comments? If so, do you think maybe that comparison is confounded with post topic?

      • I’m comparing posts without comments section open to posts from the previous six months, where I had comments open for all posts. I haven’t run any stats, I’ve just glanced at the numbers and saw the means are slightly higher, I don’t intend to infer any causality.

Leave a Comment

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.