Following on from previous posts on reforming peer review (see here, here, and here), I wanted to note a new peer review service, Peerage of Science. PoS is a private company founded by a trio of Finnish scientists, which combines several proposed peer review reforms into one package and then offers them as a service to participating journals, for a fee. Member scientists, called “peers”, can submit mss to the service, which other peers can review (anonymously). Members are obliged to review in appropriate proportion to how much they submit, an element of PoS which echoes PubCreds. Participating journals can see the mss and the reviews, and notify the authors if they want to accept the ms. Authors then can choose their preferred offer of acceptance, an element of PoS which echoes ExpressO. To provide additional incentive for scientists to join (which they can do for free, at the invitation of a current peer), and to perform reviews, peers can score one another’s reviews for quality. So if you’re a good reviewer, you can build up a good score which, at least in theory, is an objective number that you can put on your cv. There are also plans to publish the best reviews in a commentary-type journal, to provide an additional incentive to review. Incentives for review have of course been widely discussed.
PoS is just getting off the ground. They currently have about 500 peers, many of them Scandinavian ecologists. Our sister journal Ecography is currently the only participating journal.
Neither I nor Oikos endorse PoS (full disclosure: I was invited to become a peer, but declined for personal reasons), but I do find it interesting to think about and so wanted to post on it.
As with other proposed peer review reforms, I think PoS will succeed or fail depending on whether a sufficient number of the “players” (authors/reviewers, journals, publishers) think it’s to their benefit to use the service. Any serious proposal to reform peer review either has to respect, or have some plausible way to change, the incentives faced by authors, journals, and publishers. For authors and reviewers, I think the incentive to join depends very much on the number and identity of the participating journals. Prospective peers are going to be asking themselves, “Will joining PoS let me publish more papers, faster, in better journals, than I otherwise could?” Frankly, I don’t think the possibility of accumulating a good review score, or of having some of your reviews published in a “journal of peer reviews”, is going to do much at the margin to attract those who wouldn’t otherwise join. What would be attractive would be the possibility of having many journals “bid” on your mss, based on only one set of reviews. But there may be a catch-22 here, because I’m not sure the service will be attractive to journals unless there are many, many peers submitting and reviewing many, many mss, so that journals feel like they need to be able to tap into that ms stream. This is hard to judge, though, as much depends on the fees that PoS charges. How much is it worth to journals or publishers to essentially outsource their reviewing? PoS also is going to be competing with the “cascading review” services that publishers have begun to offer. Publishers are more than happy to have effectively simultaneous submissions–to their own journals–and to share reviews–among their own journals.
It’s also possible that some scientists will be uncomfortable joining PoS because they will feel like they’re “working for free” for a private company. But on the other hand, reviewing for any journal except a non-profit journal amounts to “working for free” in the same sense, as does submitting to any for-profit journal that charges author fees. There are complex issues here which need to be unpacked.
UPDATE: Mike Fowler pops up in the comments with some trenchant thoughts, and a link to an excellent post on PoS at his own blog. Mike’s clearly thought much harder about PoS than I have!
Dynamic Ecology 
True, many of the Peers registered in the service (today numbering 558) come from Finland, Sweden, Norway and Denmark because the community grows via invitations and started in Finland. But already 58% come from other parts of the world, mostly from Europe, USA, Canada and Australia. Altogether the community has members from 28 different countries.
(full disclosure: I am one of the founders of Peerage of Science)
Hi Jeremy,
I posted a less succinct review of PoS over here recently, looking at the service from the point of view of author, reviewer and editor. I hadn’t heard of ExpressO before, but it all starts to look very familiar. ExpressO seems to be a success at first glance, but there are likely to be differences between legal and scientific publication that mean PoS won’t necessarily be a success just because it’s based on the same format.
I agree that there seems to be a bit more thought required to flesh out the main idea(s), or to communicate the fleshy bits better to potential users with PoS. Some changes have been main recently that clarify some of the questions I initially had (e.g., the PoS Process Flow is a helpful guide to how things should work).
This point goes to the heart of one of the biggest problems with PoS, I think. Competition. The journals we want to publish in are already saturated. They can already pick and choose what to publish based on the fact that everyone wants and tries to submit there. If PoS is a success (whatever that might mean, but let’s say here for simplicity that everyone submits all their articles to PoS), there won’t really be any change in the sort of papers that get published in the (perceived) top journals, so there will be no competitive advantage for authors in that sense. They may get a publishing offer from a less prestigious journal, which would probably be similar to going through things the way we currently do. It may not be much faster, given the uptake of cascading reviews by traditional publishers. PoS may even slow things down if some authors just hang on that little bit longer in the hope of attracting the attention of a ‘better’ journal in the system.
Contrary to you, though, I think it’s worth being onboard as a ‘Peer’, to follow what’s going on, and perhaps influence things for the better. Bet-hedging? Moi?
All very well put Mike. Apologies for not doing a bit of googling and finding your post sooner; had I done I would certainly have linked to it.
@Mike:
Upon further thought, I do think a successful PoS (defined for simplicity as everyone submitting all their mss to PoS) does have some advantages from the authors’ perspective over cascading peer review systems. Leading journals aren’t all published by the same publisher. It may well be that everyone wants to submit all their mss to the top journal. But when those mss are rejected, many authors prefer to go to another leading journal, from another publisher, rather than having their ms cascaded down to a less-selective or unselective journal. PoS does potentially offer such authors some significant time savings, and does offer leading journals some possibility of capturing some mss that might otherwise go to other leading journals.
Yep, that’s fair assuming that publishers only wish to maintain the reviews in house.
However, the Neuroscience Consortium includes journals from a range of publishing houses and academic societies, suggesting that academics aren’t willing for such a system to be restricted in that way.
I don’t see any reason why such a reciprocal system couldn’t also work well for ecology/evolutionary biology. Other than perhaps ‘cascading’ Editors not being happy with other journals’ choice of reviewers, which might then also be a problem with PoS (which they hope to get around by grading the reviews).
I don’t think that many Editors will see this as a problem. They will spend less time seeking reviewers, and those reviewers they do have to find may be more willing to accept invitations given the lower burden on their time from fewer review requests.
Please help me out here: what makes you say that Peerage of Science is based on the same format as ExpressO? To me it seems there is nothing even vaguely similar, and also apparently the entire publishing field is very different from science?
With ExpressO, law scholars pay the service to deliver their manuscript to several law-review journals ($2.20 per journal) for consideration. What is not clear to me how this is linked to peer review (I do not know exactly how law research is published, but apparently law reviews are not peer-reviewed but handled entirely by editors – Buckley 2007: It’s Time to Stop the Blind Leading the Sighted: A Proposal to Improve the Editing of U.S. Law Reviews)? What I read from the FAQ’s from the site is that this is just a handy way for electronic submissions in a field where most submissions are still sent on paper and require author’s cv to accompany the submission? And journals handle the manuscripts independently, I can not find information about the bidding system Jeremy refers to (but maybe I just miss it)?
I am unclear on why you need help. My post provides all the help you should need.
My understanding of ExpressO (and I will correct the post if you can prove me wrong) is that authors are not obliged to take up the first offer of acceptance they receive. Please see the testimonials page:
http://law.bepress.com/expresso/author_testimonials.html
where authors are quoted talking about the multiple offers of acceptance they received, which means they’re free to choose their preferred offer, just as with Peerage of Science.
Yes, there are differences in who pays for the service. My posts on ExpressO and Peerage of Science state who pays in each case. I believe my posts are clear on the specific way in which ExpressO works like Peerage of Science.
Nothing I have posted said anything about how reviews work in ExpressO, or implied that reviews in Peerage of Science work the same way as in ExpressO.
Readers are free to click on the links I provided to each service and judge for themselves whether or not ExpressO is even “vaguely similar” to Peerage of Science. But I note that, in the comments on my post, one of your own Peers, Mike Fowler, checked out ExpressO and said “it all starts to look very familiar”. So he at least, finds ExpressO at least “vaguely similar” to Peerage of Science. Perhaps you will also ask Mike to “help you out” on the specific ways in which ExpressO is “vaguely similar” to Peerage of Science.
p.s. to Janne-Tuomas: I’d have thought you’d welcome comparison to ExpressO, since ExpressO seems to have become quite successful in quite a short amount of time. Even if its success is for reasons that are partially or even mostly specific to the legal field, its success at least proof of principle. It shows that something like Peerage of Science can work in the appropriate circumstances.
This does sound like a cool, innovative approach to the problem, and I really hope they expand and are successful. However, I am somewhat skeptical because of their community-building approach. As you mentioned, The success of this model will be strongly tied to how many “players” see its value and are willing to engage. However, from what I understood, you can only become a “peer” in the system if (a) you are invited or (b) you submit a manuscript. I fear this will create an “echo chamber”: invitations are distributed according to personal contacts, which should bias the scope of expertise available, therefore limiting community expansion through (a). This, in turn, makes the service unappealing to authors of (even slightly) different fields, and thus not allowing it to grow through (b). Many social networks have suffered from this business model. But I do hope time proves me wrong!
@Jeremy, thanks for updating with the link. I actually wrote my post last November, but it’s taken me that long to edit it down to a slightly more manageable length. And stuff my face with festive goodies.
@Janne, apologies if I overgeneralised the similarities with ExpressO, but I think the they are pretty clear: authors submit their manuscripts to an independent online handling service, which then acts as a go-between between article and journal.
I don’t think the explicit financial cost authors pay for the ExpressO service is really all that different from the credit system Jeremy has proposed in the past and PoS have adopted, just a different currency.
There is a simple exchange of currency for services rendered. This is not necessarily a bad thing, we just have to figure out as a community what currency we believe to be optimum.
@Rafael, I see it as being an opportunity (and/or requirement) for the peers to outreach to scientists in new fields. I’ve already been talking to physicists about this venture, but they’re pretty happy with the arXive system already, so persuading them to change will require some clear advantages.
Pingback: Friday links: ambitious new ecology blog, and more | Dynamic Ecology