Publishing, Peer Review and Quality Certification in the Digital Age

Every month the Special Collections department of the University of Amsterdam hosts a book salon, each focusing on a special theme. Last Thursday’s gathering focused on ‘the scientific publisher in the digital age’ and brought together a panel of three experts on the subject. Cees Andriesse, Emeritus Professor of Physics at the University of Utrecht and the author of Dutch messengers. A history of science publishing, 1930–1980 (2008), was accompanied by Geert Noorman, the head of NUV (Nederlands Uitgeversverbond – Dutch Publishers Association) and Eelco Ferwerda, publisher digital projects at Amsterdam University Press (AUP).

Questions leading the discussion were: What is a publisher? What is its function in the digital era and how is the Internet affecting the publisher’s value proposition?

Professor Andriesse, the first speaker, stressed that the main incentive for scientists to publish is reputation building and not money-making. Their foremost aim, even though this might seem ideological, is the improvement and enlargement of human knowledge. In this respect publishing is a must: scientists can only deepen their insights through dialogue with fellow scientists scattered over the earth. To ensure the growth of (accurate and valuable) knowledge, scientists let their fellow researchers judge their work on originality and correct reasoning. They then give a (honest) judgment on the value of the research, mostly in a nuanced manner with suggestions for improvements. Of course this system knows its critics, but as Andriesse states, it is the only true way to establish quality: the scientific journal is what it is today because of peer review and the organization thereof. And this is the primary task of the publisher; together with his editorial board and secretariat, the publisher is to ensure the proper functioning and arrangement of the peer review.

Andriesse goes on to consider what the influence of the Internet is on scientific publishing and thus on the peer review process. How does the Internet influence and shape peer review? As Andriesse states, being a publisher is increasingly a question of personality and networking. It is the personal qualities of a publisher that makes his or her name and brand and this will continue to be the case in the digital era. He also states that when it concerns the arrangement of peer review, a lot of discretion is needed. Andriesse ends his talk by discussing two famous Elsevier publisher-scientist combinations: Roosenvelt and Frank (Nuclear Physics) and Akert and Remarque (Brain Research). He concludes by stating that although the shape and the communication of peer review has become digital, this has not accelerated the process significantly. Emerging models like PloS and Biomed are on the rise but they are not really a distraction from all the, as Andriesse states, crap on the Internet. Andriesse clearly states that the Internet can add near to nothing to the scientific journal and its peer review.

Eelco Ferwerda, the second speaker, takes another starting point and discusses two young publishing companies, both Open Access, but each playing a different role. He states that Open Access is a growing phenomenon, where for instance the Dutch funding organization NWO has recently chosen to pursue an open access policy. This rise of Open Access is a direct result of the nature of science and the new possibilities for publishing: computers play a tremendous role in the gathering of information, and re-use should not be prohibited via copyright. Creative Commons and free licenses offer possibilities in this respect, Ferwerda states.

He starts by discussing Hindawi a STM publisher which has since 2007 been completely Open Access. And they make a profit. They make use of a very ‘reductionist’ model: they do not have any direct contact with their editorial boards, everything gets handled by mail. Ferwerda states that the business model of Hindawi is completely focused on growth and profit making and on the development of new journals. By doing market research in Web of Science they search for the best scholars in a certain niche and build up a new journal around them. According to Ferwerda Hindawi thus uses a modern and strictly commercial model with a quantitative and a-personal approach to quality. Open Humanities Press on the other hand follows a different strategy. Founded by academics in 2006, their aim is to remove the cultural barriers that inhibit scientists to make full use of the digital possibilities; their strategy is centered on trust and quality. They seek out the best, well-known scholars to support their product (also e-monographs) and they give a home and a quality stamp to journals set up by academics. By establishing connections with the Library of the University off Michigan and the Public Knowledge Project (with their OJS software) they hope to work more efficiently. They operate on a volunteer basis. As Ferwerda says, they don’t offer money but quality through both their business model and their network. The question is however, as Ferwerda ponders, if this will be a sustainable model in the future. Amsterdam University Press is somewhere in between these models: although they are in many ways a lot like OHP, AUP is a professional publisher that needs to make money. For them Open Access is not so much an enthusiasm as it is a real business model. But as OHP shows, the web has led to a renaissance of scholar-lead publishing which is forcing publishers to rethink their value: they are foremost brand builders, organizers, sellers and distributors in the digital age.

As a final example of new developments in science publishing Ferwerda mentions environments like H-net, a portal for researchers and educators covering all areas of the Humanities and Social Sciences. H-net is essentially a discussion network for knowledge exchange. Every sub-field is moderated by a team of scholars and an editorial board. Although not a publisher, H-net is an example of a semi-official form of scientific knowledge exchange. What will this model bring in the future and what will this mean for the publisher? – Ferwerda asks. 

Geert Noorman, the final speaker, brings the focus back to issues of peer review and the Internet. First of all he states that not every article needs an extensive discussion. Peer review also fulfills another function, namely that of ranking research. Being a reviewer is even a form of ranking or reputation building. And doing peer review is important work. And it is still work done by human beings that are fallible, which means mistakes are also still being made. However, as Noorman states, peer review is the only instrument to classify the results of science. It is hard work however, with, as he estimates, between 1-1,5 million articles published yearly. Do we still have enough reviewers (who can and want) to perform this job? Hasn’t peer review become old-fashioned and shouldn’t we replace it by usage statistics? Noorman clearly urges against this notion, citing figures showing that peer review is still very popular. According to these figures only 20% of the people think the current or classical system of peer review is no longer sustainable. 86 % however states they find it very valuable work to do and 91,5 % of the authors says it helped improve their work. When it comes to the digital developments, 73% of the reviewers say digital technologies has made their work easier the last 5 years. What is missing however, says 56 %, is proper information about how to conduct peer review. As to the future of peer review, the research shows processing tools will definitely increase in importance. And this is where, as Noorman states, publishers and universities could play an important role. The Internet could mean a lot to peer review: it builds communities, enlarges communication and it assists peer reviewers in doing their work faster and more efficient and it also enables meta-analysis.

However, as Noorman states, focusing solely on citation scores is a bad thing. Peer review will never become obsolete, although 25% of the researchers mentioned in the previous research think user statisctics might be an alternative to peer review). A thing publishers could focus on is the development of new tools and services that assist with better and faster peer reviewing like CrossCheck, the plagiarism detector by CrossReff.

These kinds of tools will make sure peer review remains alive and kicking in the digital age. Noorman states there should be more attention towards peer review in post-doctoral education, as it is a skill that needs to be trained and it is increasingly part of ones scientific responsibility. Noorman concludes by stating that usage statistics in some cases can be useful but that they miss the discussion element. And scientific discourse will always stay essential..

In the following discussion it becomes even clearer that the participants feel that the review of scientific publications remains essential. However, as Eelco Ferwerda states with pre-print repositories like arXiv, it gets a different function: peer review is more essential for ranking and branding afterwards, and less for direct scholarly communication. Peer review in this sense can be seen as a certificate, it is the end of a discussion, a final qualification: without this qualification you will not be admitted to the scientific annals, it serves as a threshold. With the online comments and the discussion on the preprint version a lot of rubbish also gets sifted out. In the end every article will have to be certified some where, some time. 

After a heated discussion on the need for Open Access (which according to some of the participants is being imposed on scientists by the government, publishers and libraries where others stated that Open Access was initiated by scientists) and the perceived ‘hidden agenda’ of the Dutch governments and its Open Access policy (which according to one participant will be used as an excuse to cut back on scientific funding even further, which was received very skeptical  by the majority of the participants), the concluding remarks focus on the fact that peer review as a process is first of all the property of the scientific community. But to organize it we still need publishers. To make things work better scientists should make clear what their needs are and publishers should keep on showing how they can fulfill these needs and how they can add value to the research process and outcome.

The problem, I felt, of the whole discussion on peer review as reflected on above, is that the speakers most of the time seemed to conflate peer review with (one of) its function(s): the certification of research as being qualitative. What the panelists seemed to essentially agree on was foremost the importance of the certification of scientific research by other scholars (the filter function), for which peer review is just one method. The lack of definitions used to describe peer review during the evening (not one definition was mentioned, if you don’t count ‘classical’) seemed to ignore the fact that not only there are already different levels and manners of doing peer review (from open to semi-open to blind etc.) there are also different methods to perform peer review per field. The difference is also huge between how peer review (often more an editorial process) is conducted in the HSS and in the Sciences. Statements made during the evening like ‘peer review will remain important in the digital age’ thus became quite meaningless with a term and a practice that can have so many meanings and manners. More important to question is what kind of peer review will become more important, and even more, how can we help it, advance it, adjust it or complement it in the digital age with the (alternative) tools and methods at our disposal (which are the more interesting questions concerning peer review I feel).

Next to that we might also start thinking about alternatives to peer review that still fulfill the same basic function as peer review in order to make this process more efficient. In this respect the article ‘On the future of peer review in electronic scholarly publishing’ by Kathleen Fitzpatrick gives many insights, where she separates the structure of peer review from the purposes it serves (and those purposes we want it to serve and those it actually fulfills).

She distinguishes two functions of peer review: fostering discussion and improving the work, and quality filtering (two functions Ferwerda and Noorman also already touched upon briefly during their presentations). However, the first can also already take place during the research process in an open setting, using user comments on the preprint and focusing more on the communication between scholars. Fitzpatrick goes on to establish the benefits these kind of open peer review systems offer to scholarly communication: 

“Vast amounts of scholars’ time is poured into the peer review process each year; wouldn’t it be better to put that time into open discussions that not only improve the individual texts under review but are also, potentially, productive of new work? Isn’t it possible that scholars would all be better served by separating the question of credentialing from the publishing process, by allowing everything through the gate, by designing a post-publication peer review process that focuses on how a scholarly text should be received rather than whether it should be out there in the first place? Would the various credentialing bodies that currently rely on peer review’s gatekeeping function be satisfied if we were to say to them, “no, anonymous reviewers did not determine whether my article was worthy of publication, but if you look at the comments that my article has received, you can see that ten of the top experts in my field had really positive, constructive things to say about it”?”

It is this discussion on how to really improve scholarly communication foremost (which in my opinion comes before quality – not saying that quality establishment is not also of the utmost importance) with the digital possibilities which I felt was missing[*] in especially Andriesse’s and also Noorman’s discussion of this system and which made the evening into a not very exiting all-agreeing praising of systems we now actually have the chance to improve – apparently not during these kind of panel discussions however. 

By the way, Kathleen Fitzpatrick is currently offering her book manuscript, Planned Obsolescence: Publishing, Technology, and the Future of the Academy for public review via MediaCommons Press.

---

[*] Next to a further discussion on the rise of scholarly- and library-led publishing initiatives, which can certainly be seen as threatening the traditional roles of the publisher when it comes to both arranging peer review and to arranging the production and distribution of scientific content.