Peer review: the neverending story
It seems like there is no institution that is more criticized in science than that of the peer-review system — an no one that is less mutable. While published paper evaluation metrics are being revised (such as the recently introduced PLoS article level metrics, or the Australian National Health and Medical Research Council abandonment of the Thomson Reuters impact factor system), the peer review system seems like it is here to stay. When asked, most scientists would probably paraphrase Churchill: “peer review is the worst system for judging science, except all others that have been tried from time to time”. (However, Churchill did have other working state models to compare with Democracy, whereas peer-review seems to have no, um, peers.) The latest diagnostic comes from Errol Friedberg (no relation to me), editor in chief of DNA Repair.
“—if peer review is so central to the process by which scientific knowledge becomes canonized, it is ironic that science has little to say about whether it works.” — J.P. Kassirer and E.W. Campion, Peer review: crude and understudied, but indispensable, JAMA 272 (1994), pp. 96–9
The conclusion that was reached after a few annual scientific conferences published in the Journal of the American Medical Association as to the merit of peer-review were: “(i) blinding reviewers to authors’ identity does not usefully improve the quality of reviews, (ii) there is no association between reviewers signing their reviews and the quality of the review, (iii) passing reviewers’ comments to co-reviewers has no obvious effect on the quality of review, (iv) reviewers aged under 40—-write reviews of slightly better quality, (v) appreciable bias and parochialism exists in the review system. Finally, and perhaps most significantly, developing a useful instrument(s) to measure manuscript quality remains a huge challenge“. [and in the final analysis peer review] “can screen out [studies] that are poorly conceived, poorly designed, poorly executed, trivial, marginal, or uninterruptable.” No mean feat, really. But many scientists maintain that peer -review is a screen for quality and impact, not just for screening out bad science for funding agencies and for journals.
Neither Errol Friedberg, nor the authors of the congress proceedings seem to suggest alternatives. Rather, they present examinations of the process and its effect upon the final outcome. Friedberg also suggests that one constraint, that of page numbers in a journal, has been essentially removed with the advent of electronic publication, and thus more meritorious articles can now be published. Interestingly enough, many scientists — and journals — seem to value publication quotas, as those add prestige to those papers that do get accepted.
However, there are two things of which I’m certain: change, if any, will not come soon, but also we have not heard the last critique of the peer-review system.
Friedberg, E. (2010). Peer review of scientific papers—A never-ending conumdrum DNA Repair DOI: 10.1016/j.dnarep.2010.03.003
I think a better study would be focussed on how peer review works in terms of Grants and Funding. In other words, when there’s real money involved; I think the results of linking reviewers to grant applicants would be different. Just my two cents (not awarded yet!)