Issues Series | Responding to Criticism: The State of Education in America, Part II:
[continued]
read introduction
read Part I
Peer Review and a Proposal for Revision

by Aaron Barlow

For a long time, peer review — the way many academic papers, books, and grant applications are evaluated — has been one of academia’s unexamined bits of wisdom, often assumed (without thought) to be right and good. Chris Mooney, writing on the Web site of the Committee for the Scientific Investigation of Claims of the Paranormal, claims that:

The rigorous vetting of unpublished research by independent, qualified experts — what’s often called “peer review” — is an undisputed cornerstone of modern science. Central to the competitive clash of ideas that moves knowledge forward, peer review enjoys so much renown in the scientific community that studies lacking its imprimatur meet with automatic skepticism. Academic reputations hinge on an ability to get work through peer review and into leading journals; university presses employ peer review to decide which books they’re willing to publish.

Yet it is a cumbersome, time-consuming process that can act as a brake on new and exciting thought. In addition, it may no longer even be necessary, but replaceable by more efficient and useful Internet vetting procedures.

If we could use peer review as one criterion among many for making personal choices of what to read, academically or otherwise — that is, if the rejected articles were also made available to me — I might be in favor of the process. As it is, however, peer review exists as something of a system of prior restraint. That is, it keeps works from appearing. This is somewhat akin to having Markos Moulitsas Zúniga, the proprietor of the hugely popular Democratic/progressive political blog Daily Kos, demand that all diaries be approved by a committee before appearing — instead of, as is the case now, allowing them to appear without managerial review. Markos does promote diaries he particularly approves of to his "front page," but he rarely stops a diary from appearing at all (though a very small percentage is deleted after posting). Were Markos to do otherwise, he would set my finely tuned free-speech antennae aquiver. Peer review in advance of publication also approaches censorship and needs to be rethought, if for no other reason than that the contemporary explosion of publishing and response possibilities which make the constraints it enforces unnecessary, relics of past publication-industry strictures, not a part of today's world of continual expansion.

I shouldn’t worry: Peer review may disappear without outside help. With so many avenues appearing for publication, the peer-reviewed journal or book is already becoming something of a rarity — though inclusion in one is still seen by too many as a sign of academic worth — and they may soon die from lack of attention. More and more academics, including me, are avoiding peer review completely, deciding they are no longer willing to tailor their works to constrictive standards (fearing that nothing radically different will ever get through), but wanting to be able to take chances and to see their work stand or fall on its own ability to interest readers and influence thought, not on the basis of a small committee of “peers.”

Furthermore, the peer-review process has traditionally taken a great deal of time. For even an article, a matter of months or more may go into the initial research phase of the project, with months more for the writing. Only then does the author start the process of finding a suitable journal. In each case, an editor decides whether or not to send the article on to referees (reviewers). There would likely be three, all having expertise in the field of the article, often with none of them knowing the identity of the author. Once comment has been given, the editor decides whether or not to take the article, ending a process that can take months to a year or more. If the article is accepted, rewrites might be demanded, so that more months pass before the article sees print. Furthermore, there is always the danger that the reviewers — whose names are also often hidden from the author — might have axes to grind or their own pet projects to promote, skewing their evaluations and frustrating the research and publication process still further.

Peer review has not been updated to meet current times and needs — or the explosion of scholarship over the last decades, especially when you take into account scholarship’s own “globalization.” There has only been stagnation, so much that peer review has become little more than a means of evaluation for professional academic advancement, something that has itself restrained full use of the Internet in academic discourse. As Jean-Claude Guédon, writing with Raymond Siemens, says in “The Credibility of Electronic Publishing: Peer Review and Imprint”:

[P]eer review is so important not so much because it allows for the selection of the best materials, but rather because it is widely used to manage personal careers. And the practices generally in place have contributed to creating a very strong link between individual evaluation and imprint through the conflation of individual value with journal impact factors. No wonder that electronic publishing is viewed with caution as it obviously is bound to modify a number of rules that will affect power structures within scientific communities.

Beyond this, peer review promotes inbreeding. Writing in the blog of if:book: A Project of the Institute for the Future of the Book Ben Vershbow states:

It’s unfortunate that the accepted avenues of academic publishing — peer-reviewed journals and monographs — purchase prestige and job security usually at the expense of readership. It suggests an institutional bias in the academy against public intellectualism and in favor of kind of monastic seclusion (no doubt part of the legacy of this last great medieval institution). Nowhere is this more apparent than in the language of academic writing: opaque, convoluted, studded with jargon, its remoteness from ordinary human speech the surest sign of the author’s membership in the academic elite.

This crisis of clarity is paired with a crisis of opportunity, as severe financial pressures on university presses are reducing the number of options for professors to get published in the approved ways. What’s needed is an alternative outlet alongside traditional scholarly publishing, something between a casual, off-the-cuff web diary and a polished academic journal.

The advantage of the peer-review process, when there was limited space for articles, was that a good deal of dross was removed before any peer-reviewed journal saw print. Scholarship moved forward in an orderly, if sometimes dreary, fashion. But peer review also contains problems (and not simply in the inordinate amount of time before research and publication), including the assumption that an article, if peer reviewed, has been adequately vetted. This is a dangerous assumption, as Gary Mauser points out in his article “An academic scandal: The importance of peer review”:

The rise and fall of [Michael] Bellesiles [Emory University historian and winner of the Bancroft Prize in 2001 for a book whose research on gun ownership in early America was later called into question, to say the least] opens a window on a more general problem, the fragility of peer review. According to professor emeritus David Bordua, at the University of Illinois, the real scandal concerns the willing gullibility of ideological reviewers and academic historians. The Bellesiles case represents a monumental failure of peer review. When his work was first published, reviewers hailed the book apparently without out bothering to evaluate the research. The book even won the Bancroft Prize without anyone on this prestigious panel bothering to check his footnotes thoroughly.

It was peer-reviewed? Then it must be correct. So, peer review not only constricts publication but also cannot be trusted as a vetting process. Still, the concern about dross getting through remains and is actually greater now than it ever was. Peter Roberts in “Scholarly Publishing, Peer Review, and the Internet” for First Monday, sees this as part of the reluctance on the part of academics to involve themselves in electronic publishing:

[They] often complain that the Internet is littered with ‘rubbish’: with mountains of trivia and a sea of information, within which serious scholars struggle to find anything of real value. Part of this claim is true: there is at present a great deal of shallow, inaccurate, poor-quality writing on the Internet. However, there is also some material available (sometimes only available) in electronic form equal in quality to almost anything published in the print world. The difficult step for those unfamiliar with the Internet is finding such work. I would predict that in the near future mechanisms for sorting the good from the bad will become increasingly sophisticated as academics come to accept that a shift toward electronic publishing seems inevitable. Peer review will have a vital role to play as we move into a digital scholarly future…. This does not mean that exactly the same measures for ensuring quality control as those employed with print journals will suffice (or be wanted) in all cases in electronic environments. Some kind of ‘filtering’ system will, however, be essential if the academic community is to have faith in the digital mode of scholarly publishing.

This filtering will eventually replace peer review, whether the proponents of peer review like it or not. The question is, what will that filtering look like, and who will develop it?

One person trying to encourage a system of filtering is Stefano Mazzocchi, who published the blog entry “Peer Review vs. Citation Network Topology” describing the difficulty in establishing such a system in the face of entrenched peer-review forces:

This is how academia grants credits to their research members: if you pass peer review, you earn a credit. Some journals are more renowned than others, they yield more credits and more visibility. The more you can publish, the more your peers find your stuff interesting, the more likely the university will be recognized around, the more likely students will want to study there, the more money the university makes (or, alternatively, the more money the students are willing to pay, if the number of possible students remains the same).

This is the business model of printed-press-era academia….

But what happens if a system is found that ranks scientific recognition in a better way than peer review and this system happens to be virtually free for everybody (article consumers and producers)?

What academia used to use of journals was the ability to distribute worldwide. The web is now a far better way to achieve this and academia is forcing the journals into the digital domain because hard disks are way smaller and cheaper than basements full of dead trees.

But academia finds itself locked into the symbiosis with the journals because the democracy of the web does not yield a filtering mechanism that generates that scarcity on top of which you can create a business model….

But it seems to me that academia is at a turning point: inertia is huge for such a worldwide ecosystem, but the web is radically changing how students and young scholars (like me) find and seek information, especially if they have the public web on one side and a pile of crappy custom search interfaces on top of very-expensive-for-no-reason content that your university pays for on the other.

Ben Vershbow continues, in the blog quoted from above, with a contention that blogs themselves can provide an alternative to Mazzocchi’s “inertia”:

It will be to the benefit of society if blogging can be claimed, sharpened and leveraged as a recognized scholarly practice, a way to merge the academy with the traffic of the real world. The university shouldn’t keep its talents locked up within a faltering publishing system that narrows rather than expands their scope. That’s not to say professors shouldn’t keep writing papers, books and monographs, shouldn’t continue to deepen the well of knowledge. On the contrary, blogging should be viewed only as a complement to research and teaching, not a replacement. But as such, it has the potential to breathe new life into the scholarly enterprise as a whole….

Things move quickly — too quickly — in the media-saturated society. To remain vital, the academy needs to stick its neck out into the current, with the confidence that it won’t be swept away. What’s theory, after all, without practice? It’s always been publish or perish inside the academy, but these days on the outside, it’s more about self-publish. A small but growing group of academics have grasped this and are now in the process of inventing the future of their profession.

Self-publishing by itself, however, won’t be enough, as those of us trying to teach students how to conduct research through Web possibilities are discovering. The mountain of information out there is too large, our mining tools too limited, and our knowledge of the terrain too superficial for much to be accomplished. So blogs or other means of self-publishing will not be enough, in and of themselves. Some sort of filtering, as Mazzocchi argues, must take place.

Many of the more sophisticated blogging software packages now have a rating capability, one that can elevate or lower both the main entries and their attached comments. Some even have special rating possibilities that can only be given by those who have “earned” them. Something of this nature could be applied to academic blog sites. If a site wanted, it could reserve certain (high- and low-end, generally) ranking possibilities to those who have proven to the Webmaster that they have an earned doctorate in the field under consideration. Others could rate the articles as well, but theirs wouldn’t carry the same weight.

Articles with enough positive response could be automatically added to a “recommended” list. If they reached a certain level (or duration), they could then be added as a permanent link to the site’s front page. To ensure that “spur of the moment” articles (as opposed to comments) did not appear, requirements of length and even number of citations could be instituted.

The possibilities, of course, are myriad. What is needed, unfortunately, are academics with sufficient reputation and confidence to begin and to oversee such sites, outlining requirements for posting articles and contributing their own. They also need to be in positions where they can command funds sufficient to establish the sites and ensure that they will continue for a reasonably long period of time (convincing academic institutions to maintain the sites is probably the best way of ensuring this).

One advantage of such sites is that the proponents of ideas outside of the mainstream of academic thought, such as intelligent design or holocaust denial, could present their material and see how it stands up to scrutiny within an academic community. Comments on their arguments, as well as the arguments themselves, would be rated — and even non-academics could watch the discourse, making much more informed decisions than they often can at present.

In terms of academic value, the candidate for promotion could point to the comments of peers and the ratings of articles. Contribution to ongoing debate within the field would be quite clear, as would the amount of serious activity.

Who would be willing to be early contributors to such sites? Wouldn’t most academics be too protective of their career possibilities to participate? Well, though many who have not been involved in the blogs have a conception of them as personal tales of toothpaste disasters and rants expressing unformed opinions, already a large contingent of serious academic bloggers is involved in detailed discussion with careful references. These people would certainly be willing to contribute to sites that would further expand the discussions they are already involved in online.

Will this happen? I certainly believe so. Software is now available (Scoop and SoapBlox come to mind) as platforms for such new-style journals. I just hope there are those, not only in the academic community but with the power to get their institutions involved, who will be willing to act as sponsors.

Contributors to this article: Sue in KY, JeninRI and Standingup

Discuss this article...

Editor’s note: The introduction to this Issues Series, as well as part I, "Orthodoxy Versus Bias," examine the trend in recent decades of relentless criticism of US educational institutions; the goal of many of these attacks being the transfer of control into hands of politicians.

If you like what ePMedia's been doing with research, reviews and interviews, please consider donating to help with our efforts.