Why scientific peer review matters

Posted on April 5, 2009


“Peer review isn’t all it’s cracked up to be,” and that phrase’s many variants, is something you year a lot these days. It’s a complaint that is heard most often with reference to climate science, and most often by people who understand neither the science behind climate disruption nor the purpose of peer review. So, as someone who’s undergone peer review repeatedly in both academia and as an electrical engineer, I’d like to explain why peer review matters.

First, a little explanation of what peer review is.

Peer review is a formal process instituted by an organization to ensure that a quality product is produced. In the case of scientific peer review, the product is scientific papers that meet a presumably high standard of scientific accuracy and professionalism. Scientific peer review involves distributing review copies of submitted papers to experts in the field in advance of publication. These usually anonymous experts read and offer comments that suggest improvements in the style, data, or thought processes described in the paper. The comments are then given to the paper’s authors who incorporate the suggestion, resubmit the paper to the journal for another round of review and, if there are no additional necessary improvements, the paper is then published.

In my own profession of electrical engineering, the peer reviews tend to be more in-depth than what I just described. Here’s the usual list of reviews. I’ll pick up after the specification phase, when engineers get handed a design and told “design this.” Any one of the reviews described below may be done repeatedly if the first review finds a significant number of problems.

  1. Project internal design review: When the design is largely complete, the entire design team sits down and tries to ensure that nothing major was missed. Problems are identified and corrected before the next review
  2. Independent design review: other engineers in the company, often senior engineers with decades of experience, are brought in to review the design with the engineer in charge and other members of the team. Problems are identified and corrected before the next review.
  3. Customer design review (optional): If you’re doing contract engineering work, then the customer often wants to be involved in the design review process. This review may include any outside experts that the customer hires to review the work as well. This review always happens after the independent review.
  4. Design document review: In the process of writing up the documentation in enough detail that another engineer could replicate the design if the engineer in charge were struck by lightning, small errors can be discovered. These errors would be corrected and the documentation review (usually by independent reviewers again) often turns up further small errors missed in earlier reviews.
  5. Preliminary layout review: Just because you can design an electronic circuit doesn’t mean it can be built. Design problems that cause manufacturing difficulties are first identified here.
  6. Final layout review: Due to (usually) minor changes that are made during layout, the schematic is reviewed yet again at the same time that the layout of the printed circuit board is.

One major misconception about all varieties of peer review is that the reviews guarantee no errors in the final product. Not even five or more engineering design reviews described above can ensure that there aren’t any errors in the design. In fact, one of the differences between a designer with experience and one fresh out of college is that the experienced designer plans for the inevitable errors that are discovered in testing, while the inexperienced designer thinks his or her design won’t have errors because it was reviewed.

While catching errors is an important purpose of peer review, what kind of errors are caught and corrected depends greatly on who the expert reviewers are. If a climatology paper that relies on detailed knowledge of statistics has an error in the statistics, sending the paper to only expert climatology reviewers might not turn up the statistical error. Similarly, if there’s a power problem in an electronics design, it might only be detected if a power supply engineer was one of the reviewers.

Ultimately, though, this isn’t a problem – not with peer review in general and not with scientific publications in particular. Just as there’s a check on an engineering design review called “product testing,” the publication of a peer reviewed paper provides the final check as well. Once a scientific paper is published, it’s reasonably likely that someone reading the paper will discover any errors. Discovered errors are corrected either by a retraction if the error is significant, or by the submission of another another paper describing the errors that also goes through the peer review and publication cycle.

So why does scientific peer review matter? Because the peer review process described above is a key component in a process that continually improves the overall quality of scientific research. And because participating in that process is the price of entry to be taken seriously as a scientist.

It takes confidence in your skills and knowledge in order to put yourself through the wringer that is publication. After all, your discoveries and data are then publicly available for all to critique, agree with, or even mock. The process alone provides a level of confidence in the scientific accuracy and veracity of the papers that are published.

After all, anyone can publish a blog filled with with so many numbers that it looks legitimate, but only a scientist would subject him or herself to peer review.

As a final aside, there is no such thing as a “peer reviewed book.” Books are published not based on the quality of the science as determined by anonymous expert reviewers, but rather on a publisher’s and editor’s determination of how well the book will sell. If the science happens to be interesting and accurate, great, but at best that’s a secondary consideration over profit. Scientific journals publish papers based on the quality of the science first and foremost, and then charge whatever the market will bear. Given that a single journal can cost over $1,000 per year for a subscription, the market is willing to bear a great deal.

Posted in: Uncategorized