Pharyngula Wiki

Simple stuff, in theory.

Peer review is the process of subjecting scientific work to review by other experts in the field. At Pharyngula, it is recommended that you cite the peer reviewed literature or STFU.

The term "peer review" is typically used for scientific and academic publications. When an article is submitted, it is sent to the authors' "peers" - i.e., other experts in the same field - to assess the quality of the work. A similar approach is also generally taken to evaluate research proposals submitted to Science Foundations, such as the American NSF, where the proposals are sent out to qualified scientists to assess whether the proposed projects merit funding.

Initial and ongoing peer review[]

The first, and most notable, step is when a piece of scientific work is looked at by a reviewer (sometimes referred to as a "referee") for approval prior to publication. This is generally considered necessary to quality work, but is not sufficient. It merely minimises the chances that pseudoscience does manages to masquerade as science, and that low quality work which would not contribute anything of value gets filtered out.

Following publication, peer review is an on-going process where a work is open to scrutiny by the scientific community at large - this is what publication allows a scientist to do. The process is designed to ensure that the work meets the standards of the field in question and of science in general. A paper which survives initial peer review may be shown to be rubbish when closely examined after publication.

Relevance to the scientific community[]

As practiced in modern universities, peer review is a key part of the scientific method, where the goal of the system is to ensure that work is stripped of biases, assumptions and other errors, through the scrupulous review by one's equals in the field, acting as learned professionals and mediators. Accordingly, peer in this context implies equals: i.e., the reviewers or judges should have the same or reasonably similar qualifications that the author of the work has - or claims to have. No ideology, other than the commitment to "rigorous empiricism... without which no man is a scientist," matters in asking whether a person is a scientific "peer".[1]:42

Thomas Kuhn, in the process of seeking to define the nature of science, and the nature of "scientific progress", writes in his work, The Structure of Scientific Revolutions, that the unique composition of the scientific community, as composed of disinterested (read: unbiased) intellectual equals, is alone capable of generating scientific "progress."[1]:168 Therefore, Kuhn sees peer review as embodying "one of the strongest, if still unwritten, rules of scientific life... the prohibition of appeals to heads of state or to the populace at large in matters scientific."[1]

More prestigious journals have a more thorough peer-review process and have tighter criteria for the level of work they can publish - indeed this is why they are prestigious, as only the best and most groundbreaking work is published in these journals. This isn't to belittle "lesser" journals at all; if every journal had the publishing standards of JACS or The Lancet, science would get nowhere. The two top journals in the world for science are Science and Nature and getting a paper into them is considered a great achievement.[2] The vetting process for these two journals is intense, with only 10% of submitted manuscripts going forward for publication - that's 10% of the manuscripts scientists think are good enough for Science or Nature. These top publications also tend to "coach" their submissions, improving the readability, conciseness, and clarity of the paper (again, this is why they're considered the best).[3]

Possible problems[]

Like all human activities, peer review can be subject to biases in certain situations or if insufficient care is taken in the selection of reviewers:

  1. In very competitive areas of sciences, it is possible that reviewers are tempted to gain an unfair advantage when they receive a rival's work for review long before publication, and may even try to scoop the results.
  2. In the case of competing schools of thought, it cannot be ruled out that some reviewers might be influenced by their adherence to a certain point of view in giving negative reviews to colleagues from an opposing school of thought, although most intelligent editors would be able to allow for this issue.
  3. If the field of experts for a given topic is small, there is a certain likelihood that the reviewer may have a relationship of animosity, rivalry, or perhaps friendship with the author, and this could lead to a biased review process.
  4. The peer-review process doesn't involve replicating experiments or studies in order to test their truth value. The reviews of submitted papers are only to detect glaring errors in methodology and to determine if the work, as it is presented, is suitable for publication. Therefore the process won't detect outright fraud immediately (unless it's blatantly obvious).
  5. In journals lacking real oversight from publishers, editorial board members may be able to publish their own articles after hand-picking their own peer reviewers or bypassing peer review. There was a small scandal in the mathematics community when Chaos, Solitons and Fractals was alleged to have carried hundreds of articles by its editor-in-chief, Mohamed El Naschie.[4][5][6][7]

In all these cases, the responsibility to choose unbiased reviewers and to recognize a biased review rests on the shoulders of the editors of the scientific journal a work has been submitted to. Some journals allow the authors to suggest that certain colleagues not be used as reviewers. In the case of fraud prevention, the peer-review process does not end with publication, as the article or paper remains available for all interested parties to view - if someone tries and fails to replicate the results, or finds that it contradicts research that they have done, then criticisms can be published and investigations made. Even in the face of these possible problems, the peer review process remains the most objective and qualified way to assess scientific work that has ever been developed.

Peer review as a legal benchmark[]

Without doubt, the twentieth century has seen an uptick in the number of cases where specialized scientific knowledge is required to resolve a dispute: as a basic example, the plaintiff in a medical malpractice case must prove that a drug would have saved his father's life, before he can prove the hospital's negligence in not administering the drug. Necessarily, if a party must prove science to win his case, the court must have a way of judging the science.

Courts have met this challenge by adopting Federal Rule of Evidence 702, and the Daubert test, which require that "expert testimony" - a category which compasses scientific testimony - be methodologically sound.[8] In law student's shorthand, the Court in a Daubert inquiry asks, before considering the scientific testimony, if the scientific theory/method relied upon by a party (1) has been peer reviewed, (2) has a low error rate, (3) is testable, and (4) is generally accepted by the scientific community. By adopting peer review as a benchmark for legal acceptance of scientific knowledge, the Daubert court validated the idea that science must be of sound, independently reviewed, unbiased methodology before it can be real science, and also agreed that the only cure for bad science is vigorous "cross examination."[9]

The takeaway lesson is that temporal authorities, beyond the scientific community, recognize the necessity of unbiased peer review, before science can be science.

What peer review is not[]

Many people think that the process of peer review is meant to settle the actual validity of the work, and that in any paper that has passed peer review, the science is entirely correct. This is not the case. Peer review is an "entry level" sort of test that weeds out the pseudoscience and obviously bad work, but is not intended to be a catch-all. Passing peer review and publication is indicative that (by the standards of the journal in question) the science is thorough, there are no glaring omissions, and the interpretation of the results presented are at least plausible, but this does not cement the science. Further publications and research can then use the data contained in the paper and its conclusions can be amended in later publications.

To make a legal analogy, if it is erroneously assumed that the peer review process is like a trial (the case either proven true or dismissed), the actual process is more like an arraignment, only verifying that the case has enough merit to be heard. Indeed the "trial" part of a scientific work is a very much on-going and continuous process that happens as other scientists cite the paper, or attempt to replicate or use it in their own work.

It is also worth noting whom those "peers" may be, as practitioners of pseudoscience might form a circle of pseudoscientists who start a pseudoscientific journal. It isn't the support of a claim that makes it true, it's the honest attempts to disprove a claim through experimentation that solidifies it.


  1. 1.0 1.1 1.2 Thomas S. Kuhn, The Structure of Scientific Revolutions (3d ed. 1996).
  2. An achievement worthy of 3 cases of champagne, a slap-up steak dinner at a fancy eatery and an all night drinking session at the pub later. All on departmental expenses, actually.
  3. PhD Comics - Tales from the Road, Science vs Nature
  4. Questionable Science Behind Academic Rankings
  5. Nature journal libel case continues
  6. Integrity Under Attack: The State of Scholarly Publishing
  7. El Naschie questions journalist in Nature libel trial
  8. See Daubert v. Merrell Dow Pharm., Inc., 509 U.S. 579, 590 (1993) (noting that the guarantee of “scientific knowledge” is one of “more than subjective belief” based in “the methods and procedures of science."
  9. See id. at 597 n.13 (“This is not to say that judicial interpretation, as opposed to adjudicative fact-finding, does not share basic characteristics of the scientific endeavor.”).