What constitutes "peer review"?

The peer review process varies from one journal to another, but publishers establish a set of common policy standards. For example, one of the journals with which I have direct experience (JRS) is subject to the standards given below.

http://wipjp.blackwellpublishing.com/bw/publicatio...

Journals with high standards of peer review attract manuscripts from the best authors and are widely read and cited within the research community. Typically, manuscripts that appear in these journals are reviewed first by the editor, who may reject a manuscript as unsuitable for the journal by reason of content or quality. The editor then asks three scientists with significant research experience in similar subject matter and no conflicts of interest (see Wiley guidelines) to review the manuscript. The reviewers change from paper to paper. There is not one formal group of reviewers that looks at all the manuscripts. The reviewers rate the content of the manuscript and comment on questions such as: 1. Were the experiments performed properly? I once rejected a paper on the grounds that the spectra shown were from O-ring grease rather than the sample claimed in the manuscript. The author was new to the field and lacked the experience to recognize an error. 2. Are the experimental results reproducible and reasonable? Reviewers may repeat some of the experiments and or compare the results with prior research. I have repeated experiments to ensure that the result is broadly correct. If the results are not consistent with prior research or the reviewer's experiment, the reviewer asks the author for an explanation. 3. Is the work original? I get really snarly with authors that either plagiarize others work or try to publish the same research in multiple journals to increase their publication count. 4. Is the manuscript of interest and important to the field? This is an opinion about whether the manuscript will be cited and promote further research in the field.

While deniers may set their own standards of peer review, the process should be transparent. In the case of Watts, the manuscript must be consistent with Watts point of view and there is no requirement that the manuscript has a factual basis. Both Nature and Watts have the esteem they deserve, based on the rigor of their respective review processes. Personally, I would advise Watts to publish his works on rolls of soft tissue paper so that they can serve a useful function.

Edit: I see that Trevor has the same utilitarian use for Watts blogs.

I cannot speak from experience since I'm still only a high schooler, but I can speak from what I've learned from a bit of research:

In the general form, David is right in that peer-review can take many forms. In many fields it is the submission of a paper by an editor to usually two or three other individuals who are experts that understand the work. Most often, actually, these 'referees' are independent from each other and do not communicate their thoughts with each other.

However, this is different for several other journals that have more stringent requirements. For "Nature" and "Science," which are very broad in their publications' focus, a two-step review process is used. Here is where an editorial board is used - if the board does not find that the work is a breakthrough in the field (enough to publish in the journal, anyways), it will be rejected. If the paper passes, then it is sent out to referees for scrutiny.

In this sense, the editorial board is peer-review and yet is not at the same time. The board does not check for errors as the collection of referees would, but instead focuses more on the implications of the work. These journals receive a wide range of (and many) submissions and must filter the best out. The review of the work is still largely up to independent referees.

As to whether or not Einstein's papers were peer-reviewed, I'm not sure if I can comment on that. If Einstein's papers were read and revised by Planck and Wein, then they were peer-reviewed. However, it's not necessarily that simple. It would appear that Annalen der Physik very strictly peer-reviews eery paper before it is considered for publication, and a mere review by two people does not seem to fit into this standard. It would help if I had access to the Wiley InterScience website so I can do a bit more research on the journal's peer-review process, but it's inconveniently down right now.

If the standards for peer review were less 105 years ago, and if Planck and Wien actually made suggestions for revisions and did not just read the papers, then it can be said the papers were peer reviewed. I am not sure though what constituted proper peer review in 1905 for Annalen. I need more info.

In the form with which I'm familiar, an author(s) of a paper for submission to a journal submit the paper to the journal. The editor locates other workers who do research in a similar area and asks them to read the paper and evaluate its suitability for publication. Quite often, these "referees" find flaws in the data and/or presentation and the paper is returned to the author for revision, or the addition of new data. Sometimes the paper is rejected and sometimes it is accepted without further revision.

Lik all processes, this can be abused. To follow up on the global warming denialist issue, there are "journals" set up by denialists, and the papers are "peer-reviewed" by "experts," some of whom have little or no science background (this is by no means limited to global warming denialists, by the way). In other cases (again not only on global warming) a prestigious enough author can pull strings to have the paper reviewed by "friendly" referees. Many practicing scientists see through these ploys, but those not "skilled in the art" are unfortunately duped all too often. For that reason, I pay no attention to articles claiming the unreality of global warming unless they appear in the very highest caliber journals (Science, Nature, Proceedings of the National Academy of Sciences, etc.), since they have the highest standards, and convincing evidence would be of such momentous importance (given the more than a century of work) that no self-respecting author who was convinced of the accuracy of such work would submit it anywhere else.

What constitutes peer review has changed over time.

Now, for most journals I have any experience with, a paper is reviewed by an editorial board and then sent to anonymous reviewers who are considered experts in the field.

A century ago science was much different, it was smaller and less formalized. I would be surprised if that journal even had a proper editorial board at that time. It was still reviewed by two experts prior to publication, so it was peer reviewed.

Alias's complaint is ridiculous. Scientists were less specialized then. Given how revolutionary Einstein's work is, I doubt there were many people who would have met his definition of experts in that field.

Obviously some editorial on the internet went through no review whatsoever. Of course peer review isn't necessarily a marker of quality these days. Science denialists of all stripes have set up their own journals to publish their crap.

I'm very surprised they aren't using Watson and Crick's paper as an example. It wasn't peer reviewed. Nature had the full formal process at that time, but the editor just ignored it. It was too obvious they were right.

The better journals (i.e. ones I don't edit) will send an article to a third party to determine whether to reject an article, accept it as written, or ask for some revisions. Not every submission to a peer-reviewed journal gets peer review. Sometimes the editor can tell right off that the article is not slanted toward that journal or is too long or too short. If the article is otherwise acceptable, the editor will ask an expert in the field for comment. The reviewer may or may not be biased. For example, if the article says that oikos' article is full of beans, oikos may be asked to review it. Or it might go to someone else who knows the field. The writer usually does not know who wrote the review but the reviewer knows who the writer is.

Sending out the article for comments before submission is a good idea (and is almost always acknowledged in the paper) but it is not what is meant by peer review in scientific publication. There need not be a formal editorial board, though.

Science evolves, and peer review has evolved with it. In 1899, when the American Physical Society was founded, there were just 36 members--now there are about 48,000. The most knowedgeable physicists, such as Planck and Einstein, could be aware of all the major research going on. No doubt many of the journal editors had this kind of breadth of knowledge also. This is simply not the case today. Along the way, peer review has evolved also. As part of my current research, I have papers from the Monthly Weather Review from 1906 and 1922 (as well as many more recent papers). It's doubtful that any peer review was in place in 1906, other than the review by the journal editor. The 1922 paper includes reviews by three peers, along with the paper. This is actually very useful to be able to see the reviewers' comments and it's too bad that we don't do that today.

These days peer review is a useful tool for winnowing out research that obviously flawed, but it certainly does not mean that what has been published in the peer reviewed literature should be accepted unquestioningly. On the other hand one should be suspicious of research that intentionally avoids peer review and publishes instead on advocacy websites or publications. There are no Einsteins publishing on "wattsupwiththat." I would make the same statement for Greenpeace's website--real science is not published on advocacy websites.

EDIT for Andy: Clearly you don't have the faintest understanding of peer review. If you don't know what you are talking about you should not answer questions.

In my experience, the editor-in-chief selects the most knowledgeable of his editors to oversee the review of the paper. The selected editor selects 3 people, based on his/her own general knowledge of the field, to review a submitted paper. The editor may have a list of potential reviewers, make select a person in the paper's list of references, may have personal knowledge of a colleague or may have asked the author to suggest reviewers. (On one occasion I shot down a paper whose author had suggested me. He had fanciful ideas that contracted basic physical laws.)

The 3 reviewers (if they agree they have the knowledge on the topic and the time to do the review work) read the paper, and its references, and look at other material they are familiar with and look for errors, omissions, misunderstanding of theory, etc. They return to their comments, hopefully after due diligence. If all recommend, usually subject to minor corrections, the editor asks the author to make the minor corrections and accepts the paper.

If none of the reviewers recommend the paper, or if the reviewers disagree, the editor, after reading their reasons, rejects the paper or asks the author for major revisions.

The author may made further explanations or revisions to get a rejected paper published.

The system does not produce error-free papers, but they are greatly improved. After publication, a paper published in a scientific journal is part of the scientific record. Experts in the field can rebut the paper with new lab experiments, point out contradictory evidence, correct arithmetic mistakes, etc.

The peer-review process filters most, not all of the utter dreck. Researchers can be reasonably confident citing the results.

I think of an "editorial board" as the editors available to supervise the reviews of a paper. Only 1 editor is used per paper.

[I don't know what German practice 100 years ago was like, or how it is an example of anything. Silly argument. I can't think of anybody in German physics in 1905 more able to review Einstein's papers than Planck and Wien.]

As a history guy who once had a paper peer reviewed I can tell you that the process is both formal, very formal, informal and every other kind process the human mind can think of. But in the end the folks that actually do the review tend to be relentless bastards that would like nothing better than to shoot your research down in flames, have you flogged and drummed out of the 'service' with a yellow stripe down the center of your back. No mercy is shown. Nit picking is the name of the game and God help you if your thesis is provably wrong....'yer dead meat! Believe me, anyone who intends to submit a paper for 'peer review' sweats bullets over it. If you read it in a 'peer reviewed' publication you can bet that it's right on the money. If it's not the folks that ultimately read the submission will be more than glad to point out the errors while firing flaming arrows into the backs of both the guy that submitted the paper and the guys that reviewed it and allowed it see the light of day. If a publication allows too many of these poorly reviewed articles to appear that publication is going to catch hell. As most, if not all the scientific publications have been around for a long time it stands to reason that they don't allow many mistakes to slip by. The peer review business has been around long enough to win a whale of a lot of respect...is it perfect? No, sometimes later science will blow a widely held scientific belief out of the water...it happens, but it doesn't happen often and when it does it happens because of better science...not for some political reasons as the climate deniers are attempting to do.

The better journals (i.e. ones I don't edit) will send an article to a third party to determine whether to reject an article, accept it as written, or ask for some revisions. Not every submission to a peer-reviewed journal gets peer review. Sometimes the editor can tell right off that the article is not slanted toward that journal or is too long or too short. If the article is otherwise acceptable, the editor will ask an expert in the field for comment. The reviewer may or may not be biased. For example, if the article says that oikos' article is full of beans, oikos may be asked to review it. Or it might go to someone else who knows the field. The writer usually does not know who wrote the review but the reviewer knows who the writer is.

Sending out the article for comments before submission is a good idea (and is almost always acknowledged in the paper) but it is not what is meant by peer review in scientific publication. There need not be a formal editorial board, though.

Source(s):

I have reviewed articles and had mine reviewed. I have also asked colleagues to review articles submitted to me, on an informal basis (informal review, not informal submission).

“What constitutes "peer review"? I say it consists of having material submitted read by a competent person or persons before the editor decides whether to publish.”

In respect of published works then I would certainly go along with that description. Often it is the editor (or an editor) who will circulate the material to relevant experts for critical evaluation. More often than not the editor isn’t an expert on the specific subject in question but will have a broad understanding of the wider subject.

As a simple example… an editor of a medical journal will have a good understanding of medicine as a whole, but if s/he receives a paper relating to a proposed link between the toxoplasma gondii protozoa and schizophrenia then it’s unlikely that s/he will be sufficiently competent in this area to adequately critique the paper. As such, the editor would request that suitable experts review and return the material.

Upon completion of the review the work will be returned to the editor with comments added. These comments could relate to anything, perhaps to the methodologies or processes or the way that conclusions have been drawn. The reviewer will, where appropriate, highlight errors and uncertainties and suggest improvements.

The standard procedure is for each reviewer to work independently, unaware of the contributions made from other reviewers. Perhaps more so in scientific journals, the reviewers will submit their recommendations to the editor, they may recommend that the work be published as is, be published but subject to certain criteria or be rejected.

Comments are fed back to the authors and if needs be, the work will be revised in accordance with feedback from the reviewers. After revision the material may be resubmitted for consideration.

Ultimately, as in most publications, it is the editor who makes the final decision. Technically they could publish regardless of the outcome of the review process but this defeats the whole purpose of the exercise. The norm is to take on board the comments of the reviewers and make a decision whether to publish, and what to publish, based on the feedback received.

In some situations an editor may feel it is necessary to obtain further reviews before reaching a decision on publication. This is likely to happen if the original reviewers return conflicting opinions.

Contrary to some opinions, it is not a condition of the review process that a consensus be reached prior to publication; in reality, this isn’t even the objective.

A few years ago Nature conducted an in-depth debate about the peer review process, this looked at the many aspects involved and raised a lot of interesting points.

http://www.nature.com/nature/peerreview/debate/ind...

Further points…

Never having been an editor for anything other than a school magazine and a charity newsletter means that I am not particularly au-fait with the workings of the editorial process. I have had papers reviewed, some have been published and some haven’t, and I’ve reviewed the work of others.

As regards Einstein, Wien, Planck etc, although the papers may not have been formally reviewed in the manner that a submission to Science or the Journal of Applied Meteorology and Climatology (JAM) would be reviewed, the material has still been subjected to the same critical analysis and has undergone the peer review process.

I would have thought that being critiqued by Wien and Planck, would represent the ultimate in peer reviewing. I wonder how much of Watt’s or Goddard’s posts would have passed muster with the likes of these people, I suspect much of it would have ended up being used as toilet paper.