How honest and unbiased are peer reviews?

Well, the short answers is 'they're not!'.

Any system in which human beings are involved is subject to personal preference, bias, dishonesty, etc. Everyone knows that systems in which humans are involved are ultimately flawed. The problem really comes down to one question - what's the alternative?

The peer review process is far from broken. The real problem is journals not the peer-review process. We have so many journals now that the quality varies considerably. Papers that were rejected in high quality journals end up in crappy journals. Papers in which an experiment was repeated are rejected for replication - this is a particular problem in the medical sciences where negative results are under-represented. And the big problem is that the public has no access to these papers unless they pay a hefty fee (something universities are dealing with by storing and publishing their own publications on-line).

The peer review process is only one side of the coin though. The other is conferences. Scientists attend conferences to get their work out there ... to start discussing ideas and results before publication, to meet colleagues and get their input, to stand in front of peers and defend their results. It's at these meetings that collaborations are formed, people start to get to know what others are doing, and that feeds into the peer process.

What's interesting is that many people on Y/A will happily discuss the 'slow down in temperatures'. Skeptics will point to the data and argue that the models are wrong because the temperatures have 'flat lined'. How can one argue that the peer-review process is flawed when evidence that could turn out to be a problem for AGW is published, available, and can be quoted by skeptics? That would tend to suggest that science and the peer process is working well enough!

I have had a similar experience to Paul and Paul's Alias 2, not with Phys Rev, but with Nature. A colleague and I tried to publish just about as controversial a paper as you can publish. Nature had no interest in publishing it, and they came up with a number of phony excuses. The first one was that the work "would only be of interest to specialists in the field". This concerned an apparent posited violation of the Second Law of Thermodynamics, so when we pointed that there might be MANY people that would be interested in that, they removed that objection, and came up with a different one--there was no experimental work to support it. Then we told them about corroborating experimental work that had been published in another journal. Then they told us that Nature only published new and previously unpublished results. Eventually the editor said something honest, like "It doesn't matter what the reviewers say, we are never publishing your paper." Once he said that, we moved on and the paper was eventually published in the journal Physics of Plasmas.

I submitted a paper last year and it was sent out to a reviewer that does not like my work, which is something of an alternative to his own work. Even though the initial review was anonymous, I guessed who it was and after the first revision he let me know who was doing the review. I answered all his objections (although I believe I went too far to satisfy him in one aspect) and the paper was published.

I think something along the lines of peer review is necessary, but there can be a tremendous amount of variation on how honest, unbiased and useful it is. Sometimes it works great, sometimes it fails. If people honestly believe in their work, though, and they are competent and persistent, I think they can virtually always get it published. It may not be in their first choice for journals, but that's just a fact of life. More people want to be published in Phys. Rev. or Nature or Science than can be.

P.S. for Paul: That's pretty funny about YA and retarded time!

There was the famous case of a couple of years ago when Richard Lindzen complained that he was blocked from publishing in PNAS whereas PNAS stated that his paper was rejected for not being good. The important information is that PNAS allows the author to choose two of the reviewers. Even Lindzen's guys rejected the paper.

The reviewers' report is here (11-page pdf)

http://www.masterresource.org/wp-content/uploads/2...

Lindzen had previously published his Iris theory but it was proved so riddled with errors that even he admitted being embarrassed by it

http://www.nytimes.com/2012/05/01/science/earth/cl...

Lindzen goes on claiming that he's blocked by bias in the system even though his own buddies find problems in his work, and even after his previous very bad paper was published.

Whatever the problems with peer-review, rejecting papers that are flawed should not be considered bias. Lots of bad papers get published to the later embarrassment of the journal, such as Lindzen&Choi by AMS or Spencer&Braswell by Remote Sensing.

The journals need the quality review to maintain their reputations. The high-quality journals are those that are most cited and provide the greatest building blocks to the advancement of human understanding. The peer-reviews for quality are an essential part of that.

Yes, that is pretty much correct. It can be good, or stink depending on circumstance. "Nature" published Mike Mann's hockey stick paper despite its flaws (1), then resisted the obvious corrections. However in 2004, "Nature" did publish the objections (2). In 2003, "E&E" published this paper: (3) when "Nature" wouldn't. "E&E" is a peer reviewed journal, but some Warmists do not consider it so, because it does not bias the review process to favor their AGW point of view (4). The IPCC makes a joke out of peer review (5). However, Warmist still think that it is the highest standard of peer review (6). The NAS often publishes garbage that is not written by experts, and not peer reviewed at all, but attempt to pass it off as though it is. Then there is that junk article about how 97% of all scientists support the Warmist view on AGW (7).

Edit @Baccheus: Blocking bad papers is not 100%. However, if Lindzen ever does publish something in PNAS or Mann ever publishes something in E&E, I suspect that those papers would make interesting reading.

Wow, you came to the right person.

Peer review is often pretty lousy. I submitted a paper to Physical Review, the most prestigious physics journal in the world, which absolutely was not a crackpot paper, and actually was very important. One referee wrote something to the effect of "This goes against what is currently thought, and therefore must be wrong" The referee REALLY said something that horrific, and was not embarrassed. Another referee said that my paper was a supposed disproof of black hole formation and thus could not be correct. In reality it was not a disproof of black hole formation and I had explained why in the paper. But if it was, then it would have been such, and unless they could find a flaw, then I would have been correct. The third referee also said inane stuff, but I don't remember what he said.

The editor handling the paper was an assistant editor, and he may not have been so bad. He had actually sent it to referee after referee even after the previous ones trashed it. I wrote to him and asked for a specific flaw, and said that "This goes against what is currently thought and therefore must be wrong" was not acceptable. He, perhaps because he did not want to defend his referees, sent it to the editor. The editor wrote back supposedly finding a simple high school algebra error on my part. At first, not having had experience with journals, assumed I really made the error I was accused of. But when I lookedf at the calculation in my paper it was obvious the editor was wrong. I did indeed "drop a term" in my calculation, but it was because it legitimately cancelled with another term.

I wrote back to the editor, and he admitted he was wrong, but then magically immediately found other supposed errors...that again were not real errors. It was clear to me that he was not rejected the paper because he was finding real errors, but rather he decided he did not like the conclusion of the paper and thus was finding (bogus) errors. I am reminded of a poster here who did not want to accept the possibility that global warming could cause human exctinction and gave an argument that the "carbon" we are emitting was once in the atmosphere in the past, blah blah blah. When it was pointed out to her that it was not in the atmosphere in the past she did *not* change her position. Her argument was construction to support a prejudiced position rather than her having formed a position on the basis of a reasoned correct argument.

The editor claimed that my paper was unacceptable because it did not address Weinberg's proof of Birkhoff's Theorem. In reality, around 1/3 of my paper was showing there was a flaw in Weinberg's proof. But even had I not addressed Weinberg's proof, I still had shown a counter-example to Birrkhoff's Theorem, and thus we would be in an interesting position even if I had not found Weinberg's specific error. This was typical of the editor--he was claiming things were not in the paper that were and things were in the paper that were not. If I remember correctly, the discussion sort of ended with him saying that Birkhoff's Theorem had been around for 93 years and thus could not be wrong. Very scientific of him. If he had not said that, other people had--the "Birkhoff's Theorem has been around for a long time" argument was given to me explicitly by a lot of people, and likiely implicitly by everyone else.

continued

Higgs is right. However, he should add that peer review is the best method we have for reviewing a paper.

That's a hard question to answer but if you have some free time I suggest perusing this site to get a flavor of what's going on out there in science land: http://retractionwatch.wordpress.com/

They also have some specific categories like:

Faked Data: http://retractionwatch.wordpress.com/category/by-r...

Image Manipulation: http://retractionwatch.wordpress.com/category/by-r...

The story on the Lewandowsky "denier conspiracy" paper is a real pager-turner: http://retractionwatch.wordpress.com/2013/03/28/wh...

I'm Paul's Alias 2. My answer was too long for yahoo to post it, so I am signing in on another account to continue it.

continued from previous

Another paper I submitted used in the proof of something that the (electromagnetic four-vector) fields of a particle in uniform motion move along with the particle without a time lag. If I use the physics term for "time lag" this post might get deleted because it also is a politically incorrect term for a person of low intelligence. Anyway, the referee pointed out the Lenard Weichert (LW) formula, and noted the time delay. But in reality, for the specific case of a particle with *uniform* motion, terms cancel, and the LW formula turns out to not imply a time delay. And it is obvious this must happen for a particle in uniform motion, or else Special Relativity would be contradicted.

You are supposed to learn in undergrad physics that the fields move with the particle without time lag if the motion of the particle are uniform. So I scanned a page from a common undergrad textbook and sent it to thee editor. He, who apparently did not know undergrad physics directly, was not able to learn it from what I sent, and sided with his referee.

am not telling you red herrings. This is really how things are. For example, Gell-Mann did not send hias paper theorizing quarks to Physical Review, because he knew they would reject it. And the journal he did send it to made him remove references to quarks as actual physical particles. He could only claim them as unphysical mathematical abstractions. So much for the merits of peer review!

One of the Non-Deniers here, a guy who usually is not bad, told a Denier yesterday that if it really could be shown that global warming was bogus all the researcher would have to do is send the paper to Nature or Science. Global warming is not incorrect, but if it was, there would just be no way Nature or Science would be interested. Many of the Non-Deniers here do not understand how the scientific community works, and they have an unnatural obedience to authority.

I would add the supplemental question of how thorough is it? If I read something and it sounds reasonable I often do not take the time to disseminate it. I have far too much on my plate to take that much time with each issue I come across. I know for a fact that others treat many things in the same manner. The human mind is geared this way.

<<How honest and unbiased are peer reviews?>>

As the objectives of the peer review is to ensure 'standards of quality, improve performance, and provide credibility', they are by definition honest and unbiased.

That said, it all depends on the quality of the journal responsible for the peer-review process.

Lately, with many denier 'scientists' getting their papers rejected by the mainstream journals (in other words, not having passed the peer-review process), alternative 'journals' have popped up (ie Energy & Environment whose editor admitted she was following her political agenda with regard to published papers) or papers are submitted to journals typically not specialized in climate science (and thus the quality of their peer-review process is challenged).

<<Can you even expect unbiased and honest peer reviews if your subject is controversial.>>

With a 97% consensus I would hardly call it controversial.

<<Political Economist Robert Higgs..>>

Robert Higgs, adjunct scholar at Cato and Senior Fellow at The Indepedent Institute, both fossil fuel funded science deniers spreading propaganda.

BTW, has Watts managed to get his latest paper peer-reviewed yet?