Peer-reviewed...

[vortmax]

"Peer review: a process by which something proposed (as for research or publication) is evaluated by a group of experts in the appropriate field ."

I've been reading a lot of posts in this section regarding AGW and noticed that the term 'peer-reviewed' is mentioned very frequently by AWG advocates. Now I understand what the term 'peer-reviewed' means (and I agree that it's fundamentally a good thing), but my question is this:

Who are the 'peers' and what are their motivations?

When it comes to the AGW debate and the countless peer-reviewed 'facts' that are conveyed by AWG advocates, there's an important fact that is conveniently left out....money. There's A LOT of money to be made by advocates of AGW....A LOT. Did I say a lot? This coupled with another important fact that large amounts of money can corrupt even the most esteemed people leaves a question in my mind that the ‘peers’ who are doing the ‘reviewing’ my not be so neutral. I know that scientific data doesn't lie, but the people behind the data certainly can.

For what’s it’s worth, non-AGW advocates do not stand to gain the comparable large sums of money (grants, job security, etc.) by their position as do AGW advocates...

Discuss.

[hoosierwx]

I agree money could be a factor in some cases. What might be of greater influence is importance and prestige. Climate science was a back burner field until AGW. I would have to think that all things being equal even the best minded scientist might be swayed if even unconsciously to make their field more important. I am not saying that this has absolutely occurred, but something that has to be factored into the equation.

[BethesdaWX]

This is one of the reason why peer-reviewed studies, on both sides of the debate, cannot be used as a source of ammo to shoot down other papers.

Peer reviewed studies are just as error prone as non-peer reviewed studies, as has been demonstrated.....

[skierinvermont]

Anybody familiar with peer-reviewed papers (from any field) will understand how it is not even comparable to non-peer reviewed works.

[BethesdaWX]

Anybody familiar with peer-reviewed papers (from any field) will understand how it is not even comparable to non-peer reviewed works.

much different in the processing..... still a moot point though.

[stormguy80]

I could not disagree with you more on that last point. Many of those arguing against AGW have plenty to gain. How could you not know this? The oil companies, and any other big business whos activities are in part responsible for increased Co2 stand to loose out if we decide to take aggressive action against AGW. That’s why many of them hire their own experts (whom I’m sure they can afford to pay much, much more than university professors on the AGW side of the debate) to refute the evidence of AGW. Anyway, truth be told I’m in the middle ground on this issue as I think we have had some part of the warming that has occurred but that other factors are likely involved as well. I just think you can’t use the “damming the origin" argument against one side but not the other. It works both ways.

"Peer review: a process by which something proposed (as for research or publication) is evaluated by a group of experts in the appropriate field ."

I've been reading a lot of posts in this section regarding AGW and noticed that the term 'peer-reviewed' is mentioned very frequently by AWG advocates. Now I understand what the term 'peer-reviewed' means (and I agree that it's fundamentally a good thing), but my question is this:

Who are the 'peers' and what are their motivations?

When it comes to the AGW debate and the countless peer-reviewed 'facts' that are conveyed by AWG advocates, there's an important fact that is conveniently left out....money. There's A LOT of money to be made by advocates of AGW....A LOT. Did I say a lot? This coupled with another important fact that large amounts of money can corrupt even the most esteemed people leaves a question in my mind that the ‘peers’ who are doing the ‘reviewing’ my not be so neutral. I know that scientific data doesn't lie, but the people behind the data certainly can.

For what’s it’s worth, non-AGW advocates do not stand to gain the comparable large sums of money (grants, job security, etc.) by their position as do AGW advocates...

Discuss.

[bobbutts]

I could not disagree with you more on that last point.

Yep, I was going to say.. This same argument is constantly used by the pro-agw crowd.

Big oil and other interests push skepticism just like they did for tobacco, etc etc etc..

I think it's safe to say that there is outside non-scientific influence going both ways.

[AtticaFanatica]

"Peer review: a process by which something proposed (as for research or publication) is evaluated by a group of experts in the appropriate field ."

I've been reading a lot of posts in this section regarding AGW and noticed that the term 'peer-reviewed' is mentioned very frequently by AWG advocates. Now I understand what the term 'peer-reviewed' means (and I agree that it's fundamentally a good thing), but my question is this:

Who are the 'peers' and what are their motivations?

When it comes to the AGW debate and the countless peer-reviewed 'facts' that are conveyed by AWG advocates, there's an important fact that is conveniently left out....money. There's A LOT of money to be made by advocates of AGW....A LOT. Did I say a lot? This coupled with another important fact that large amounts of money can corrupt even the most esteemed people leaves a question in my mind that the ‘peers’ who are doing the ‘reviewing’ my not be so neutral. I know that scientific data doesn't lie, but the people behind the data certainly can.

For what’s it’s worth, non-AGW advocates do not stand to gain the comparable large sums of money (grants, job security, etc.) by their position as do AGW advocates...

Discuss.

Good questions. First off, I have published several peer-reviewed articles as both a lead author and as a co-author, and I have also served as a reviewer, all for AMS journals. As a result, I feel like I can contribute and provide some perspective even though I have no desire to engage in climate change discussion specifically.

The AMS reviewers are typically chosen by one of the journal's editors. The journals have multiple editors, all of which have significant experience in research and peer-review, and are usually from labs or universities. For example, even though I have a bunch of publications, it'd be a very long time before I could even consider being an editor.

When the journal receives the article, the editor who receives the article is tasked with choosing individuals proficient in the subject area to review the article. Every paper I've been involved with except one has had three reviewers. The exception was when I was one of two reviewers for a BAMS article, so I think it's typically three reviewers and occasionally two. The reviewers are anonymous unless they don't want to be. A colleague of mine received a paper review with the reviewer's name signed (as you can imagine, it was largely a positive review). The reviewers do not know who the other reviewers are either, though they do get to read the other reviews as they are sent out by the editor. Reviewers are almost always published scientists, though I know of one case in which a reviewer was a first year grad student not yet published but was well-versed in a highly specialized area.

In my experience, those chosen to review an article are often a. those that have published studies in the specialized discipline that the article in question deals with and/or b. those that are prominently cited in the article in question. In my opinion, this is entirely appropriate. I have also found that editors often don't know who to ask to review a paper, probably because they don't know enough about the subject area in question. When that happens, the editor's experience in the field becomes beneficial as they often ask colleagues they work with or worked with in the past for suggestions. In fact, I ended up reviewing a paper because somebody suggested me to the editor, who I did not know at all.

It depends on the specific journal but a reviewer's options on the paper include: "Accept as is" meaning no changes, the paper is ready for publication in the form in which it was received (never happens), "Accept with minor revisions", the paper needs changes but just minor ones and then can be published, "Accept with major revisions"/"Reconsider after major revisions", the overall paper thesis is probably worthy of publication, but major content edits and/or concerns need to be made/addressed, and finally "Reject", the paper is not worthy of publication. In the end the reviewer is merely making recommendations to the editor, the reviewer is not the ultimate arbiter of manuscript publication. I believe a recent analysis of AMS journals found that the proportions are roughly 1/3 each for accept with minor revisions, reconsider after major revisions, and reject. The reviewer will usually see a second version of the article after revisions have been made, unless they request not to (sometimes, the reviewer will suggest "Accept with minor revisions" to the editor, provide the edits, and tell the editor that they don't need to see the article again).

At the risk of sounding naive, my motivation when reviewing papers is to determine whether the paper contains content that advances the science in a meaningful way, and if so, improve the quality of the science being published in the process. I consider it an honor to be asked, not to the article's authors but to the editor and to the community at large. I don't owe the authors anything (except my best effort) and they'll never know who I was (unless I was sloppy). I do, however, owe it to the editor, who should have no vested interest in the publication of the article, to do my best. I can tell you that if I did not put in an honest effort on a review, the editor would know, and it would probably be bad for my career, I consider that a good financial motivation. I do get to put on my resume that I reviewed articles for this journal or that journal, but that is not contingent on me approving or rejecting the article. In fact, in my experience, if colleagues take any type of "pride" in their reviews, it's in rejecting poor articles, not in approving them.

Science is competitive, and like with most industries, competition is a good thing, provided there are basic mechanisms in place to prevent gross and widespread corruption, in this case scientific fraud. It seems unlikely to me that fraud is a major problem with reviewers. For one, I'm pretty sure that several times, papers I have authored or co-authored have been reviewed by scientists who study pretty much exactly what I study, and therefore compete with me for grant money. As a result, they hit me hard, and that's good, bring it on, defend your work or get out. Second, corruption through review is a terribly inefficient process because there are three reviewers. As a result, a corrupt review is likely to stick out and will probably be given less weight. I'm pretty sure I've experienced this first-hand. I don't know for sure, of course, but I'm pretty sure I know who a reviewer was for one of my manuscripts and why they suggested "Reject" to the editor (not science-related). The other two reviewers had no problem with the manuscript and the editor decided it was worthy of publication.

That brings me to the editor. If there is a potential deficit in the system, it's definitely not in the reviewers, but in the editor. As I mentioned above, the editor decides if a manuscript gets published. The editor also chooses the reviewers. As a result, an editor that wanted to railroad an article for whatever reason could choose (or conspire with) reviewers sympathetic to the editor's point of view or ignore the reviewers' suggestions and reject/publish the article against the recommendations of the reviewers. The second option is mostly a non-starter, since this type of behavior from the editor would eventually be reported to other editors or the chief editor from the reviewers. The first option is a legitimate concern, but for it to affect the whole of scientific opinion, it would have to extend to multiple editors, and many many reviewers. That's getting to the point of widespread, conspiratorial corruption. Is such a scenario conceivable? Yes. Likely? I don't think so. The system isn't perfect, but my experience is that it works pretty damn well.

A final note on money. I've never been too much of a believer in the "money at stake" argument in met/climate work since it's indirect money. Yes, getting grants impacts a professor's likelihood in getting tenure and an extra month or two salary in the summer and impacts a lab scientist getting a promotion, but it's not like they're pocketing that money. Regardless of the results of a study, there is always more to look at, more questions to ask, more to understand. Right answers are nice, right questions are better, and ultimately more profitable for scientists. I think that's a good thing.

[skierinvermont]

Wow.. thank you for that very refreshing first hand perspective, I've never heard the process described in such a way before. Most of my understanding came from reading about the process, reading peer-reviewed articles, and from talking with professors but they never went into as much detail.

[nzucker]

Wow.. thank you for that very refreshing first hand perspective, I've never heard the process described in such a way before. Most of my understanding came from reading about the process, reading peer-reviewed articles, and from talking with professors but they never went into as much detail.

When is your first peer-reviewed article being published? [/matt stern]

[skierinvermont]

When is your first peer-reviewed article being published? [/matt stern]

[stellarfun]

AtticaFanatica, thanks for the exposition.

Five comments.

1.) Authors who find a paper rejected in a peer-reviewed journal often submit the paper to another peer-reviewed journal for publication. There is a pecking order in journals, and authors have to gauge which journals are most likely to publish their paper, given its content. Scientific journals are sometimes ranked by impact factor and the authors love to get their articles published in journals with the highest impact factor ratings, but rarely do.

The highest ranked journals from an impact factor (in one set of rankings) are:

Nature

Science

New England Journal of Medicine

Cell

Proceedings of the National Academy of Sciences

2.) Often, after an article is published in a peer-reviewed journal, readers will write letters to the editor, seeking clarification, or taking issue with either facts or findings. Such letters (if they have merit) are typically printed, and the authors of the paper are expected to reply, with the replies also printed. (Letters congratulating the authors do not make into print; these add nothing to the science.)

3.) Occasionally, a peer-reviewed journal will publish a comment on a published article in the same issue. The comment might elaborate on the significance of the article (Wow, this is BIG news! sort of stuff), or might raise a contrary view (e.g., the science in the article is inconsistent with known physical laws, and the findings needs to be replicated) To my knowledge, comments are not peer-reviewed. (Also, if the science in the paper is contrary to other science (published elsewhere), the authors are expected to address this in their paper.)

4.) Articles are often published in the expectation that other scientists will try to duplicate the findings through their own independent research, thus confirming the science, or raising questions about its validity.

5.) Published articles later found to be based on flawed science, erroneous methodology, or simple fraud, are formally withdrawn by the peer-reviewed journal..

A good example of the to and fro that goes on when dramatically new science is published is what happened after Alvarez et al first published the science for the K-T event.

With regard to #5, a recent example was the British doctor who fabricated and manipulated data purporting to show a link between childhood vaccines and autism.

.

[AtticaFanatica]

AtticaFanatica, thanks for the exposition.

Five comments.

All good points. Regarding #'s 2 & 3, you are right that comments are not peer-reviewed (at least in AMS journals) unless a lab has their own internal review system like NSSL has. The comment typically takes issue with results or methodology from the paper in question. The authors of the article are allowed to see the comment before it is published and are given an opportunity to publish a reply to the comment. If they do respond, both the comment and the reply will be published in the same issue of the journal. A good example of a comment can be found with regards to an article here; the comment is here and then the author's reply here. Comments are, in my opinion, underutilized, probably because they are not anonymous like reviews are.

[Hambone]

I hire scientists all the time. They all know where the check comes from. That doesn't mean they will totally prostitute themselves, but they know where the check comes from.

The vast majority of money flowing into climate study flows to AGW proponents, guys like Skierinvermont are dependent on keeping those checks flowing.

[AtticaFanatica]

That's nice and all, but I don't know what that has to do with this thread.

[WeatherRusty]

I hire scientists all the time. They all know where the check comes from. That doesn't mean they will totally prostitute themselves, but they know where the check comes from.

The vast majority of money flowing into climate study flows to AGW proponents, guys like Skierinvermont are dependent on keeping those checks flowing.

That should be no great surprise when studies indicate that 97% of all actively publishing scientists doing climate related research agree that the climate is warming and that human activities are the primary cause.

[WeatherRusty]

so the long explanation of the peer review process by an actual participant and scientist is irrelevant to this thread?

Why would you suggest that? By detailing the peer-review process and how it is conducted, a reassuring confirmation in the soundness of the process is realized. That same peer-reviewed process is at work in the case of climate research as confirmed by the general acceptance of that science by institutions such as the National Academies of Science, the American Geophysical Union and virtually every other prominent scientific organization and academic institution which assesses the situation.

The scientific debate is not with regard to the general principles of AGW, but rather in the details. The major debate is purely political in nature.

[ninox33]

I hire scientists all the time. They all know where the check comes from. , guys like Skierinvermont are dependent on keeping those checks flowing.

I hope you cheque their references and back grounds mmmmm

[A-L-E-X]

AtticaFanatica, thanks for the exposition.

Five comments.

1.) Authors who find a paper rejected in a peer-reviewed journal often submit the paper to another peer-reviewed journal for publication. There is a pecking order in journals, and authors have to gauge which journals are most likely to publish their paper, given its content. Scientific journals are sometimes ranked by impact factor and the authors love to get their articles published in journals with the highest impact factor ratings, but rarely do.

The highest ranked journals from an impact factor (in one set of rankings) are:

Nature

Science

New England Journal of Medicine

Cell

Proceedings of the National Academy of Sciences

2.) Often, after an article is published in a peer-reviewed journal, readers will write letters to the editor, seeking clarification, or taking issue with either facts or findings. Such letters (if they have merit) are typically printed, and the authors of the paper are expected to reply, with the replies also printed. (Letters congratulating the authors do not make into print; these add nothing to the science.)

3.) Occasionally, a peer-reviewed journal will publish a comment on a published article in the same issue. The comment might elaborate on the significance of the article (Wow, this is BIG news! sort of stuff), or might raise a contrary view (e.g., the science in the article is inconsistent with known physical laws, and the findings needs to be replicated) To my knowledge, comments are not peer-reviewed. (Also, if the science in the paper is contrary to other science (published elsewhere), the authors are expected to address this in their paper.)

4.) Articles are often published in the expectation that other scientists will try to duplicate the findings through their own independent research, thus confirming the science, or raising questions about its validity.

5.) Published articles later found to be based on flawed science, erroneous methodology, or simple fraud, are formally withdrawn by the peer-reviewed journal..

A good example of the to and fro that goes on when dramatically new science is published is what happened after Alvarez et al first published the science for the K-T event.

With regard to #5, a recent example was the British doctor who fabricated and manipulated data purporting to show a link between childhood vaccines and autism.

.

Im just going to address this point because I was having a discussion about this tonight. There shouldnt have been so much "to and fro" -- that there was is just another example of stubborn ego on the part of the old guard ("establishment") who were unwilling to let go of their cherished beliefs because they felt threatened.

You see this time and again in science (in every human endeavor, really) and this is a huge inertial driven force that holds progress back. Fortunately, progress eventually happens nevertheless and leaves these old fogeys behind.

http://en.wikipedia....saur_Extinction

After the publication of their seminal paper in 1980 an outcry was heard from some members of the geological community as the impact theory was a significant challenge to conventional theory, and an often acrimonious scientific debate ensued. Ten years after this initial proposal, evidence of a huge impact crater called Chicxulub off the coast of Mexico strongly confirmed their theory. Unfortunately for Alvarez, the finding of the crater was subsequent to his death.

I think one of the greatest and most salient quotes from any scientist ever came from none other than the great Max Planck, father of quantum mechanics.

"A new scientific truth does not triumph by convincing its opponents and making them see the light, but rather because its opponents eventually die, and a new generation grows up that is familiar with it."

^^^ that right there tells you all you need to know about the flaws of the human personality.

Restated: "You can't teach an old dog new tricks."

Another example of the failure of the process was when pharmaceutical companies forged the research data for drugs such as Avandia and Vioxx. The FDA in its usual resistance to going against the industry was hesitant in instituting recalls, but fortunately the Journal of the American Medical Association and the New England Journal of Medicine called the pharmaceutical companies out for what they had done in the "research" they had published, plus mounting public pressure and the multiple billion dollar class action lawsuits also had a nice effect