The Sad Reality of Peer-Reviewed Scientific Research

Scientific journals, even the most popular peer-reviewed ones, remain human organizations and therefore cannot avoid the human imperfection in rationality.

The process of peer-reviewing consists of having other contributors of a research journal read and criticise on your proposed article prior to publication. Usually several researchers are solicited and hold the responsibility of making sure content and writing style meet the expectations of scientific research :

  • proposing a form of new or innovative content,
  • with references to relevant prior work,
  • with a scientific approach,
  • and written in an academic and understandable form.

It is not rare that this process takes one year or more for a single paper consisting of a few pages. In some cases, the paper is rejected for reasons that are generally explained. Otherwise, it is published and its abstract (and sometimes full text) can be found online on the journal’s website or through aggregated databases.

Peer-reviewing is commonly seen as a preventive process to ensure scientific correctness and relevance of the papers submitted.

There is however a few things every researcher or user of research papers should be aware of.

Publication in a scientific journal does not mean it is right.

Indeed, for many reasons the content can end-up being wrong. Science take for granted something is valid until proven otherwise. So a publication can seem correct and relevant to peer-reviewers and thus be published. Another reason for publication of wrong content is poor peer-reviewing. The causes can range from poor level of competence of the peer-reviewers selected on this particular matter, or lack of time and consideration for the proposed paper, resulting in a rushed and risky acceptation. See the Jan Hendrik Schön scandal at the Bell Labs. (I just realized his supervisor was the same as mine during a week at the Bell Labs: Christian Kloc 🙂 the world is really small. Too bad I didn’t know that in 2005, I would have had tons of questions).

To try and avoid those papers, readers like to use certain metrics that yet cannot fully reflect the correctness of papers. Typically, they first look at the prestige associated to the paper. Indeed, some papers have very high expectations, like Nature or Science, while other journals can be more tolerant regarding the originality of the work proposed. It is a fairly good reflex, although there has been a few cases of abuse: See the Jan Hendrik Schön scandal at the Bell Labs. Readers also look the number of references, and their origins. The more and the most prestigious the journals cited, the better. If only a few references are presented, it’s either a breakthrough or a paper likely to be irrelevant. Readers also like to look at the number of time that a paper was cited in other papers. Indeed, common sense gives credit to someone’s whose work and served as the base of other people’s work. However, there has been cases of entire fields of science completely ruined by an initial early study that was ill-conducted, yet assumed true because it’s the only study available. A typical example would be Berger’s publications on Strength Conditionning. A work published in 1962 declared that maximal strength gains were observed for a number of sets superior or equal to 3. Since that time on this has been held true an dwidely practiced and repeated although it is very incorrect. See a full review on this issue here.

The problem with peer-review is that they are peers, with all the problems that this means.

Do you remember the that it was in class when your professor was too lazy to grade you and your schoolmates. They would tell the students to grade each other, and what’s the result ? A lot of students over graded. Why? Because they are your friends, you can’t give them a bad grade. Conversely, giving them a good grade you may ensure getting a good one from them.

Like kids at school, that give more importance to grades that the value of real understanding, adult researchers can show similar behaviour in the peer-reviewing process.

As a peer-reviewer, you may end up reviewing a scientifically relevant paper that simply opposes one of your friends’ work. This is more likely to happen in fields where an important paper is submitted to a set of the top reviewers in that field, whom are likely to be friends (or more rarely, ennemies!).

A typical example is that of Carpinelli trying to submit his paper in 3 journals on Strength Conditionning. Although reviewers agreed on the relevance of the content, higher people in the journal (Director in Chief, etc) gave order to reject the paper. This paper, although very well researched, was a review that is highly critical to the reputation of the American College of Sports Medicine (ACSM). Because of conflicts of interest , that paper was simply rejected. It seems a large number of the top authors in the rejecting journals were also top contributors to the paper incriminated. Read here the full story.

Conversely, because you are friends with the reviewers, this can occasionally let a poor paper be published. Again, being friends with the reviewers is a rare thing in most cases, it has happened though and will keep occuring.

In conclusion, there are many reasons that can make relevant papers not to be published or, on the contrary, irrelevant paper to be published. One of them is peer-reviewing. The field of scientific research intrinsically holds the eternal defect of being done intrisically imperfect humans. Science is not the field of absolute truth although it aims for it. Instead, scientific research ought to be seen as a constantly improving temporary best acceptable truth”. This being said, as brilliantly expressed by editor-in-chief of the Journal of Exercise Phisiology, Robert Robergs (2004) in “A CRITICAL REVIEW OF PEER REVIEW: THE NEED TO SCRUTINIZE THE “GATEKEEPERS” OF RESEARCH IN EXERCISE PHYSIOLOGY” :

Just because an article is published does not mean it is correct. All scientists need to be aware that when an article is published, the door is open to constructive criticism.

(…)

Conversely, treating publications and colleagues with reverence promotes a type of class structure within the Scientific Method that functions like a disease that eats away at whatever integrity the process once had. The end result is a poorly researched discipline and/or topic.


Personally, I feel there are too many poorly researched topics within exercise physiology and its related disciplines. I am tired of being a victim and a witness of the diseased editorial peer review process within too many journals pertaining to exercise physiology, applied physiology, and sports medicine. I am proud of my professional and ethical duty to publish the manuscript by Carpinelli, Otto and Winett. I know that all who read this manuscript will gain in their understanding (…)

Isn’t that the goal we are trying to achieve in editorial peer review?

Be Sociable, Share!

One thought on “The Sad Reality of Peer-Reviewed Scientific Research

  1. Who reviewed this very article? 🙂

    YB: Good one 😉 But this is not a research journal Erwin, although it aims at providing links and sources as much as possible.

Leave a Reply

Your email address will not be published. Required fields are marked *