The Sad Reality of Peer-Reviewed Scientific Research

Scientific journals, even the most popular peer-reviewed ones, remain human organizations and therefore cannot avoid the human imperfection in rationality.

The process of peer-reviewing consists of having other contributors of a research journal read and criticise on your proposed article prior to publication. Usually several researchers are solicited and hold the responsibility of making sure content and writing style meet the expectations of scientific research :

  • proposing a form of new or innovative content,
  • with references to relevant prior work,
  • with a scientific approach,
  • and written in an academic and understandable form.

It is not rare that this process takes one year or more for a single paper consisting of a few pages. In some cases, the paper is rejected for reasons that are generally explained. Otherwise, it is published and its abstract (and sometimes full text) can be found online on the journal’s website or through aggregated databases.

Peer-reviewing is commonly seen as a preventive process to ensure scientific correctness and relevance of the papers submitted.

There is however a few things every researcher or user of research papers should be aware of.

Publication in a scientific journal does not mean it is right.

Indeed, for many reasons the content can end-up being wrong. Science take for granted something is valid until proven otherwise. So a publication can seem correct and relevant to peer-reviewers and thus be published. Another reason for publication of wrong content is poor peer-reviewing. The causes can range from poor level of competence of the peer-reviewers selected on this particular matter, or lack of time and consideration for the proposed paper, resulting in a rushed and risky acceptation. See the Jan Hendrik Schön scandal at the Bell Labs. (I just realized his supervisor was the same as mine during a week at the Bell Labs: Christian Kloc 🙂 the world is really small. Too bad I didn’t know that in 2005, I would have had tons of questions).

To try and avoid those papers, readers like to use certain metrics that yet cannot fully reflect the correctness of papers. Typically, they first look at the prestige associated to the paper. Indeed, some papers have very high expectations, like Nature or Science, while other journals can be more tolerant regarding the originality of the work proposed. It is a fairly good reflex, although there has been a few cases of abuse: See the Jan Hendrik Schön scandal at the Bell Labs. Readers also look the number of references, and their origins. The more and the most prestigious the journals cited, the better. If only a few references are presented, it’s either a breakthrough or a paper likely to be irrelevant. Readers also like to look at the number of time that a paper was cited in other papers. Indeed, common sense gives credit to someone’s whose work and served as the base of other people’s work. However, there has been cases of entire fields of science completely ruined by an initial early study that was ill-conducted, yet assumed true because it’s the only study available. A typical example would be Berger’s publications on Strength Conditionning. A work published in 1962 declared that maximal strength gains were observed for a number of sets superior or equal to 3. Since that time on this has been held true an dwidely practiced and repeated although it is very incorrect. See a full review on this issue here.

The problem with peer-review is that they are peers, with all the problems that this means.

Do you remember the that it was in class when your professor was too lazy to grade you and your schoolmates. They would tell the students to grade each other, and what’s the result ? A lot of students over graded. Why? Because they are your friends, you can’t give them a bad grade. Conversely, giving them a good grade you may ensure getting a good one from them.

Like kids at school, that give more importance to grades that the value of real understanding, adult researchers can show similar behaviour in the peer-reviewing process.

As a peer-reviewer, you may end up reviewing a scientifically relevant paper that simply opposes one of your friends’ work. This is more likely to happen in fields where an important paper is submitted to a set of the top reviewers in that field, whom are likely to be friends (or more rarely, ennemies!).

A typical example is that of Carpinelli trying to submit his paper in 3 journals on Strength Conditionning. Although reviewers agreed on the relevance of the content, higher people in the journal (Director in Chief, etc) gave order to reject the paper. This paper, although very well researched, was a review that is highly critical to the reputation of the American College of Sports Medicine (ACSM). Because of conflicts of interest , that paper was simply rejected. It seems a large number of the top authors in the rejecting journals were also top contributors to the paper incriminated. Read here the full story.

Conversely, because you are friends with the reviewers, this can occasionally let a poor paper be published. Again, being friends with the reviewers is a rare thing in most cases, it has happened though and will keep occuring.

In conclusion, there are many reasons that can make relevant papers not to be published or, on the contrary, irrelevant paper to be published. One of them is peer-reviewing. The field of scientific research intrinsically holds the eternal defect of being done intrisically imperfect humans. Science is not the field of absolute truth although it aims for it. Instead, scientific research ought to be seen as a constantly improving temporary best acceptable truth”. This being said, as brilliantly expressed by editor-in-chief of the Journal of Exercise Phisiology, Robert Robergs (2004) in “A CRITICAL REVIEW OF PEER REVIEW: THE NEED TO SCRUTINIZE THE “GATEKEEPERS” OF RESEARCH IN EXERCISE PHYSIOLOGY” :

Just because an article is published does not mean it is correct. All scientists need to be aware that when an article is published, the door is open to constructive criticism.

(…)

Conversely, treating publications and colleagues with reverence promotes a type of class structure within the Scientific Method that functions like a disease that eats away at whatever integrity the process once had. The end result is a poorly researched discipline and/or topic.


Personally, I feel there are too many poorly researched topics within exercise physiology and its related disciplines. I am tired of being a victim and a witness of the diseased editorial peer review process within too many journals pertaining to exercise physiology, applied physiology, and sports medicine. I am proud of my professional and ethical duty to publish the manuscript by Carpinelli, Otto and Winett. I know that all who read this manuscript will gain in their understanding (…)

Isn’t that the goal we are trying to achieve in editorial peer review?

Computing in the 21st…year before Christ

Microsoft Research and Microsoft Research Asia organize every year a conference called “Computing in the 21st century”. The subtitle was “New Horizon of Computing”.

This year’s version had no less than 3 Turing award winners (Turing awards are something like the Nobel Prize of computing), the Senior Vice President of Microsoft, and a Managing Director at Microsoft Research Asia and at the same time IEEE fellow (a very prestigious title of one of the largest and prestigious scientific community in the world).

Shall you have any interest in the future of computing, based on this prestigious casting, common sense would tell anyone to cancel everything and just go get your get enlightened by insightful talks.

After an opening session in which Microsoft Research seemed to re-invent Google Maps and present that as innovation, and a remarkable speech reading at the public from officials at NTU and NUS, speech was given to the first keynote speaker, Turing award winner Raj Reddy. Although I do respect his good intentions on giving the poor a chance of education though e-learning, disappointment settled down when he presented a completely obsolete piece of work running on Internet Explorer 4 or 5 (a 1998-looking online library web page included ASF videos and text on various topics).

Later, Senior Vice President of Microsoft, Rick Rashid, mentioned solving the heat problem in data centers by better distributing spatially the processes in the data center so that the hot machines aren’t close to each other, thus using less air-conditioning.He stressed that fact that this was done to preserve the environment. That’s 50% of the computer energy consumed: Data centers, the other half being computers of people like you and I. It’s better to listen to this when you do not know the meaning of symptom-solving or Microsoft Vista 🙂

Later, Managing Director at Microsoft Research Asia and IEEE fellow Hsiao-Wuen Hon really got us discreetly laugh at Microsoft, presenting totally irrelevant innovations such as :

  • image debluring, a topic fairly obsolete nowadays in digital imaging.
  • a tool that tells you which word is more used than another if you hesitate between the two (that’s what EVERYBODY does using the Google number of results as a metric for word popularity and use, it takes a few hours to program from scratch for a graduate student in computing).
  • Gigapixel images. Again that’s very obsolete. Especially of you do that by stitching (=putting together) images taken at different angles. Guess what? That’s what they do. He showed how cool it is to be able to zoom and (unlike in classic images) be able to still get high definition. They innovate all the more by adding ambient sounds and audio descriptions as you flyover different monuments or locations, it also allows advertising because you would see the stores on the pictures by zooming. Waw! This part may have been of interest to anyone having been in jail for the past 5 years thus never hearing about nor using Google Maps.
  • a bilingual search engine that allows inputs in both Chinese and English. It’s open source since I did very complex reverse engineering and am glad today to share this with you: Take a dictionary, take a search engine, shake well, add decoration ( 2 tbsp. of marketing and a whole fancy logo). Tadaaaaaa !
  • A collaborative travel website based on Google Maps (oops Microsoft Map). I don’t really get how on earth they intend to draw the attention of internet users that are just fed up hearing about new collaborative social networking websites.

Finally, a Microsoft speech on the future could not go without showing how cool Microsoft Surface is. Firstly, this system was invented by Jeff. Han and not by Microsoft Research as they claim. Secondly, the technology is great, but what do you do with it that is useful? Jeff. Han designed various novel uses of a multitouch system that do create an added value to government agencies for example. That’s precisely where Microsoft fails by sticking to obsolete and useless examples of use of the multitouch function. A technology alone is nothing, the added value that it brings to people should be it’s reason to exist. Multitouch interfaces and other gadjets are typical of this gap between technology coolness and real-life usefulness.

The Q&A session was a series of people asking questions that the board would answer right away even if they did not fully understand them. You know…when you hear an answer and you forgot the question. And the one answering usually ends with “I hope that answered your question”, and the guy who asked says Yes just out of politeness… If you never felt that you probably never went to school 🙂

To the computing safety, the answer of Turing award winner Raj Reddy was “Charge Emails one cent or penny to avoid spam”. Mhh Okay..

Always look at the bright side, the snack were good and in spite of a tremendous lack of insight, it was comical. Thank you Microsoft Research 🙂

More seriously. Take home lesson:

  • A very good researcher is generally much less realistic and much less in phase with business than any very bad entrepreneur.
  • When someone has a long list of awards in a specific topic, assuming that these people are well-rounded or pertinent about other topics is generally a wrong assumption.
  • Not because you have a room full with 2000 people should you make no effort to bring insight the attendants, especially if your name is big and your conference is named “Computing in the 21st Century”.
  • Not because you work for Microsoft Research does it mean you will work on innovative or hot or possibly marketable projects. All the “innovations” I saw today were all either obsolete (no or irrelevant market demand or market potential) or just mash-ups of what other companies have done earlier and better. Microsoft apparently can afford investing in such useless projects as most of those presented. Is it just investment in corporate image, or are they really adopting a strategy or exploration and risk? If the latter, their projects should really be shortlisted to try and reach something close to the relevance of Google Labs’ projects. Or is Microsoft Research working on really interesting things yet to be seen and does not want to talk about them yet? If so, they should be aware sometimes better to show nothing rather than low-quality contents because that clearly destroys one’s credibility.

How to Write a Research Proposal

While sorting my desk I found back a summary that I had written based on the excellent yet simple NSF recommendations on writing a Research Proposal : Twelve Steps to a Winning Research Proposal by George Hazelrigg.

To a large extent, this advice can be applied to business proposals as well, aiming for a contest or a grant.

Here is an even shorter version:

  1. Show you have researched previous work.
  2. Your proposed research must match with the philosophy or interests of the authority you are sending your proposal to.
  3. Make sure you are eligible for the program.
  4. You must suggest RESEARCH, and nothing like other research-related activities: development, market study, etc…
  5. Propose a detailed plan on how you intend to conduct your research, present the steps, the alternatives, the difficulties.
  6. State clear objective.
    The objectives of the proposed research are
  7. Present the context.
    How does my proposed work fit in/impact other people’s work?
  8. Grammar, spelling, vocabulary: PERFECT.
  9. Be concise, Respect format.
    How much is the reader going to enjoy reading my proposal?
  10. Write for the reviewers.
    What expertise do the people I am writing this for have?
  11. Proof the document before sending it.
    Who can help me read this and point out mistakes?
  12. SUBMIT ON TIME.

For step 11 (getting colleagues, friends or family read your document), it is often not done properly. People tend to read the report THEMSELVES so as to avoid imposing a burden on their peers. If you really have no one to help you with this, make sure to keep away from your document for at least 3 days before you look for mistakes. This helps your brain look at the document with a new eye. On the contrary, while rushing to self-proof a document, we tend to overlook the mistakes: the brain assumes it is reading what WAS INTENDED to be written, which is often a wrong assumption…

Back to getting peers to help you: In order to make sure you do not impose a burden on them, BE THERE anytime they need you review anything. We often by default just escape this kind of tedious tasks: What do I gain in this?. Just SAY YES, or even suggest to help before people ask. GIVE TO OTHERS. But do it well and more importantly do it genuinly, i.e. without expecting anything in return. The good side effect of this being that people in your circle of trust will be available for you when you need help from others.

Problem Solving: Problems, Symptoms, and Remedies

When facing what seems to be a problem, we nearly always cure/fix the symptoms and not the core issue itself. Distinguishing symptoms from the problem itself is however quite simple.

Symptoms, a necessary alarm

There is nothing more natural for a person than wait for symptoms to come up so as to identify a given problem. For example, many dentists and doctors will agree that most patients are brought to consult them only after starting to feel pain or see some funny things on them. In the same fashion, you would only fix a car when you get obvious sensible signals that it is not functioning well. On the other hand, how else are we supposed to know? We indeed don’t want to engage in a plethora of preventive actions for everything and that’s completely right. So far waiting for symptoms is fine, but our answer to them gives a lot of room for criticism and questioning.

Symptom-solving, an omnipresent issue

In most cases indeed, the pain, in itself, is not the problem. It is just a symptom of something going wrong in one’s body or mind. More generally, when a problem occurs, what arises to out senses (often unpleasantly) is symptoms. It is necessary to detect the problem. Nonetheless, there is a sad constant in people that is to just get rid of whatever shows there is a problem, regardless of the problem. That is typically symptom-solving, you would be scared to realise how often we do that. Here are a few example of common inadequate problem-solving cases taken in everyday life:

  • Coffee and Energy Drinks
    Symptoms cleared : tiredness
    Real problem : exhaustion, lack of sleep, anxiety…
    Appropriate Problem Solving : Ask yourself why your are tired, get some sleep.
  • Chewing gums and mints
    Symptoms cleared : bad breath
    Real problem : Oral bacteria
    Appropriate Problem Solving : Brush teeth, use tongue cleaner and mouthwash…
  • Face Make up
    Symptoms cleared : visible acne, face scars…
    Appropriate Problem Solving : Cure acne (itself a symptom not a problem !!! the real problem is excessive sebum secretion, which itself could also be a symptom of stress, itself a symptom of your bad relationship with your boss !)
  • Rushing, last minute
    Symptoms cleared : delay
    Real problem : Lack of organisation or motivation
    Appropriate Problem Solving : Question habits, plan, schedule…

It takes one of those unnatural efforts to really wonder what the real problem is when you could actually just fix the symptoms temporarily. It is not to be mistaken with being lazy, but given only symptoms that are annoying, it is very easy for the mind to think of them as problems, “something you don’t want”. Besides this, another reason why it’s particularly hard to get the mindset of appropriate problem solving is that, very often, it is beneficial in the long-term: It’s a long term way of thinking….Unfortunately most people do unconsciously prefer comfort in the present at the price of long but bearable suffering (symptom-solving) rather than big pain and then long-term comfort (problem-solving). Money loans are a typical expression of this human tendency.

Problem-solving and Innovation

The everyday process of entrepreneurs, researchers, consultants and the alike basically boils down to facing problems and having to solve them. The ability to solve a problem is particularly vital in this field because symptom-solving often results in costly expenses. Let’s say you are a major web-based company and your website is running particularly slowly. You either don’t care what’s the reason and spend twice as much in a more powerful hosting solution, or you fix what was making the website slow and eventually move to a cheaper hosting solution because you don’t need much performance any longer.

Besides cost, another focus should be time efficiency and scalability. Fixing symptoms is an eternal process, especially when your hack introduces new symptoms. Also, symptom fixing prevents scalability in that whatever transformation you make of your system will have to include the fix. Worse, it may require to adapt the fix as well… Such a waste of time and resources.

Human Body : How The Pharmaceutics Industry shortcuts Natural Body Control System

Our lack of lucidity in problem-solving is the gold-mine of the pharmaceutical industry, although a branch of it tries to tackle the problems at the very root instead of old school symptom-curing.

The human body has this very peculiar ability to send alerts to itself. These are receivable through the senses for (I like to believe) you to be aware of the problem and of course do something about it. Most engineers know this functioning as control. When having a precise objective, control is the mere ability to monitor the outcome (output) in real time correct the action (input) based on observation (the difference between objective and observation). A typical control system could be an elevator for example. The speed of the motor (input) could be proportional to the number of stairs to go (difference between objective: desired stair, and action observed: where is the lift now?).
You do control all the time without realising. When you drink your orange juice in the morning: the reason why you get some and without spilling it over is because you do control: the angle of your hand and cup is the input, and the flow of beverage is the output.

Look at the body as a control system now. Most of the time when you don’t feel good or have a big big headache, the core origin is very likely to be psychological, and if not many cases are found to be curable using the power of your mind over yourself. Doing the contrary, if you just ask yourself what’s wrong and focus on good things it doesn’t generate any profit. So instead the industry likes to sell you their complete collection of pain-killers. The pain-killer for colds, the pain-killer for migraines, the pain-killer for when you feel sad, the pain-killer for when you have a headache after reading your pain-killer bill… the stomach pain-killer to ease the stomach-ache caused by the tons of pain-killers you just absorbed, plus the 10 extra drugs to cover the side effects of each type of pain-killer… You see what I mean ? That’s big money ! One typical issue in symptom solving, is that by introducing a solution to a symptom, you introduce other problems, or side effects in the case of drugs. And it sometimes it gets to the ridiculous point where people choose to do nothing about their given problem because each solutions brought more trouble overall than if they do nothing.

More powerful than problem solving, problem avoidance

  • Isn’t back care the best medicine for back problems ?
  • Isn’t sleep the best remedy for tiredness ?
  • Isn’t protection the best cure for AIDS ?
  • Isn’t correct nutrition the best supplement for fat management ?
  • Isn’t good diplomatic relationships and constructive discussion and compromise the best remedy for wars ?and so forth…

Are we truly dumb enough to learn things the hard way when generations and generations have done it before ?

When it comes to personal health, today’s society just does not want to bother about being careful on what they expose themselves to. Look at the proportion of smokers, look at the sports people engaging in extreme activities with little knowledge and preparation, look at the massive amount of commonly tolerated junk foods and beverages…Once more, all this is a direct consequence of our inability to weight to harmfulness of things when they don’t impact us in the short-term. The underlying thinking process being that whatever happens (shall they believe that something bad will happen), they’ll just have to take the appropriate pain-killer…