Most Research Findings Wrong

 An August 2005 article in New Scientist stated that "Most published scientific research papers are wrong, according to a new analysis. Assuming that the new paper is itself correct, problems with experimental and statistical methods mean that there is less than a 50% chance that the results of any randomly chosen scientific paper are true."

A verdict that may make the philosophically inclined among our readers ponder how fragile our concept of "reality" and "truth" is. Or, to speak with the more pragmatic, "What the hell...?"

Study leader John Ioannidis from the University of Ioannina School of Medicine in Greece didn't look at any particular research papers to come to his disturbing conclusion. Instead, he used statistical methods based on the frequency of intrinsic flaws in scientific studies--such as too small sample sizes, poor study design, researcher bias, selective reporting, and so forth.

"We should accept that most research findings will be refuted," Ioannidis told New Scientist. "Some will be replicated and validated. The replication process is more important than the first discovery."

We don't find that very hard to believe. After all, the definition of a scientist in the 21st century has changed quite a bit from the original one. In the past, a scientist was an exceptionally curious individual with an exceptionally open mind, willing not to jump to conclusions but to investigate every possible avenue in the name of science.

Nowadays, with many studies sponsored by the industry or organizations with ulterior motives, research is habitually used to prove a point rather than come to an unbiased result. And as we know from pharmaceutical studies, more often than not undesirable outcomes of studies are being swept under the carpet and only the positive ones published.

But even with the best of intentions researchers are not fault-proof, as Ioannidis' analysis shows. Take the problem of statistical significance. A study is declared statistically significant when there's "only" a 1-in-20 chance that the results could be pure coincidence. But in today's complex fields of science, there are so many hypotheses to sift through that "If you test 20 false hypotheses, one of them is likely to show up as true, on average," states New Scientist.

That combined with too small studies--for example, drawing conclusions from a study with only 20 test subjects--and poorly defined protocol and endpoints makes for research pliable as Play-Doh in the hands of scientists.

Another major factor is if a field is "hot," according to Ioannidis. The more researchers are frantically competing for statistically significant findings in a given scientific area, the higher the risk of false results.

Here's a wonderful example of a false scientific finding that--while apparently overlooked by seasoned researchers--was discovered in your WWNK editor's presence by a complete layman.

One recent and much-published study on the effects of marijuana claimed that the drug allegedly causes new growth of brain cells. The "proof" for this hypothesis was an experiment with lab rats, which, after consumption of marijuana, managed to find food hidden in a maze much faster than their undrugged counterparts.

When your editor showed the article to a friend who used to smoke pot in college (and inhaled), he called out: "They didn't grow new brain cells... they just got the munchies!"

Quod erat demonstrandum.

** Advertisement **

Get in Before it's Too Late!

For a short while, likely only six to twelve months, alert investors have the opportunity to accumulate carefully selected gold stocks at bargain basement prices--prices that all but guarantee the sort of spectacular returns for which Doug Casey is famous.

His latest gold stock win just posted a 972% return. There's still time for you to get in ahead of the clueless masses and make an investment fortune. Act Now...

Posted 01-31-2006 1:05 AM by Doug Casey