"Studies Have Shown": Finding Frauds in the Age of Misinformation

What kind of music do coked out rats prefer? There’s a scientific study for that. Can the smell of lemon extract be used to induce erections in primates? There’s a scientific study for that. How about the role of oral sex in fruit bat copulation? There’s a scientific study for that.

Welcome to the “scientific age,” where you can find out anything you want to know. To paraphrase an old adage, if you can imagine it, it’s already in a scientific study. In a time flooded with demand for research, the body of scientific literature is overwhelmed with seemingly meaningless, contradictory, and outright useless information, and most people have no idea what constitutes “reliable data.” For example, some studies have shown that aspartame causes cancer while other studies have shown aspartame is non-carcinogenic. Which studies can you trust?

The reality is that science, despite romantic notions about ethical scientists and the infallible objectivity of the scientific method, is an industry. Academic promotion, notoriety, and profit all stand to be gained from research publication. Politicians, academics, environmentalists, manufacturers, and medical companies all have a serious vested interest in the outcome of scientific research, blurring many ethical lines. This year, in fact, a team of academics deliberately published seven fabricated research articles in respected journals to demonstrate the prevalence of political activism within the field.

So how can the average consumer of scientific literature know what to trust? Perhaps the most concise and effective guidelines I’ve found came from reading about a different—though at times equally dubious—body of literature: American Supreme Court decisions.

In 1857, Abraham Lincoln delivered a speech on the infamous Dred Scott v Sandford Supreme Court decision (which concluded that the slave Dred Scott couldn’t sue for his freedom because he wasn’t a person). In his speech, Lincoln argued that the Court’s decision, although final, should not automatically be treated as the standard for American jurisprudence. Instead, he gave several criteria necessary for establishing a case’s credibility as a good judicial precedent. These same principles can be applied to promote a more reasonable approach to the analysis and acceptance of scientific research.

First, the conclusion should be unanimous. If a topic has significant, substantiated data supporting competing claims, then it can’t be treated as a settled question. This is especially applicable in facets of the climate change discussion. The reality of man-made climate change is widely accepted and scientifically supported. The extent of the human impact and long-term projections, however, can vary between studies and often pose points of contention.

Second, it should have no partisan bias. Always note who is funding, conducting, or publishing a particular study and ask why they are interested. The infamous “Vaccines cause autism” study published by former doctor Andrew Wakefield was retracted in part due to a conflict of interest (the other part: vaccines don’t cause autism). Wakefield stood to make a projected $43 million from multiple business ventures selling diagnostic kits for “autistic enterocolitis,” a falsified condition Wakefield claimed to have “discovered” in his research.

Third, the conclusion should be consistent with reason. Carefully review any conclusions that don’t pass the “smell test.” A recent study showed that global ocean temperatures were rising at far faster rates than previously thought, which subsequently incited a demand for major regulatory policy overhaul. Some scientists were shocked by the drastic difference, and deeper analysis of the data showed that the accelerated rate of change was actually the result of mathematical errors.

Fourth, there should be no assumption of facts. Analyze the data presented to ensure the conclusion reasonably follows from the evidence. Fudged data is, unfortunately, more common than most people realize and can take many forms. This cancer-related research article—widely cited before and after its retraction—was structured around several findings for which no original data could be produced, meaning data was likely fabricated.

And finally, the conclusion should be reaffirmed by other cases. Most scientific literature sites show how many times a given article has been cited in other research, which can be used to help gauge how a submission’s quality holds up over time. Conversely, websites like Retraction Watch monitor withdrawn papers and related statistics. When all else fails, scientific claims should be treated with an air of healthy skepticism until they can be successfully replicated and demonstrated over an extended period of time.

Blind faith and confirmation bias, on the other hand, have serious and deadly consequences. Just ask the parents whose children are suffering from the recent spike in measles outbreaks. What will it take to convince people not to risk children’s lives for a crock conspiracy theory? (Actually, funny you should ask: there’s a scientific study for that.) The consumer’s best defense against such misinformation is to be a thorough, critical, and skeptical reader. By applying the logic of Abraham Lincoln and the available resources online, students of scientific literature can arm themselves against the growing body of fraudulent, deceptive, and unsubstantiated studies.