Understanding Clinical Trial Publication Bias and Unreported Results

Jump To Section

• The Scale of the Publication Bias Problem
• Ethical Obligation in Clinical Trial Reporting
• Trial Registration as a Partial Solution
• Media Influence on Result Dissemination
• Risks of Data Manipulation and Cherry-Picking
• Full Transcript

The Scale of the Publication Bias Problem

Dr. Stephen Evans, MD, presents a stark statistic at the outset of his discussion with Dr. Anton Titov, MD. He reveals that a mere 40% of clinical trial results are reported within the legally mandated timeframe. Furthermore, a significant 30% of clinical trial results are never reported to the public at all. This creates a massive gap in the complete medical evidence base available to doctors and researchers.

This issue, known as publication bias, means the scientific literature does not reflect the full picture of a treatment's effects. The problem extends far beyond COVID-19 therapies and affects all areas of medicine, including both drug and non-drug treatments.

Ethical Obligation in Clinical Trial Reporting

Dr. Stephen Evans, MD, emphasizes a fundamental ethical duty owed to the public and to trial participants. Patients who volunteer for clinical trials assume a risk for the benefit of future patients and scientific advancement. There is a profound obligation to properly analyze and report the results back to the public once a trial is completed.

Failing to publish results constitutes a breach of trust. It wastes the contribution of participants and can lead to other patients receiving treatments based on an incomplete, and therefore inaccurate, understanding of the risks and benefits. Dr. Anton Titov, MD, highlights this as a critically important issue in modern medical research.

Trial Registration as a Partial Solution

A major effort to combat publication bias is the practice of trial registration. Dr. Stephen Evans, MD, explains that registering randomized trials prior to their conduct is a key strategy. This creates a public record of the trial's existence and its intended outcomes, making it harder for sponsors or investigators to simply ignore negative or neutral results.

Registration allows watchdogs and other researchers to pursue those who registered the trials to find out the answers. While this system goes a long way toward ensuring transparency, Dr. Stephen Evans, MD, notes it is not a perfect solution. Investigators can still find ways to circumvent the spirit of the requirement even when a trial is registered.

Media Influence on Result Dissemination

The conversation between Dr. Stephen Evans, MD, and Dr. Anton Titov, MD, delves into the powerful role media interest plays. Dr. Evans points out that everyone, including journal editors, "likes headlines." Publication tends to occur preferentially for what he terms "really good news or really bad news."

Extremely positive results or reports of severe adverse effects are deemed newsworthy. Indifferent or null results, such as a study conclusively showing that a drug like hydroxychloroquine doesn't work, often generate less interest. This media-driven dynamic shapes which studies get published and subsequently receive public attention, skewing the perceived evidence.

Risks of Data Manipulation and Cherry-Picking

Even with trial registration, a significant risk of data manipulation remains. Dr. Stephen Evans, MD, describes how investigators might "cherry-pick" the most interesting results, even if they were not the pre-specified primary outcomes. He quotes a famous adage in research: investigators can "torture the data until it confesses."

Statistically, variation in results means one can nearly always find something interesting by searching through the data. This practice leads to the publication of extreme results, whether extremely good or extremely bad, while the more nuanced truth that lies between them may remain unpublished. This ultimately corrupts the evidence base that guides clinical practice.

Full Transcript

Dr. Anton Titov, MD: Let's discuss the publication and dissemination of results of clinical trials in general. The pharmaceutical industry is the most profitable industry of all. Yet the results of only 40% of clinical trials are reported within the time prescribed by law, and 30% of clinical trial results are never reported.

Isn't there an obligation to the public and to the patients who took part in clinical trials to properly analyze and report the results back to the public when the trial is completed? What is the reason for the results of so many clinical trials not being reported to the public?

Dr. Stephen Evans, MD: This is a really important issue. It isn't only in relation to COVID-19; it's in relation to all sorts of treatments, and not only drug treatments. We have made major efforts to prevent this bias that we call publication bias by trying to make sure that randomized trials are registered prior to their being conducted.

Then at least you can pursue those who registered the trials to find out what the answers are. The problem is that we all like headlines. We all like good news. Unless you're a newspaper, or somebody who is paid to look at adverse effects, then you really like bad news.

So the problem is that publication tends to occur preferentially for really good news or really bad news. The indifferent news is not of interest to people. It isn't what the editors like; they can't have a nice headline.

The news that hydroxychloroquine doesn't work is, in most instances, not of great interest, whereas the idea that it does work is of greater interest. Now, because of our big concerns, even a study which shows that something doesn't work is of interest, and it will probably get published. But the general problem is there.

The problem of publication bias is addressed to a degree by these registers. But even then, it's still possible for people to manipulate their results and cherry-pick the best results, even when a trial has previously registered that they're going to look at a particular outcome.

And that isn't very interesting. It turns out they will nevertheless have a headline, another outcome that is more interesting. If you're a statistician, you know that the variation in results is really quite a concrete trouble.

You can nearly always find something interesting from your results. You will go searching for those results. Or as someone once put it, you torture the data until you make it confess.

The problem is that there is a tendency to like extreme results, whether they're extremely good or extremely bad. We see that in newspaper headlines; good news or bad news can sell newspapers if it is made dramatic.

The results that are in between and indifferent are of less interest in the media in general. The problem is that science is really interested in those intermediate results, the ones that don't show effects.

To get a proper view of the entirety of the evidence, we need to be sure we see all of it. Having registration of trials goes a long way towards doing that. But even when we do that, we find that some investigators will change, cherry-pick their results, and produce the ones that show an effect, show something interesting.

By interesting, it may mean very good, or it may mean very bad. It's interesting when there are adverse effects. We find this in a number of areas of medicine, where the results that are published tend to be at one extreme or another, and the truth that lies between them may not be published.