The Problem With COVID-19 Treatment Trials

F. Perry Wilson, MD, MSCE


July 14, 2020

Find the latest COVID-19 news and guidance in Medscape's Coronavirus Resource Center.

This transcript has been edited for clarity.

Welcome to Impact Factor, your weekly dose of commentary on a new medical study. I'm Dr F. Perry Wilson from the Yale School of Medicine.

There has been an explosion of research into COVID-19, from its underlying biology to its potential treatments. I have not only lauded the scientific community for this rapid-fire pace of research but have contributed to the growing body of literature myself. All good, right?

Well, maybe not. It's possible that there might be too much of a good thing here, as this article, appearing in JAMA Network Open, shows.

Researchers from MD Anderson queried, the central US registry for (in theory) all clinical trials. Since 2007, if you were going to run a clinical trial that you wanted to publish eventually, you had to register it in before it got underway. The idea is to prevent the "burying" of negative clinical trial results. It's not perfect, but nowadays almost all legitimate trials have an entry on this site.

Okay, so the researchers looked for trials involving COVID-19, and the numbers here are really staggering.


After removing suspended and halted trials, they found 674 specific randomized trials of COVID-19 interventions. Most of these were treatment, not prevention, trials. Of those treatment trials, 132 — nearly a quarter — were randomized trials of chloroquine.

One hundred and thirty-two randomized trials testing chloroquine.

This could be a problem.

Remember that in a randomized trial, one group always wins, even if the drug doesn't do what you think it will do. One group always — by chance alone — has more deaths or longer length of stay, or whatever you are measuring. We account for that, though; we can use math to tell us how weird the results of our study are, assuming that the drug doesn't work.

Let's say I enroll 200 people in a trial of a magic bean to cure COVID-19.


One hundred swallow the bean, 100 get a placebo bean. If you saw, say, 10 deaths in the placebo group and nine in the bean group, would you be terribly excited? Your intuition should tell you no; a one-person difference in death seems like it might just be due to random chance. In fact, you'd get a result like that, or even more extreme, 81% of the time. Nothing to write home about. That 81% is the P value in these clinical trials. We have (rather arbitrarily) defined a P value of .05 as our threshold for statistical significance.


Using our bean example again, four deaths in the bean group compared with 10 in the placebo group is a bit weird. Results like that would happen 10% of the time even if magic beans don't work. But three deaths in the bean group — well, a result like that happens only 4.5% of the time; we've passed that P value threshold.

And that should feel about right to you. If you did this trial and had only three deaths in the magic bean group compared with 10 in the placebo group, you might really start to think, Huh, I guess that old peddler knew what he was talking about.

But that's for one trial.

What if I did 132 trials of magic beans?


Assuming that the magic beans are bunk, on average just over three of those 132 trials would be positive at that P value of 5% threshold. But, of course, it depends how the chips land. You can see that sometimes you get as many as eight positive trials out of 132, even when you are using a magic bean.

Now, I should note that a similar amount of trials would show just the opposite, that the magic bean increases the death rate significantly.

But here is the problem: What do you think will get talked about on Facebook, Twitter, and the nightly news?


That's right. Positive trials get way more airtime than negative or neutral ones, and that shapes the public perception of drug efficacy. And, honestly, it shapes physicians' perceptions too.

There's another problem with all these trials of the same thing: There aren't enough patients. Of the 201 trials recruiting solely in the US, the total expected enrollment was 146,688 individuals; 87,000 patients are needed to be enrolled in chloroquine-specific trials alone. It is not easy to recruit patients into clinical trials. This many, even with a raging pandemic, is not feasible. Many of these studies will never finish.

So, what are we to do? Well, the number-one thing is to realize that there will be individual randomized trials of interventions that don't actually work but are nevertheless positive. This will happen. You will see on the news that a new randomized trial shows that chloroquine or some other drug reduces mortality in COVID-19, and you will not be told of all the other trials that show that it doesn't. First, we need to be aware of this.

Second, we need to encourage researchers to work together on these projects. We need to foster collaboration across medical centers and research institutions, get these teams working together, do 20 really amazing trials instead of 120 mediocre ones. And that means fixing some of the regulatory hurdles around data sharing and IRB oversight, but also making it worthwhile for researchers who are desperate for high-impact publications to join a study as a (gulp) middle author.

In the meantime, I'll be out conducting 132 trials of my magic bean. You'll hear about, say, three of them right here on Impact Factor.

F. Perry Wilson, MD, MSCE, is an associate professor of medicine and director of Yale's Program of Applied Translational Research. His science communication work can be found in the Huffington Post, on NPR, and here on Medscape. He tweets @methodsmanmd and hosts a repository of his communication work at

Follow Medscape on Facebook, Twitter, Instagram, and YouTube


Comments on Medscape are moderated and should be professional in tone and on topic. You must declare any conflicts of interest related to your comments and responses. Please see our Commenting Guide for further information. We reserve the right to remove posts at our sole discretion.