how to tell good research from bad

Psychedelic drug research has made a dramatic comeback amid a heady mix of declining societal attitudes, the lure of commercial opportunity, concerns about the “war on drugs” and a desire to develop new ways to treat mental illness.

You may have read in the media that there’s a new study showing that ketamine can eliminate depression, or that psilocybin is effective in treating post-traumatic stress disorder, or that microdosing LSD makes you more creative.

With all this zeal, what research is worth your time and, above all, your trust? Of course, what your time is worth depends on what you want.

I am a doctor, drug researcher and clinical investigator. So I’m interested in whether psychedelic therapy can be a new form of medicine. This question requires evidence from clinical trials. This is what I will focus on here, although some of the principles apply to medical research more broadly.

magazines

Your source first. Good scientific research is published in peer-reviewed scientific journals. Peer-reviewed means that independent experts have read the paper and criticized it anonymously. This is an important form of verification. If the journal you are viewing does not support peer review, continue.

Some magazines claim to be high-quality companies that publish peer-reviewed articles, but are actually pop-up money-making programs that publish everything.

Spotting these is a bit like spotting a spam email or social media post. Poor grammar, spelling, and formatting errors, substandard websites, and statements that seem too good to be true are all telltale signs of a journal that wouldn’t let the truth get in the way of a good publishing fee.

In contrast, high-quality journals tend to be long-established, indexed in scholarly databases such as PubMed, and tend to have good impact factors (a measure of how often the journal’s articles are cited). While this is not a perfect metric, it is useful as a guide and is stated on the journal’s home page. A higher number is more reassuring.

With a quality diary, you’re already halfway there.

authors

Before reading anything about the paper, find out who the authors are, where they work, and what their disclosures and funding sources are (this is usually given at the end of an article). Authors who excel in their field often have excellent reputations.

But they also have more to lose from results that don’t fit their theories. They tend to be paid consultants to companies looking to commercialize new treatments.

Just because a study comes from a groundbreaking, top-class institution doesn’t mean you should trust it blindly. In fact, the very teams that were pioneers may also be deeply biased. In other words, why would we have ended up in such a stigmatized field unless we had a strong positive bias?

However, institutions and research teams with good reputations deserve them because their peers respect their methods and believe their results. So, overall, go for the most respected authors, but keep in mind the other factors that come into play.

Data

Now look at the paper for yourself. For clinical research, the multicenter, randomized, placebo-controlled trial is king. Almost all psychedelic research is not (yet).

Initial tests take place in a facility. That’s fine, but says nothing about whether the treatment works beyond that institution. To do this, you need a multicenter study. The more centers the better.

If it works in many centers, there is more reason to believe it will work in the real world. This is called “generalization” and is an unanswered question for psychedelics.

Randomized and placebo-controlled refers to the random assignment of participants to two or more groups, one of which will be treated with a placebo (dummy pill). If you don’t have a placebo control group to compare to, you don’t know if the effect you’re seeing in the treatment group may not have happened.

If there is no randomization, any effect you observe could be due to something else common to one of the groups.

Early psychedelics trials were often not randomized or controlled. That’s fine, but not much can be inferred from these “pilot” studies. They just show that the research can be done.

Big tries

The more participants a study has, the more “statistical power” it has to detect a real effect (or a real lack of an effect). This often requires hundreds, even thousands, of participants.

These trials are expensive, which is why many large-scale clinical trials are funded by companies to raise the money to run the trial. But do not refuse commercial studies.

Yes, profit and health care are not easy bedfellows. However, commercial trials are much more tightly regulated than non-commercial trials. Almost every drug we have today has been approved based on commercial trials.

pre-registration

All clinical trials should have a “pre-registered primary outcome”. The primary result can be anything: a blood test result, an imaging finding, or a measure of depression. The study is designed around this result.

Pre-registration is done on sites like clinicaltrials.gov before the trial begins. If the researchers didn’t preregister their hypothesis, primary outcome measure, and methods of analysis, they might have picked the results you’re reading.

In other words, if you torture your data hard enough, they will tell you anything you want. This is one of the great research sins.

If I toss a coin ten times, then I do it again and again, at some point I randomly get ten heads. Here is the same principle. The more measurements I include in an experiment and the more methods I choose to analyze the data, the more likely I am to get a “significant” result.

One last thought before you go. No clinical study or research can tell you anything for sure. The more a result is replicated, the more credible it becomes.