When it comes to studies, there is no one size fits all – minimum sample size, ideal methodology or best biomarker. The importance of different aspects of a study will vary depending on the ingredient, the health benefits, the population being studied, and the goals of the study.
Annegret Nielsen, Senior Consultant at Analyze and Realize, points out that the way a study is conducted and reported will vary depending on the objective, ie. whether the evidence is in support of a health claim or whether it is carried out to support marketing communications.
However, she points out that the gold standard is always a double-blind, placebo-controlled, randomized study, and that for foods and dietary supplements, studies should be conducted in a healthy population to enable a health claim to be made.
Where do I begin to criticize a work?
Everyone has a different format they follow when reading a research paper – many read the abstract first, while others go straight to the methodology and conclusion, and still others go straight to the authors’ background. There is no “right” or “wrong” as long as the reader critically analyzes the design and results, rather than simply accepting the conclusions as fact.
Miguel Toribio-Mateas, research and design specialist and clinical neuroscientist, explains that the summary can be poorly written, doing the work a disservice. Therefore, it is important to read the abstract before making a decision about the study.
“My neurodivergent brain likes to take in an article from multiple angles at once. I’d like to know who the authors are… and I’m going in and out of the methods section to find out how the researchers went about collecting the data.
“The giveaway is the description of the methods. Who was involved in the study, how were the data needed provided, for how long, etc. These few lines can prove crucial when attempting to assess the quality of a study.”
sample size
In addition to ensuring that the sample size is large enough to produce statistically significant results, Nielsen notes that it is important to measure both baseline and post-treatment markers of health in order to measure change to be able to It’s not good enough to simply rely on the post-treatment results and compare the treatment to the placebo group if you don’t know what their baseline values were.
There is a trend toward big data from personalized nutrition companies, but Toribio-Mateas notes that bigger datasets don’t always mean better, especially when we, as an industry, want to enable new discoveries and innovations across a wide range of research areas.
“For the sake of innovation, I would not automatically discard evidence from well-designed RCTs with one control and only 20 participants on each arm. Hypothesis testing has to start somewhere, and if we asked for huge sample sizes as a non-negotiable requirement, we would end up with very few studies, mostly funded by those who already have a lot of power in the field.”
Results
When reading the results of a study, we want to see that the effect size is relevant for that specific indication and the duration of treatment is long enough to achieve the desired results.
However, Toribio-Mateas argues that there are benefits to gleaning the human input of the participants in addition to the hard data.
“Usually the participant provides samples. These are anonymized and the person becomes a number. Participant-reported results, obtained through validated self-reported measures, are essential in my honest opinion.
“The way we experience health is subjective, and we’re not talking about clinical trials of life-saving cancer drugs. We are talking about food and dietary supplements. Let’s hear what the participants have to say. I would like to see more RCTs that actually interview participants so that rich data can be collected that defies the sensitivity/specificity of validated measures.”
disruptive factors
When it comes to nutrition, and especially when it comes to probiotics, it is important to take all confounding factors into account, emphasizes Nielsen.
“When the ingredient goes through the gut, food intake has a big impact on the results.”
She also recommends doing a background check on potential confounders to ensure everyone is accounted for.
How can you tell if there is prejudice?
To determine if there is a risk of bias, it is helpful to read the researcher profiles and the funding sources.
Toribio-Mateas says it’s important not to assume that ownership automatically signals bias.
“Independently conducted studies are essential, but I think we need to take independence with a pinch of salt. One thing is that a company funds a study and controls the study protocol, has someone on their payroll directly involved in conducting the study, and is clearly using the research as an excuse for marketing.
“A very different situation is a company that provides some funding that is supplemented by a legitimate research organization, ie a university using government funds, with the scientists involved advancing their own research interests.
“If we discard small studies that test groundbreaking hypotheses based on potential funding bias, we risk stifling innovation. We have a huge amount of evidence on the mechanisms of how pro, pre, syn and postbiotics work. We need to start getting solid human evidence now.”
lay translation
Ewa Hudson, director of insights at biotic ecommerce analytics firm Lumina Intelligence, says her biggest irritation when reading academic studies is a lack of clarity.
“We need to be told the exact demographics and the exact ingredient that was studied and it helps if the results are provided in plain language to help the reader understand the implications for the broader market.
“Often the language can be very scientific and medical and it can be difficult to know whether or not a noted improvement is actually significant. Even if the authors only provide a few sentences that provide their own interpretation of the results, it goes a long way in helping the reader better understand their study.”
Toribio-Mateas believes that accessibility in the writing and presenting of studies is critical because it ensures that scientific knowledge can be understood by a wider audience, which is essential for advancing scientific discovery.
“Some of the studies that have impacted me most deeply are written in language that is not overly complicated or technical. Of course, terms specific to the field are used, but the researchers make it easy for the reader to access the meaning of their work,” he explains.
“Learning should be for everyone, not for people in ivory towers. And you shouldn’t have a PhD in bioinformatics to understand whether an intervention worked or not.”
How much can we rely on the peer review process?
Toribio-Mateas has worked in scholarly publishing in the past before he was challenged to publish as a scholar, so he knows “how tedious it can be” to get peer-reviewed work.
He says: “The scholarly peer-review model isn’t perfect, but it isn’t, so I trust journals to do at least some groundwork on my behalf. The more prestigious the journal – think Nature for example – the more accuracy and finesse I expect from quality assurance.”
