How to Navigate Product Enhancements Through Data

You’ve probably heard it countless times: we’re in the “consumer age” – the consumer has a choice. This realization has compelled organizations across all industries to take a hard look at the experience they deliver.

Many companies have realized that with intuitive products and innovative features, they can significantly improve customer experience and, in turn, financial results. But how do you know which features are really effective? Finding product improvements that move the needle in a meaningful way requires a deep understanding of what users are doing.

Let’s see how to do this.

hypothesis formation

Before any experiment, you need a hypothesis – a testable/falsifiable belief about how or why users perform certain actions. Experiments must begin with data analysis to figure out the problem you are trying to solve and identify the potential solutions to that problem based on expected impact, scope, and effort.

Besides, a hypothesis is specific to a particular time and place—it is concrete. Shocks to the economy (e.g. our recent inflation spikes), social changes, technological changes and more can lead to changes in user interaction. These can introduce lurking variables that invalidate a hypothesis, so one must remain vigilant when assessing long-term and short-term variables that affect user behavior (and consequently hypotheses).

Given the complexity of isolating a variable and separating signal from noise, one must perform an ongoing cycle of data analysis, brainstorming, design, and experimentation.

Data analysis and modeling help identify key drivers of consumer behavior. But to understand why Acting consumers requires a great data team to understand relevant emotions and subconscious behaviors by talking directly to consumers through user research to understand what they think about their own experiences.

By properly identifying the root causes of consumer behavior and the reasons why they influence action, you are armed with information to form an informed hypothesis and deliver experiences that can (and must!) be continuously improved through experimentation.

Experiment, validate and falsify

Given the cross-functional collaboration inherent in experimentation (i.e., data science, design, user research, engineering, and product must all work together in lockstep on changes), teams must continually refine processes to ensure experiments are prioritized with expected impact. And since the cost of experimentation can be high, you should also make sure that everything is done carefully, and that every step is carefully planned and reviewed.

Finally, when you test, you know that sometimes a wrench is thrown in and you need to retest, sometimes even going back to the drawing board to try different product changes and personalizations until a hypothesis is clearly confirmed or disproved. Luckily, there’s a lot to be learned from this experience if you catch it real quick. For example, an experiment we conducted resulted in a 30% drop in patient sign-up rates on our platform – our hypothesis was not falsified, we simply moved the call-to-action too far down our homepage (which actually showed how critical the login workflow was).

In addition, depending on the sample size, sometimes a hefty grain of salt has to be consumed. For example, you might have a sample size of 10,000 people but make decisions for 10 million while trying to generalize for future consumers (aka the entire population of the United States).

It’s important for a data science team to hold each other accountable by disseminating updates widely across the organization and helping ensure objectivity by getting outside opinions when you need to prioritize or make difficult decisions. Be fearless in removing changes and features that don’t improve metrics.

For example, if you find something isn’t working, running a classic A/B test with two separate workflows can help you figure out which option is delivering the greater success, whether it’s click-through rates, signups, or sales.

I would recommend methodically testing an aspect before making any major changes so you can be as confident as possible that there aren’t any lurking variables.

To make it count

Developing a culture of continuous experimentation is not easy as the process can be expensive given the extensive resource coordination mentioned above. And getting results takes time, so make every hypothesis as valid as possible and every experiment carefully thought out.

But through regular collaboration, the entire team can become more than the sum of its parts, helping each other identify and fill collective blind spots while contributing to each other’s work. Through this partnership and constant evolution, companies can be assured that they are meeting the consumer expectations needed to succeed.

By applying some of the pointers above, we hope you’re well placed to use a data-driven approach to product improvement – good luck!

About the author

Yohann Smadja is VP of Data Science at cedar. With 12 years of experience in the field, Yohann leads the Data Science team responsible for understanding the vast amount of data available to realize the company’s vision.

Sign up for the free insideBIGDATA newsletter.

Follow us on Twitter: https://twitter.com/InsideBigData1

Visit us on LinkedIn: https://www.linkedin.com/company/insidebigdata/

Join us on Facebook: https://www.facebook.com/insideBIGDATANOW