Creating a culture of Experimentation: Let Data lead the way

Management jargon like ‘Extreme Experimentation’, ‘Fail Fast’ have been around for some time now. Much of this thinking and consequently, success has come from the software industry. But once you step outside the Silicon Valley, you will find hard pressed to find successful instances of experimentation translating to actual shareholder value. In my years of working with Fortune 500 leaders, I have found a stubborn chasm between desire and execution – one that goes beyond systems and processes.

It is clear that this is above all, a challenge of cultural transformation – one that moves away from trusting expert judgment to a more incremental approach informed through faster customer feedback. This is a journey that, if executed well, will create not just a data and system architecture but also impact the organization structure and KPIs. In other words, this should trigger a wholesale cultural change. And like all transformation ideas, there need to be a series of initiatives.

1. Invest in a Cross-functional Design of Experiments Team: Most organizations have digital platforms. And many of them run basic experiments limited to testing out multiple website changes (e.g. A/B testing). This thinking needs to expand beyond these ‘cosmetics’ to experiment with deeper changes – e.g. pricing, product offering changes. Such initiatives require changes not just to the website – here are just a couple of examples:
   1. Product offering experiments: This will require a change in how the product structures are created – instead of individual SKU Bills of Material (BOMs), you will need to create option BOMs, with dynamic optional combinations
   2. Pricing experiments: This will require a change in pricing methods – instead of overall product price, you will need to set up a line structure that prices individual feature combinations

This will require a cross-functional team with a mandate to build this capability. An initial manifesto could look something like this:

• Design multiple experiments in line with business goals. This requires a heavy dose of Data Science (see below)
• Implement process changes – from changing say, how pricing gets done to how product changes can be deployed across physical and digital channels
• Design the right set of KPIs to track not just the lift from individual experiments, but also to track the impact of these changes through implementation
• Orchestrate the IT infrastructure to deploy these experiments

In our opinion, this is best owned and orchestrated through the Marketing Strategy or Corporate Strategy function. A leading home improvement retailer invested in this capability within the CFO organization – and used this function to drive experiments across channels – from in-store experiments (e.g. store-level promotions) to experiment with omni-channel implementation scenarios (e.g. Buy Online Pick up in Store).

1. Not all Learning needs to come from field experiments: The proverbial data haystack has many needles. To begin with, the historical product and pricing changes can provide signals on the customer stated preferences – e. the traditional lift from these changes. Even more, data provides the opportunity to tease out revealed preferences, essentially signals that customers communicate through indirect mechanisms – e.g. relative preference for specific attributes (e.g. storage capacity vs. processing power) expressed through features (response to memory upgrades vs. RAM upgrades). Discrete Choice Models often help understand the value customers assign to product attributes (i.e. decompose a product price into the individual attributes). This could be a good starting point to understand price-value of product features. And then abstract out the features to attributes – which can then be imputed back to new features. A B2B tech manufacturer used this strategy to understand the price-value at a feature level of its Server product portfolio. This formed the basis of option-pricing for the next generation of Server products. Needless to say, this was the only viable approach given that it was not possible to run field pricing tests in a highly competitive market.
2. Build Data Science Capability to extract value from data: It is clear that both of the above will require Data Science capability. And this capability becomes even more important given multiple challenges around not just the quality, but often surprisingly, the quantity of data.
   • Data Quality: Experiment data is more often than not, notoriously noisy. There are often multiple factors at play – both external (e.g. competitor launch, market dynamics) and internal (e.g. marketing promotion calendars, Supply chain considerations around availability etc.). Solving for these truly requires a blend of Art and Science:
     • Experiment design: Design the right test/control methods and the right measurement approach. From A/B testing to sophisticated methods like MAB (Multi-Armed Bandit)
     • Attribution modeling: Deploy Machine Learning models to tease out the attribution of the lift to a specific set of experiments from all other factors.
   • Data Quantity and Context: Most companies do not have the luxury of massive data sets the way Facebook, Amazon or Google do. More often than not, experiments need to deal with sparse datasets (e.g. small samples, poor response rates). And in some cases, there is not enough information in the incoming data to be able to easily execute experiments. For instance, without any prior information about the visitor to a website, how do you decide the right page to serve in an A/B test?

As companies across industries try to improve engagement with their end consumers, building a Design and Execution of Experiments capability is no longer nice to have restricted to the company’s website changes. We believe that the time is right for investing in building the right Organization, Data and Technology eco-system that can create, launch and sustain this process across the enterprise – Product Design and Launch and Pricing are two areas where there should be an immediate opportunity to invest in building this capability.