When Google wanted to discover the “stickiest” shade blue for a button, it ran a multivariate test with 41 variations. Forget creativity – data provided a concrete answer. It’s a classic example of how multivariate testing, or MVT, can be used to solve problems.
Whatever you want to know, multivariate testing can offer insights. BGL Group, owner of ComparetheMarket.com and other online brands, recently faced a challenge when implementing cumbersome new GDPR and Insurance Distribution Directive legislation.
“We wanted to implement it in a customer-friendly way,” says Laura Mullaney, associate director of BGL Group. “Ahead of deadlines for implementation we were able to use multivariate testing for proposed customer journey designs with real customers using the website to determine the best approach. This enabled us to test customer response to some significant changes and understand the best approach to take for our customers, while adhering to our regulatory responsibilities.”
The company uses the same MVT logic to test all sorts of hunches. “We can test a journey or design even down to a demographic for insight on what works best to generate value. This can feed in to really significant initiatives, from rebranding through to development of adaptive or mobile-first design journeys,” Ms Mullaney reveals.
Datacentric thinking must be met with creative decision-making
But here’s the kicker. Datacentric thinking is a dominant force in shaping the customer experience. It is so powerful that less experienced brands are at risk of becoming dependent on it.
Gilles Boisselet, creative strategy officer of UNIT9, a consultancy working with brands such as the BBC, Huawei and Nike, says ‘data versus humans’ is a classic conflict. “Software salespeople will try to convince you that data testing and creative conceptualisation are the same thing. And companies looking to cut costs may think software can replace the people in charge of creative decision-making. But the reality is that data testing and creative decision-making are two different beasts.”
A lack of knowledge of the merits and limits of data methods like multivariate testing are partly why, according to Gartner, 85 per cent of gig data projects end in failure. In the worst case scenario, they forfeit creativity and become slaves to the data.
“Data testing more or less works from a mathematical base using dichotomy and statistical averages. So it trends towards value. But this just promotes uniformity and means it will become a struggle to differentiate yourself,” says Mr Boisselet.
“Genuinely creative people who can crack originality still need to be part of the process. Just look at how our new era of auto-completion is making our texts, emails and Google searches become homogenised. True creatives, like hip hop artists, know that you get to rule culture by breaking the rules, not mathematically adhering to them.”
Relying on multivariate testing and the like can take too long
There’s a second danger. The volume of data collected by companies can be overwhelming. Companies can be paralysed by the possibilities, and the chance to explore hundreds of avenues with a technique like multivariate testing can result in years of procrastination.
“I worked with a FTSE 100 company on a project that they thought would take two years,” says Payal Jain, managing director of JCURV, an agile working consultancy in the data sector. “But you can’t wait that long. In three months the market may have changed. The truth is that it doesn’t matter whether you have AI, robotics, or machine-learning, you still need to humanise the data process.”
Ms Jain’s solution to the logjam is to create agile teams. These self-sufficient ensembles are able to come up with their own ideas, build models, and test them rapidly: “You may have 100 things on your list to look at. We say, break it down. Find the most important five. Then let an agile team take them on. This way you can see results as fast as two weeks.”
This approach to handling data also promotes creativity, without diminishing the contribution of data. “Give teams autonomy,” advises Ms Jain. “Those closest to the problem often know best. Another FTSE 100 client wanted to spend millions on a machine-learning and AI solution. But the team working on the job said we don’t need to spend that. We have a batch process that does 80 per cent of the job. It worked and they save a huge amount of money. The truth is that you can spend a fortune on buying huge servers to store data, on buying the best data tools, but the money will be wasted unless you humanise your processes.”
Multivariate testing is good, but it must be paired with human creativity
As a happy dividend, giving data teams control over what they test and how leads to happier teams. “There is a war for talent,” says Ms Jain. “Data scientists are paid a lot. When leaders are prepared to embrace cultural change, and return power back to teams working closest to the data, they generate more loyalty and engagement, in addition to better results and top-line growth.”
The old cliché of “data is the new oil” was always a little tricky in practice. Oil is a commodity with a scientific refinement process. Data is altogether more slippery. Asking questions, drawing insights, and knowing how to react to evidence, is still a profoundly human endeavour.
In the end, there will always be a place for a creative leap. BGL Group increased market share by 76 per cent in a year when it gambled on a new creative direction with its bold and slightly bonkers Compare the Meerkat ad campaign. The talking aristocratic meerkat Aleksandr Orlov is still the face of BGL’s most valuable brand. It’s a reminder that although multivariate testing is a powerful force for good, there is still space for old fashioned human ingenuity.
