How to conduct a management experiment

Experiments are a great way to turn theory into practice. But in traditional in-company programmes, where teams of executives tend to work on a strategic project, which they present to their CEO at the end, there is rarely any follow-up. The implementation becomes someone else’s problem. 

In our Executive Education programmes here at London Business School, we have dropped the projects and introduced experiments: now the emphasis is on doing something, rather than just talking about it. 

Many companies have embraced management experiments, including Roche, If Insurance, Vodafone and Sberbank. 

Andrew MacLennan, one of our guest lecturers, explains: “Organisations try new things all the time; sometimes in a formal, planned way, and other times less so. But real experimentation is about more than trying out new things: it applies scientific methods to test new ideas robustly and enable leaders to draw firm conclusions about them.” 

Finding out what works – and what doesn’t

Experimentation is not just about discovering what works. It also reveals what doesn’t work. Organisations can sidestep enormous costs and risks simply by not implementing ideas that, however attractive on paper, will not create value. There are also huge gains to be had from experimentation that are less easy to quantify in terms of the bottom line. 

For one, it encourages a humble mindset. So often, when subjecting our assumptions and beliefs to a process of experimentation, we discover that they are simply wrong (usually they are wrong in proportion to the length of time we have clung fast to them).

Experiments can reveal that long-applied practices are ineffective, or much less effective than we imagine – but they can also show that they work for very different reasons than we assumed.

So, what’s the secret to designing and running a management experiment? “It begins with realising that experimentation is as much a mindset as a methodology. There are opportunities to experiment all around us, in all of our day jobs,” says MacLennan. 

Rob James, a programme director who has overseen experiments in many client companies, adds some practical advice: “You have to be very disciplined about narrowing the scope, to make sure you are clear what hypothesis your experiment is addressing.” 

The Vodafone Business Excellence programme 

One company that has wholeheartedly embraced this approach is Vodafone. Although mobile telecoms is a relatively young industry, the technology has evolved quickly and Vodafone operates in a very fast-moving world – this creates huge potential for experimentation. Now in its third year, the Vodafone Business Excellence programme brings together executives from all over the world for an intense week at London Business School. 

“We introduce the mindset and methods of experimentation, inviting participants to generate ideas to put to the test. Groups form organically around these ideas, and often people from very different regions and disciplines come together to explore common interests,” explains MacLennan.

The experimentation teams work together for several months, and are encouraged to start small, testing ‘safe’ ideas to generate insights quickly. This sounds simple but can be challenging; it’s tempting to default to conventional ‘big-project planning’ mode, where far longer is spent thinking and conducting research without actually changing anything.

Teams progressively build more ambitious and more robust experiments. MacLennan notes that “Sometimes, the ideas ultimately tested bear little resemblance to the original vision. But that is part of the magic of experimentation. It’s a highly flexible, iterative process – and if you knew the outcome in advance, there would be no need to experiment!”

Running a competition

In one seminal experiment, Vodafone wanted to generate ideas on monetising its existing dataset to enable best customer experience and identify new revenue streams. It decided to run a competition with the European Space Agency (ESA), an existing partner, in which start-ups won seed funding and practical support to accelerate product and market development. In parallel with the competition, the experimentation team launched an internal innovation programme which generated five new big-data ideas, two of which were taken to the board for consideration for implementation. 

Marco Galanti, Vodafone’s global account manager for the ESA, said the experimental findings showed that targeted engagement of external partners “enriches and diversifies the opportunities for Vodafone and can provide pathways to exploit data and ideas in commercial propositions.” Along the way, the competition generated a rich series of ideas, closely matched with company goals, and – as Vodafone’s David Gonzalez reported – “discovered all kinds of talent, which would normally be very hard to reach.”

Testing solutions with experiments

Another team used experimentation to tackle an existing business problem: as more Vodafone solutions became virtual or web-based, it became harder to demonstrate them physically to clients. The team saw great potential in making better use of digital product demos, but also numerous uncertainties, so they decided to use a methodology introduced to them at London Business School to articulate intended outcomes in detail and identifying how outcomes may be undermined by risks.

Because the sales cycle for some solutions can take many months, the team enacted a short experiment using digital demos to generate leading indicators of success. Although this initial test used a small sample, feedback was very encouraging, with 90% of respondents reporting that the digital demos increased their understanding of the value of Vodafone’s solutions. 

As significant as the results was the mindset change reported by the experimentation team. Key observations were that applying the experimentation process introduced at London Business School was extremely helpful in framing thinking and testing assumptions in a structured way to discover what really works. The experiment proved the value of real-world testing over trying to deliver ‘perfect’ solutions and showed that rapid, baby steps are ideal for generating fast feedback and building confidence in strategy.

The impact 

MacLennan notes that the benefits of experiments are multi-faceted: “One of the most rewarding things we see is participants using experimentation as a vehicle to engage their own teams. They typically share insights from the programme with them and challenge them to come up with their own ideas to test out. Some participants have successfully embedded experimentation as ‘the way we do things’ in their domains within Vodafone.”

James notes that you have to separate the business impact of an experiment from the learning and development impact. Companies are always on the lookout for tangible business benefits, but these are highly variable. Some experiments quickly fail, which can in fact be a great result if it means avoiding a costly and time-consuming implementation. Some demonstrate that an idea can succeed, but it then takes several evolutions to find the right way forward. And a few demonstrate immediate and significant financial benefits. 

For James, the learning impact should always be put front and centre: “we always ask people to identify their objectives, individually and within their teams, before starting the experiment. This helps to reinforce the real intent behind the experiment, namely that they become much savvy about how they can influence change within their company.” 

And because well-designed experiments are inexpensive and low-risk, all these outcomes are valuable. It’s all about creating insight. As William Blake put it, “The true method of knowledge is experiment.”