Nine dos and don'ts of big data

As AI becomes mainstream, big data is being applied to every aspect of business – but how do organisations extract value from it? How do you use it for key executive management decisions in the real world – and what are the pitfalls to avoid?

Don’t leave it to the data analysts

The days when ‘technology’could be left to the chief information officer are long gone. You can’t simply give the analytics team the information and ask them to tell you what the business should do next. Maybe, in the next iteration of AI, the analytics will be able to tell you what to sell, who to hire, how to build the website, how to market to your customers and even how to micro-manage the business – but they still won’t be able to tell you what business to be in and why. And, regardless how much data the CIO may have at her or his disposal, they may be no better than the guy at the bus stop when it comes to spotting external threats to the business and new opportunities, let alone how to respond to them. That means the CEO and senior management have to stay on top of what the data is telling them, lead the analytics team in addressing real business concerns, and work constantly to make sure all the data is being targeted at intelligent investment.

Don’t get lost in translation

Many companies now have teams of software engineers pulling the data in from ever-expanding sources, including social media, smartphones, sensors, payment systems and cameras. The question is what to do with it. How do you interpret it so you can put it to positive use, rather than just dig the database ever deeper? You need business analysts who can read the information and see how to use it to spot market opportunities, identify problems, come up with solutions and lead change. In other words, you need business ‘translators’ who may not be fully-fledged data scientists, but who are sufficiently proficient in analytics to take the numbers and know how to apply them for the benefit of the business. And there aren’t many around just yet, so it will likely mean identifying the people in the company who understand the global picture of the business (and who may also have a quantitative background) and getting them up to speed on the analytics side. You also need to invest in training them in terms of leadership skills, so they feel confident in championing change throughout the organisation.

Don’t drown in a sea of data

There is a natural urge to want to capture every atom of the business’s legacy data, then cast around wandering what to do it. Resist the urge – identify how the data was acquired (is it from sales, operations, social media, ‘open’ sources or elsewhere?), have specific business applications in mind, and make sure your data strategy connects directly to the analytics. This means, above all, avoiding the temptation to build complex models from the get-go. Instead of trying to capture all historic information, decide the business priorities – or even one overriding priority – then identify what data is likely to be useful in addressing it, and add to it gradually. This will enable you to develop a sound process and practice – in effect, good data governance – which can then be augmented and refined by linking new and different datasets to yield new insights.

Know where to start

Following on from this, begin by identifying the most promising sources of value to the business. That means developing an organic view of opportunities and pinpointing those components of the value chain with the greatest potential. Is it inventory optimisation or product development? The next step is to identify as many use cases as possible and look at how you can apply new data and techniques to them to generate new insights. Decide your order of priority, based on potential financial impact, suitability to the business, and likely speed of implementation.

Democratise that data

One of the most common reasons for lack of uptake of data analytics is that the people who can put it to best use lack meaningful access to it. Avoiding this pitfall requires a three-step strategy. First, it means making sure that the data is accessible to as many people as possible, so dispense with any organisational hierarchies that may impede access. Second, you need to drive consensus on the validity of the data, so there is agreement that it is, in effect, a single source of ‘truth’ for the business. Third, building on equal access, you need to develop an egalitarian culture whereby everyone is allowed to ‘play’ with the data without fear or favour and try to generate new ideas – or kill established company nostrums that are no longer fit for purpose – in such a way that their ideas, however counterintuitive,  will be given equal airtime when it comes to decision making.

Be ready to trade secrets

Traditionally, companies of all shapes and sizes guarded all market and business information closely as a way to retain know-how and competitive edge. This is no longer necessarily the case and there are plenty of industries where sharing the data, for example as part of an industry bloc or cross-sector group, increases the comprehensiveness of the data and enables individual businesses, who may operate in different segments of the market, to enhance their offerings and create greater value. That doesn’t mean, of course, that you must always tell all, but you need to know what other data sources are out there that may be useful to you and whether, by pooling, you can create a whole that is larger than the sum of its parts.

Cultivate a culture

To get the most from the data in terms of generating ideas that will lead to new innovations, you need to foster a test-and-learn culture throughout the organisation: senior management sets out the vision, employees are encouraged to identify where the opportunities are and develop proofs of concept, then the data is used to analyse the results. This is a learning process, so it also means a no-blame culture; you are actively looking for new and possibly highly counterintuitive insights and testing them. If the hypothesis stands up, you move on to look at implementation and, if it doesn’t work, you treat the ‘mistake’ as a valuable lesson for the next iteration.

Don’t be afraid to kill your darlings

Another common failing is to try to make the data fit a particular business agenda or pre-existing idea. This is a natural tendency – humans take a long time to learn what works and what doesn’t and, once they have worked out what does, find it hard to let go of a winning formula. That’s where the machines come in. Data analytics will tell you with amazing speed when something isn’t working, whether it’s an actual business situation or a test-case scenario. So, no matter how attached you are to your idea of what the next big product launch should look like, be prepared to walk away as soon as the data tells you different. Data can do many things but – if you’ve got the analytics right – it can’t lie.

Don’t doubt the doubters

There’s a fundamental paradox in being able to use data analytics successfully as a leader. One the one hand, people who can be trusted to make accurate predictions about future events need to doubt themselves and the information on which they base their forecasts constantly. As a data analyst, you live in a world where you are constantly absorbing new information, very quickly. Is the situation as it appeared yesterday the same today? Is what I was saying with complete certainty really true? You need to be able to reassess things continuously and let go of your original belief. Even more than that, you need to be aware of the assumptions underlying your original belief. So, the data analyst’s mindset is relentlessly  self-critical and questioning. But this, of course, is at complete odds with the characteristics required of a leader or manager. You need to exude confidence and authority. The trick is to be able to balance the two; to recognise that nothing is guaranteed in terms of future events but still be able to motivate your people to implement the strategy you have decided. It’s an intrinsically complex communication challenge – you need to be able to see the world bottom-up, from the perspective of the data analyst, but communicate top-down because you are driving the process of change. And that is something the machines will never be able to do.