If someone had mentioned the phrase “big data” only a couple of years ago, few people would have understood what they meant. It’s now crept into management-speak, although not all lay people understand it. They may eventually have good reason to be very grateful for it.
If one thing has changed in the last 12 months, then it’s awareness of the idea internationally. Duncan Ross, director of data science for Teradata, points to the first O’Reilly’s Strata conference in the United States, at which 98 per cent of the people were North American. “Now that’s down to about 50 per cent and the first Strata London event is happening in October,” he says.
Huge amounts of people in different territories are starting to realise they have a lot of under-deployed data at their disposal, if they knew how to utilise it. Nick Patience, director of product strategy at predictive and analytical software developer Recommind, says that a number of markets already depend on huge amounts of data moving. With legal documents, for example, huge amounts are sifted and presented to barristers by non-specialists.
“Despite potentially not knowing a great deal about the case, they would make an initial appraisal and present a manageable selection of information to senior counsel for review and subsequently to the court,” he says. “The problems are obvious – the time and cost implications are dramatic, human error is inherent and the chances of missing critical documents are high.”
People are starting to realise they have a lot of under-deployed data at their disposal
Retail also has a huge amount of data coming in, from disparate sources. “By being able to group information by concepts and context, such information access technology also enables people to find information that they may not have even thought about looking for; if they had to rely on exact matches of keywords, such information would remain hidden from them,” says Mr Patience.
The problem is expanding rather than contracting and for many companies there is no obvious way around it. A lot of the attention recently has been on data from social media, as retailers and other businesses suddenly have to assimilate what’s being said about them on Facebook, Twitter, LinkedIn and elsewhere, as well as all the other inputs. This is not restricted to retail; the ability to detect every nuance in the market and predict where it’s going to go and what to do about it can apply to a great many areas.
Mr Ross points to yet another source of huge information, even larger than social media. “Machine data is even bigger,” he says. “One of our clients is in the automotive industry, Daimler, and they say a modern Mercedes has 90 processors in it.” Each of these processors can collect data.
The future is starting to look very exciting indeed, at least partly because of the possible application of big data. It’s necessary to hedge statements about the future with “starting to”, “possible” and other qualifiers, but have a look at what Jake Porway and the team at Datakind.org are doing.
They have a theory that, if the best brains on the planet are engaged using data, 2 per cent can be added to a company’s profit, as Mr Ross explains. “They point out that you could gather enough data for an early indication of drought,” he says. “At the moment, the first indication the UN gets is when 100,000 people turn up at a country’s border needing relief. If you could use big data to predict the drought and do something to prevent the crisis…” Now there’s a thought.
Big data is often described as a problem or something vaguely “big-brotherish”. Deployed correctly it could be a big opportunity.
