New York City has appointed its first director of analytics. His job is to use data scientists to unravel the Big Apple’s trickiest problems. Mayor Michael Bloomberg’s pick was Mike Flowers, a lawyer who made his name in Iraq using data to identify terrorist hotspots.
One of the analytics director’s first missions was to find a way to help the police discover illegal housing conversions. Houses meant for four were being chopped up to house twelve. The result was fire hazards, rats and nuisance to neighbours.
Mayor Bloomberg wanted a crackdown. The old approach was to rely on inspectors’ hunches and tip-offs. But the new team opted to rely on big data. They pulled up every possible source of data related to property in New York and looked for patterns.
Rather than relying on guesswork, they let the data do the talking. For example, the frequency of brickwork maintenance was found to have a correlation. So were rat sightings, fire incidents, tax payments and a ton of other things.
The Flowers team could then rank the 20,000 leads they had by their probability of leading to an illegal conversion. Inspectors could concentrate on the most likely violators and the big data approach raised the success ratio from 13 per cent to 70 per cent. This is the power of big data.
Big data analytics is one of the most potent, but potentially problematic, techniques in business today
So how can your firm get the most from big data? The trick is to turn your company into a place where big data lies at the heart of every decision.
One of the highest hurdles is knowing what the data is likely to tell you and what it can’t. After all, until you’ve examined the data, you can’t be sure what treasures are hidden within. But this is big data – you can’t trawl through it by hand.
It is a headache, admits Dr Michael Wu, Chief Data Scientist at Lithium, a big data analytics house focused on social media. “The traditional approach is to ask a question and then to collect data to answer it,” he says. “Big data does the reverse. You gather data without a purpose. So my best advice is to ask questions, then let your data scientists conduct exploratory analysis.
“There is no fixed method to this. They play with the data and get a sense of what it can tell you. Then the analyst may conclude that an answer can be found in the data or tell you that they need more data. What you do not do is simply let the data scientist look for correlations. What use are correlations if they don’t tell you anything commercially useful?”
At no stage must you become a slave to the data. Eddie Short, head of business intelligence at KPMG, says humans remain essential to the process. “Hunches remain a vital part of the process. You look at the data and make a leap,” he says.
To work with big data, you’ll need to deal with data scientists and decision scientists. If you feel intimidated, well, you probably should be. Dhiraj Rajaram, founder of data analytics firm Mu Sigma, which works for 100 of Fortune 500 companies, describes his employees as “like Tony Stark from the Iron Man films – they need to combine the best human elements with the optimum bionic environment”.
It is so hard to find these supermen that Mr Rajaram prefers to train his own data scientists: “Our training takes at least three years. We have been described as a cross between a lab in Silicon Valley, a consulting company and a university.” He’s not exaggerating – Mu Sigma has its own university, created to provide an intense education in business, data and management, unrivalled by traditional establishments.
These challenges mean big data analytics is one of the most potent, but potentially problematic, techniques in business today. And there are sceptics who question its validity and relevance to ordinary business.
Clive Humby, inventor of the Tesco Clubcard and a man venerated by the retail industry, warns: “It can be a massive red herring. My energy company has installed a smart meter at home so it can take readings every second instead of every three months. That’s an increase in data of 7.5 million times. But so what? Are they going to spot that my fridge is about to blow up? People get delusional about big data.”
Fortunately the library of success stories is seemingly endless. Whether it’s SAS using big data to identify fraud in the insurance industry or IBM’s Deep Thunder weather analytics package helping farmers know when to irrigate their crops, the benefits are now being measured in billions.
Big data is intellectually challenging and, on occasions, baffling. But there’s no doubt it is going to change the world.
