When top OpenAI investor Microsoft unleashed a ChatGPT-infused Bing search on the world, it wasn’t long before it ran haywire, comparing journalists to Hitler and gaslighting its users. Of course, these deranged tirades were not really an AI going rogue or anything of that sci-fi ilk; the tool is a probabilistic program that, having scraped the internet and all the junk on it as its source, returns answers that it thinks are likely to be correct. The whole episode did, however, highlight the need for a considered approach to AI deployments, especially when they’re public-facing. Above all, it demonstrated that AI needs precise use cases informed by good, up-to-date data, and guardrails to ensure it’s on the right track.
“Microsoft, Bing, OpenAI and ChatGPT have done the world a favour,” comments EMEA field CTO at Databricks, Dael Williamson, “because on the one hand, they’ve shown us the art of the possible – but they’ve also shown us the respect we have to give to training data.”
As amusing as the headline-grabbing antics of abusive chatbots might be, what will really be front of mind for most businesses as they seek to leverage artificial intelligence is how it can help them work smarter and more efficiently. For example, Williamson saw the power of AI in his previous career in proteomics, with simulations for drug discovery that used to take 25 days now taking just a few hours. And across all kinds of industries, businesses are using AI in ways that might not make headlines but are helping them provide better solutions and services. Whether we’re aware of it or not, many of us interact with AI on a daily basis – from the navigation tools that plot courses for Uber to Amazon’s recommendation engines.
“It all starts with data,” says Williamson. “Before businesses can create AI models that actually deliver value, they need to ensure the source data they’re building from is accurate, complete, timely and fair.”
Before businesses can create AI models that actually deliver value, they need to ensure the source data they’re building from is accurate, complete, timely and fair
While the transformational potential of AI really is enormous, and may change the world in unforeseen ways, most businesses will be seeking to use AI to improve their business processes. Decision-makers have certainly noted the potential. In a recent MIT and Databricks technology review survey, CIOs estimated that AI spending over the next three years will increase in security by 101%, data governance by 85% and new data and AI platforms by 69%. To ensure that it’s AI driving the efficiencies rather than a tail wagging the dog situation where the technology is in search of a problem, businesses will need to first identify the use cases that would actually benefit from these rollouts and, crucially, ensure their data is in order.
Artificial intelligence is only as good as the data that feeds it. Unfortunately for weary data scientists, who spend an astonishing 80% of their time searching for the stuff, most organisations are sitting on incredible treasure troves of data, but it’s scattered and hard to find. This is unsurprisingly a barrier to using it effectively, let alone for building effective AI models.
If not hidden down the proverbial sofa, this data is siloed, disconnected and sorted in different databases and formats. In short, staff in department A may not know about the data in department B, and even if they do, they’d struggle to connect it. To get around this, businesses need to unify their data environment. “We call it the ‘lakehouse’ concept – think of it as the production and distribution of data and models,” says Williamson of this open architecture proposal, “where it covers all the value units you’d typically want to have your data go through.”
By unifying all of your business data and applying governance to it, the data becomes much more observable, making it easier to maintain and manage data integrity. With this data organised, accessible and standardised, businesses can pick and choose which data sets are the most appropriate for the model they’re building, whether that’s large language models, computational models, deep or machine learning, and then build the applications on top of that.
“That’s the technology, but the hard bit is change management and trust,” says Williamson. No wonder; those aforementioned fearful headlines often frame artificial intelligence as a uniquely disruptive force that’s set to play havoc with society as we know it, shredding the social contract and discarding its hapless victims. That’s not the case at all – most businesses will simply be attempting to drive efficiencies, using automation to sluice away the most dreary manual tasks, which often don’t scale without a little technological assistance.
Take the humble elevator, for example, notes Williamson. For many years, lifts were staffed by an attendant, greeting users and pulling the levers. It took a long while before people trusted these newfangled automated contraptions enough to press a button, but now it’s as intuitive as crossing the road. Change can take time, and that’s why it’s so vital organisations manage it carefully, rolling out AI deployments with openness and transparency. At the very least, they should work with technology that operates a sort of “glass box” model – as opposed to an opaque “black box” with all the inner workings hidden away – so that users understand exactly what is going on and why.
“If you translate it to people, process and technology, technology needs to be simplified and made uncomplicated, while process is the real ‘unlock’ to create efficiency, build trust and transparency through that,” says Williamson.
Today, it’s really only the dawn of the AI era, but soon enough it’ll become evident that people will largely interact with machines as co-pilots, much the same as other transformative technologies like the printing press and the internet. Communicating this to users is key: “We need transparency, open data and trust,” Williamson says, with projects that demonstrate their value to staff outside of data science functions. “The few enable the many – that’s the bottom-up way of thinking about it. There also has to be a top-down commitment from the C-suite and all business leaders to work together; a partnership between those two groups, where everyone is rowing in the same direction.”
To find out more, visit www.databricks.com
