Enterprise needs are increasing dramatically to compete with high-tech companies that are taking over traditional verticals driven by superior innovation. Chief data officers (CDOs) are confronted for the first time with a future in which they can no longer be conservative and rely on the inertia of their big enterprise to succeed.
Full of data silos, large organisations do not currently have the technological capability to fight disruption from high-tech companies and startups. On one hand, the separation between their operational databases and data warehouses results in them conducting analytics over stale data, collected the day before in the best case.
On the other hand, they are resorting to using NoSQL databases to handle semi-structured data and combat scalability issues in their traditional SQL databases, resulting in even more data silos and new problems, such as the lack of a powerful query language and full data coherence, the so-called ACID properties.
With the need to overcome this lack of scalability in existing databases causing enterprises to waste the efforts of their engineers by up to four times, this is no longer a viable option. Yet adopting NoSQL alternatives is also wasting engineering efforts on compensating for their lack of powerful query capabilities, consistency and interoperability with SQL databases.
The huge waste of resources is driven by the fact that all databases are overprovisioned, so computational resources are dimensioned for the highest peak they can expect. For on-premise deployments, this practice means spending a lot of power and cooling in the datacentre without need.
The number of people in operation teams doing shifts in non-working hours is proportional to the number of computers active in the datacentre. In many businesses, the actual load during non-working hours can be an order of magnitude lower than during working hours, meaning organisations can be spending ten times more than needed 80 per cent of the time.
Since all traditional verticals are becoming commodity sectors, reducing the cost of the infrastructure and its engineering resources will become increasingly critical to enable companies to compete with the new high-tech companies and disruptive startups.
Downtime is also a major issue. In businesses with peaks that are hard to plan for, such as ecommerce and travel tech, existing databases often just collapse during extremely busy periods such as Black Fridays, sales and other seasonal flash crowds. Who has not suffered trying to buy something during Black Friday on the website of their favourite retailer? The financial consequences of that retailer losing sales due to unscheduled downtime or an unresponsive website can be huge and damaging.
High availability is a significant source of concern for CDOs. Current databases either offer primary back-up approaches that result in losing data if the primary fails or offer active-active approaches with serious performance issues.
The future of data management lies in converged data management solutions that combine many of the capabilities of different database managers – operational and analytical, SQL and NoSQL – while offering new features that are increasingly critical in today’s business landscape such as elasticity and active-active replication in a single database manager.
CDOs must take a bet on new database managers or succumb to the high-tech US companies and startups taking over traditional verticals
As their organisation faces more disruption, CDOs must take a bet on new database managers offering solutions to these challenges or otherwise succumb to the high-tech US companies and startups taking over traditional verticals with innovation they cannot compete with.
About the author: Dr Ricardo Jimenez-Peris is a former professor and researcher on ultra-scalable data management, and renowned speaker having been invited to give talks at many Silicon Valley companies such as Facebook, Twitter, Salesforce, … He is founder and chief executive of LeanXcale, a startup making a next-generation database manager aiming at filling many of the existing gaps.
