As the amount of data has exploded in recent years, executives are under tremendous pressure to make good use of it all. However, they are often hampered in their efforts and turn to DataOps to use data effectively, but face challenges along the way.
Ninety-seven percent of surveyed companies said they are investing in data initiatives, but only 47.4% are competing for data and analytics, according to NewVantage Partners’ 2022 Data and AI Executive Survey. Only 39.7% managed data as a business asset, and only 26.5% reported having a data-driven organization.
In response to such underperformance, business leaders leverage the principles and practices within the DataOps discipline to help use data effectively to make decisions, uncover insights, and drive automation and intelligence initiatives.
Out of 403 technical and business data professionals, 90% of respondents said they plan to make moderate to large investments in DataOps in the coming year, according to the April 2022 State of DataOps report from the Enterprise Strategy Group (ESG), a study firm and division by TechTarget.
What is DataOps?
DataOps – short for data operations — is a collection of practices, principles, technologies and staffing intended to create efficient handling of data.
“The heart and soul of DataOps is orchestration. Moving, processing and enriching data as it moves through a pipeline requires a complex task workflow with numerous dependencies,” said Neel Shapur, head of data advisory and architecture at services company Genpact.
This discipline borrows concepts from the world of software development, such as B. Agile development principles and the iterative and collaborative approach known as DevOps. It aims to help enterprise data stewards securely deliver the right data to the right users at the right time by breaking down data silos, adding automation and enforcing governance rules.
“DataOps provides the ability to quickly and reliably find, trust, and understand data,” said JP Romero, data management practice lead at Kalypso, a consulting firm and IT service management company.
DataOps Challenges
According to the ESG report, organizations are using DataOps to address several challenges related to the use of their data.
Data leaders cited challenges related to ensuring regulatory compliance, adhering to governance standards, and timely access to new data as key drivers for DataOps adoption.
Still, enterprise data executives, researchers, and consultants said organizations also face numerous challenges in successfully embracing and maturing the DataOps discipline.
Key DataOps challenges in implementing, using, and scaling DataOps across the enterprise include the following:
1. Uncertainty about what DataOps entails
The concept of DataOps has been around for almost a decade now, but – like its DevOps cousin – there is no single formula or all-encompassing guide for exactly what it entails. In fact, technology vendors typically have their own, slightly unique, perspective on the needs of the discipline, said Jay Limburn, IBM’s vice president of product management for data and AI.
Consequently, data stewards must identify emerging best practices and the approaches that are beginning to standardize in order to create a DataOps program that works for their organization, said Mike Hendrickson, vice president of Tech & Dev Products at Skillsoft, a maker of Tutorials management system software and content.
Data leaders should also be agile so they are ready to adapt as the discipline matures and new enabling technologies emerge.
“DataOps is still early in the maturity cycle. Expect many changes and advancements from both tool and process perspectives,” said Shapur.
2. An inability to find where and how to start
The amount of data is staggering. According to research firm IDC, in 2020 the world created or replicated 64.2 zettabytes of data. It is estimated that global data creation and replication will grow at a compound annual rate of 23% through 2025.
While no organization can even remotely handle that volume, most still have more than they can handle. According to experts, it can be difficult for them to know where to start applying DataOps principles and how best to mature them.
“Define what success means to you” is where businesses should start, said Ramesh Vishwanathan, practice consulting director at IT service management company TEKsystems. It’s the same advice he gives to all of his client companies.
Organizations should identify areas where they can experiment and practice a DataOps discipline before expanding it to include all data and data usage within the organization. In other words, take the Minimum Viable Product approach and grow from there.
“Focus on a DataOps MVP that addresses a limited set of data use cases from fundamentals to value delivery,” said Hector Rueda, data science technical manager at Kalypso. “Once the Minimal Viable DataOps product is proven, scale out by expanding the scope.”
Organizations should also find ways to measure the effectiveness of their DataOps programs.
“This will help you know if you are going in the right direction,” said Vishwanathan.
3. Missing data basis
DataOps brings people, process and technology together to orchestrate the effective, efficient and secure flow of data within an organization. To do this, organizations must have key components in all three areas.
More specifically, experts said that organizations looking to leverage DataOps should understand:
what databases they have and the quality of that data;
how that data currently flows through the company;
whether data silos still exist and how to eliminate them;
how the company intends to use data;
what data governance exists; and
the technology components and talents they have to support all of these elements.
“DataOps requires a combination of technical investments, organizational restructuring and change management. There are technical, operational, human and cultural barriers,” said Shapur.
Yet many organizations lack some or all of these elements. They also lack a data culture as well as a data strategy.
The lack of these fundamentals can hold back attempts to successfully implement a DataOps discipline in an organization.
To counteract this, data stewards should focus on building the basic elements of a data program so they have everything they need to adopt DataOps.
Businesses need to put data and the delivery of data facts at the center, rather than treating it as an “afterthought” within the software development cycle, said Dan Sutherland, senior director in the technology consulting practice at Protiviti.
Businesses need to pay more attention to all elements of the data lifecycle, including design and development of the data pipeline, storage, data modeling and consumption patterns, Sutherland said.
In addition, they should prioritize:
replace old data tech stacks with modern ones that provide full visibility into the data pipeline;
invest in data literacy training company-wide; and
train their data teams so they are prepared to work in this new environment.
Creating a solid data strategy that highlights the benefits of bridging the gap between where it is now and where the organization must use data to meet critical goals is critical.
“Identify leaders who can support the program,” Romero said. “Reach out to the people who will be positively impacted by the adoption of DataOps. Invest in a data literacy program that can foster a healthy data culture.”
4. Lack of buy-in from leadership
Another DataOps challenge for organizations is persuading leadership to support the DataOps effort. The more mature an organization’s data culture is, the easier this is to achieve.
“It’s easy for people who use the data on a daily basis to say, ‘We need this,’ but sometimes leadership doesn’t always see the need. They don’t see the point of doing DataOps,” Romero said.
Others echoed this comment, noting that leaders in organizations without a mature data culture or gains from data-driven insights are often reluctant to support investments in DataOps.
It’s easy for people who use the data on a daily basis to say, “We need this,” but leadership sometimes doesn’t always see the need. They see no point in doing DataOps.
JP RomeroHead of Data Management Practice, Kalypso
Data leaders can overcome this lack of support by championing “the development of a data strategy and highlighting the benefits of bridging the gap between data strategy location and need,” Romero said.
“Companies are beginning to realize that data needs to be viewed as a strategic asset,” he said. “Those who align their data efforts with their organization’s strategic imperatives will find it easier to get value from their data and fund their data programs.”
5. Problems dealing with change
DataOps requires people to fundamentally change the way they work. That kind of change won’t happen overnight, Shapur said.
“Getting employees to adopt new practices and agile ways of working isn’t easy,” he said. “Often they lack the skills and time to learn new skills. DataOps also requires a new way of thinking about software practices.”
As such, data stewards should incorporate change management principles into their DataOps plans to ensure they can successfully engage people in new ways of working and thinking about data.
