Starting the DataOps Journey

In the previous blog, “Save Time and Reduce Data Errors with DataOps,” we examined how putting DataOps in place with the right monitoring and alerting can resolve or eliminate common data errors without any impact to customers. Here, we’ll explore the implementation of a DataOps initiative. 

Data operations requires embracing next-generation systems and infrastructure, fully leveraging their value toward the objective of digital transformation. First, there needs to be an understanding of the operational maturity of the organization, as well as any potential barriers to changing current data practices. You can get started quickly by simply having a few key individuals adopt DataOps practices. In fact, you probably have some of these change agents in your organization today. You may work with a data scientist who has the technical skills to collect data from just about any internal or external source, consolidate it into a personal data set, and deliver ground-breaking insights. You may also have an experienced data architect in the organization who challenges the status quo, capitalizes on innovations, and uses influence to transcend silos and achieve enterprise results. 

Implementing DataOps through a Data Operations Center

To effectively expand DataOps across the enterprise, you need to move beyond the individual raw talent of a few subject matter experts and adopt a change management system to establish a repeatable process. In short, you need a way to teach employees how to adopt and apply DataOps best practices again and again. To do that, you’ll need to formulate a plan, develop an implementation strategy, and formalize a team to deliver real value to the enterprise. 

One of the ways to start the DataOps team is to build on a competency center that already exists in the enterprise, for example, transforming a security competency center into a data operations center (DOC). This center of excellence would influence and direct the data sets to be developed and rolled out.   

The DOC is not responsible for figuring out who is doing the integration, operating on the data, or analyzing the data and why. It is not responsible for transforming the business and doesn’t oversee execution. Rather, the DOC’s role is to determine best practices and enable all of these activities to happen with the lowest cost and friction so that those who work with data (data scientists, business users, and/or data engineers) are empowered across the enterprise.

The DOC in Action 

Let’s look at an example use case where the DOC would be beneficial. Let’s say you’re operating a large IoT company with equipment that is located globally around the world, and you have sensors in all of them. You know these machines are extremely expensive and not always mass-produced, so when a part fails, there is a significant impact on overall business and revenue. 

One of the obvious use cases for DataOps in this scenario is preventive maintenance. This use case has been well studied and applied in the past. Assuming a wide range and type of equipment, the latency to perform preventive maintenance (collecting the information, sending it across the enterprise where it’s needed, analyzing it, and pushing it on to subsequent processes) becomes very complex, and the data potentially loses its value if all of the above is not happening in a timely manner. 

A DOC can help enable the preventive maintenance sequence by helping us to better understand data properties of the IoT devices. For example, if we want to identify conditions that occur before equipment failure, we might apply a data set to the required edge analytics and gain greater visibility into the order of events that identify those conditions. DataOps solutions will not invalidate the full data supply chain but rather enable the value extraction closest to the source where that value can be utilized. 

An Analogy to Demonstrate the Benefits of DataOps 

In an air traffic control analogy, it doesn’t matter who operates the airline, who the passengers are, or where the luggage is going or how it is routed. DataOps ensures that airports have the smoothest, friction-free operations so that everyone benefits, and that users who rely on data can extract value without necessarily having knowledge of how to fly the plane, manage luggage, or handle security, reservations, etc. Air traffic control has an aspect of governance, but with DataOps, we focus more on best practices by incorporating the following: 

• Continuous data integration 
• Real-time analytics 
• Collaborative communication 
• Automated reporting and monitoring 
• Feedback loops 
• Compliance and safety integration 

By employing DataOps within the air traffic control framework, the airport manages to significantly reduce delays and improve overall efficiency. Flight schedules become more adaptable, safety standards are upheld, and the air traffic controllers can respond to issues faster, enhancing both operational performance and the passenger experience. 

Wrapping Up

So, how does an existing team change when there is a DOC in the enterprise? What value does the DOC add to the operations of a data science team, data engineering team, or analytics team?   

Digitalizing functions that can be automated and monitoring everything to make sure that the workflow works effectively across the board on an ongoing basis frees employees to focus on higher-value work. We would further argue that a well-run data operations center will provide a seamless experience, ensuring that teams have the information they need to implement solutions without unnecessary process and “paperwork.” You might be thinking, “DataOps sounds very compelling, but what exactly are some of its practices?” Stay tuned for future articles that dive into various DataOps elements, including data competency and DataOps professionals.