The focus of businesses today is on improving customer service, risk management, inventory, supply chain management and many other traditional business functions. They cannot do this without access to the right data.
Business requirements may differ across various industries but the underlying data requirements are similar – data must be integrated, current, detailed, and immediately accessible. This is where deploying an operational data store can provide a great solution.
Definition of an ODS
The development of the operational data store (ODS) has been driven by business needs. An ODS contains operational data from different sources with the purpose of providing end-users with an integrated view of enterprise data. By storing a snapshot of current data, it is ideally suited for real-time or near real-time decision making and reporting.
An ODS is different from a data warehouse in terms of its update frequency. An ODS is updated daily, hourly, or even immediately after transactions on operational data. It overwrites operational data as new, current information comes in.
Main advantages of an ODS
- The ODS offers more accessibility to operational data and handles uncomplicated querying on a small quantity of data. It may lack the benefits of a data warehouse but it has the advantage of being fast and light.
- An ODS offers information for operational and tactical decisions based on current or near real-time data. Data can be used almost immediately when accounting for timing differences from the different reporting applications.
- The ODS integrates data from new and existing systems, creating a central repository for data. Instead of limited reporting offered by using source systems, this enables the creation of more comprehensive operational reports.
- Organizations have a more complete view of various business functions and the current state view makes it easier to identify and diagnose issues when they happen, such as when a customer order goes astray.
- Businesses can build rules on an ODS so that data changes in a particular system trigger an action on another system.
- Only a few people have the security to access systems of record but an ODS does not have the same security issues and is more resilient to cyber-attacks. More people have access to reporting.
- The ODS can act as an intermediate stage before data goes to the data warehouse. It can put data in a consistent format and therefore improve the quality of data in the data warehouse.
Some ODS challenges
There are a number of challenges that need addressing when using an ODS, such as data transfer, transformation and volume.
Populating an ODS with data can be a complex activity. Integrating and consolidating data from multiple disparate data sources can create various challenges, such as how to match data from two different systems.
For instance, businesses may have data coming from the cloud, some cloud system or a traditional in-house database management system. An ODS has to handle the data coming from all these sources and ensuring cloud security is just one of the challenges many businesses face today.
What data to put into the ODS and what data not to put into it is another issue. If the ODS contains unnecessary data, the performance will degrade and the management complexity will increase.
Does the entire ODS become the “single version of the truth”? If this is the case, then all affected legacy applications and possibly the data warehouse will need modifying which can potentially be a major undertaking.
Sourcing and extracting data is followed by transformation to eliminate any remnants of operational application silos. This includes cleaning junk data to reduce redundancy and transforming all the data into a single format. The extent and type of transformations will depend on the data delivery timing requirements.
When businesses want data within a very short time after transactions, they may not be able to transform or integrate the data in the same way as when they had more time for processing. They have to decide how frequent they want the data to be and balance this with the refresh frequency and how much transformation and integration they want.
It may be necessary to consider the use of an Extract Transform Move Load (ETML) tool with the capability of handling high update volumes and a wide range of data structures and formats.
Data volume: The ODS will be smaller than the data warehouse but may still contain huge amounts of data. As the data volumes grow, so do the costs of managing the ODS.
Data structure design: The ODS structure must take into account the mixed workload of transactional updates, tactical queries, and some analytics. Typically the data accessed in the ODS is transactional in nature but at the same time, the design must be able to handle an update intensive workload from the various data sources.
The main purpose of an ODS is to offer querying and reporting on very current operational data. The reports are created on a short time window of data. Reports may be affected by the limited integration and transformation of data from all the different sources.
To query history, a data warehouse or application-specific data marts must be used instead of an ODS because the ODS allows for simple queries but does not handle complex queries or the need for elaborate reports. Should a business need information about what goods to order to stock at different times of the year or the best-selling products over the past six months, the ODS is not the right tool to use.
A final word
An ODS does not substitute the need for a data warehouse to keep historical data, handle data analysis and search large amounts of data. However, it provides an important way to integrate data from various sources and provide real-time or near real-time data. Today’s business decisions are heavily influenced by the availability of an ODS and the current, integrated, accessible data it provides.