Approaching Operational Data Integration
(Featured image credit: IBM)
What is operational data integration?
Operational data integration (OpDI) is one of the fastest-growing practices in data integration, as TDWI Research revealed in 2007. In brief, OpDI stands for “implementations, projects, or initiatives commonly described as the consolidation, collocation, migration, upgrade, or synchronization of operational databases,” according to Philip Russom of TDWI.
In other words, OpDI implies data integration between operational (nonanalytical) applications, “whether in one enterprise or across multiple ones.”
Today, as most organizations make efforts to implement BI and data warehousing initiatives enterprise-wide, there is a growing demand for different integration techniques, including EAI (enterprise application integration), EII (enterprise information integration), ETL (extract, transform, and load), data federation, etc.
“All the techniques and tool types under the broad rubric of data integration operate similarly in that they copy data from a source, merge data coming from multiple sources, and alter the resulting data model to fit the target system that data will be loaded into. Because of the similar operations, industrious users can apply just about any technique (or combination of these) to any data integration implementation, initiative, project, or practice—including those for OpDI.” —Philip Russom, TDWI
Operational data integration is needed to address several challenges data-driven companies face.
Why is it important and hard?
Redundant data in multiple nonstandardized databases across different departments is a serious issue, which leads to increased IT costs and hinders unified visibility into a company’s business processes. A lot of nonstandardized applications throughout an enterprise may also provide redundant data. To fix this, one needs to standardize data and initiate database consolidation, migration (if necessary), and so on.
“But beware, because migration projects are intrusive—even fatal—in that they kill off older systems after their data has been moved to another database platform.” —Philip Russom, TDWI
However, it’s not that straightforward. Very often, you cannot get rid of a legacy system, or it’s not easy. According to Philip, “sometimes it’s best to avoid the risk, cost, and disruption of data migration and leave redundant applications and databases in place.” In this case, an organization may need to implement synchronization workflows and tools. Yet this initiative may face its own challenges, too.
“When each database involved in synchronization is subject to frequent inserts and updates, it’s inevitable that some data values will conflict when multiple systems are compared. For this reason, synchronization technology must include rules for resolving conflicting data values.” —Philip Russom, TDWI
When put in place, according to Philip, data synchronization becomes “a permanent piece of infrastructure that runs daily for years before reaching the end of its useful life.”
As enterprises are heading for more agility and real-time data integration to increase efficiency and accelerate business responsiveness, the demand for operational data integration is only growing. At the same time, Philip notes, they still lack features needed for data quality, MDM, scalability, etc.
“OpDI solutions are in serious need of improvement or replacement. Many are hand-coded legacies that need to be replaced by modern solutions built atop vendor tools.” —Philip Russom, TDWI
For more, see a recent TDWI’s article by Philip covering the main practice areas in OpDI—migration, sync, and B2B data exchange—as well as common pitfalls and best practices involved.
- The Three Most Common Problems Faced During Data Migration
- What Is the Difference Between ETL and EAI?
- Aberdeen Group Outlines Four Hidden BI Cost Factors to Avoid