Process of Data Integration

One of the most important tasks in the process of data integration is to establish realistic expectations. The term data integration invokes a perfect coordination of diversified databases, software, equipment and personnel in an alliance that runs smoothly, without the lingering headaches that mark less complete information management systems. Think again.

The requirements analysis phase offers one of the best opportunities in the process to recognize and assimilate the full scope of the complexity of data integration. It is possible that paying close attention to this analysis is the most important ingredient in creating a system that will live to see adoption and maximum use. However, as the field of data integration progresses, other common obstacles and compensatory solutions will be easily identified. Current integration practices have already highlighted some family challenges as well as strategies to address them, as described below.

For most transport agencies, data integration involves the synchronization of large amounts of variable and heterogeneous data from internal legacy systems that vary in the format of the data. It is possible that old systems were created from flat files, networks or hierarchical databases, unlike the most recent generations of databases that use relational data. Data in various formats from external sources continue to be added to legacy databases to improve the value of information. Each generation, product and national system has unique requirements to meet to store or extract data. Therefore, data integration may involve different strategies to manage heterogeneity.

In some cases, the effort becomes an important exercise in the homogenization of the data, which may not improve the quality of the data offered. Data quality is a major concern in any data integration strategy. The information provided must be deleted before conversion and integration, or an agency will almost certainly face serious data problems later. The impurities of the inherited data have a compositional effect; by their nature, they tend to focus on high-volume data users. If this information is corrupt, so will the decisions taken. It is not uncommon for undiscovered data quality problems to arise in the process of cleaning up the information that needs to be used by the integrated system. The problem of incorrect data leads to procedures for the regular monitoring of the quality of the information used. But who has ultimate responsibility for this work is not always clear.