There are numerous examples in the data warehousing industry of projects that have failed as a result of a lack of data quality and completeness.  Successful data warehousing projects all share a common emphasis on the ongoing analysis of the state of quality of source databases or files. 
 

ABSi’s experts can assist your organization with the automation of data analysis or profiling, such as completeness, value frequency distribution, volumetric, outlier analysis of low occurrence values and other reasonability analyses.  All of these are essential components of an emphasis on organizational data quality. 
 

We feel that it is essential that data quality assessments begin with an alerting and monitoring capability for operations staff in order to identify data issues when they occur. It is not uncommon for data issues such as malformed files, truncated data files, or missing data to not be detected until the time of a processing cycle, or in the worst case, when users themselves identify missing data from downstream data marts or databases. 
 

ABSI can both develop and improve the software components that your organization uses to provide alerting and monitoring of data completeness of data quality issues.  Using software for cataloging and archiving incoming files, and reporting/alerting scripts, output from these software “monitors” can be distributed to key operations personnel (e.g. via email) to highlight files with internal file or record problems including file truncations due to communications failures at either the sending or receiving site or problems with file creation at the site. 
 

For the complex data warehousing or data ingest environment, we have also pioneered the use of data ingest tools to identify and report on the absence of files or data that are expected.  In cases where there is large volumes of data from large numbers of locations, received at varying frequencies and volumes, it can be very difficult for an organization to determine what is missing. 
 

In large databases, it can be problematic to quantify absence, but through the use of Statistical Process Control tools, we can detect abnormal or problem “variation” and perform “Process Characterization”, all of which are of immense importance in this identification of data quality issues.
 

Bottom line, ABSI can help your organization answer the question, “what arrived and when” and “what did not arrive – what is missing”.   The ability to answer these two question is key to the improvement of organizational productivity and success.