Data profiling is concerned with data quality and this entry describes the main data quality categories of Column profiling, Dependency profiling and Redundancy profiling. In most commercial applications data quality is a primary concern as large organizations tend to duplicate data stores, causing redundancy and referential integrity issues. Anyone concerned with software testing data processing systems needs a good grasp of data profiling concepts and techniques.

We have singled out this resource as an introduction to data quality and data profiling, even though it is in fact a documentation set for an open source tool, DataCleaner. The first section of this documentation gives a good introduction to data profiling and data quality. The rest of the documentation is worth reading, even though you may not use the tool, as it gives an insight into data profiling tool capabilities. The tool itself is listed in our Data quality testing tools category.

Data cleansing is a reactive measure, as opposed to data validation that proactively eliminates ‘dirty’ data during the creation phase. Data cleansing involves removing or correcting the incorrect data. For example typo errors can be corrected with a mass spell checker, or against known lists of correct possibilities, for example postal codes. Where the correct entry cannot be determined then the entries are ‘flagged’ for inspection and manual correction.

The idea is that non-transactional data, i.e. Customer, Vendor, Product etc. becomes duplicated within a large organization and inconsistencies are introduced, thereby compromising data integrity. If you haven’t thought about the subject before consider an application that you have recently worked on and identify the master data and ask the question: What other data bases does this master data have to synchronize with?

From MSDN this article goes into detail about why master data should be managed. The article then breaks down what master data management comprises of, including identifying sources, identifying producers and consumers of master data and developing a master data model.