Data standardization, cleaning, and validation

Cryptic data values

data items that have no meaning without understanding a coding scheme

Misfielded data values

data values that are correctly formatted but not listed in the correct field

Data consistency

the principle that every value in a field should be stored in the same way

Data cleaning

process of updating data to be consistent, accurate, and complete

Data de-duplication

process of analyzing data and removing two or more records that contain identical information

Data filtering

process of removing records or fields of information from a data source

Data imputation

process of replacing a null or missing value with a substituted value

Data contradiction errors

errors that exist when the same entity is descrived in 2 conflicting ways

Data threshold violations

data errors that occur when a data value falls outside of an allowable level

Violated attribute dependencies

errors that occur when a secondary attribute in a row of data does not match the primary attribute

Data entry errors

all types of errors that come from inputting data incorrectly

Data validation

process of analyzing data to make certain the data has the properties of high-quality data

Visual inspection

process of examining data using human vision to see if there are problems

Basic statistical tests

performed to validate the data

Audit a sample

one of the best techniques for assuring data quality

Advanced testing techniques

possible with a deeper understanding of the content of data