3 data cleaning, measurement, and descriptive statistics

Data cleaning

• the process of identifying, investigating, correcting, and documenting problems that could distort an analysis 

Important: cleaning does not mean changing inconvenient values. Every correction needs a defensible reason and should be documented. 

Accuracy 

• Variables should represent what was actually observed or measured

Consistency

• Categories, formats, and definitions should be applied the same way 

Usability

• The dataset should be organized so summaries can be produced correctly 

Some key things to consider when cleaning data

• Merging datasets

• Data inspection

• Handling missing data

• Removing duplicates

• Dealing with outliers 

• Standardization 

• Data transformation 

Keep the raw file unchanged. This gives you a recovery point and makes every later cleaning decision traceable 

Measurement level determines valid summaries 

• A level of measurement describes what information a variable carries and which mathematical comparisons are meaningful 

Nominal → ordinal → interval-ratio 

Nominal level of measurement

• Variables measured at the nominal level are categorical

• They might be represented by numbers,  but do not have numerical sense

• Although they are categorized, they cannot be ranked 

  • Ex: gender, race, religious affiliations  

• They cannot be added, divided, or manipulated mathematically 

• However, they can be compared 

  • ex: there are more sophomores than seniors in the class

The ordinal level of measurement

• In addition to being categorical in nature, variables measured at the ordinal level can be ranked from low to high 

  • Ex: socio-economic level (lower class, middle class, upper class) ; language ability (beginner, intermediate, fluent)

• Although they are ordered, we cannot distinguish the distance between each categories

  • Ex: we cannot say that the middle class is 2x or 3x better than the lower class 

The interval-ratio level of measurement 

• Variables measured at the interval ratio level not only permit classification and ranking but also allow the distance from category to category (or score to score) to be exactly defined 

• They are numeric, and have equal interval s

• All mathematical operations (addition, subtraction, division, etc.) can be performed on them 

Descriptive statistics summarize what we observed 

Purpose

• Organize and communicate the main patterns in a dataset

• 
  

Scope

• Describe the observed cases without automatically generalizing beyond them 

Examples

• Counts, percentages, averages, ranges, frequency tables, and charts

Statistics can describe or infer

Descriptive

• Summarize patterns in the data we observed

Inferential 

• Use sample evidence to make cautious claims about a larger population

Univariate

• Describe one variable at a time

Bivariate or multivariate

• Examine relationships among two or more variables 

The variable guides the summary 

Variable (what was measured) → 

Measurement level (nominal, ordinal, or interval-ratio?) → 

Summary (count, percentage, median, mean, or spread?) → 

Display (table, bar chart, histogram, or another clear visual?)

Percentages and proportions 

• Percentages and proportions give a frame of reference for describing a variable

• Percentages are standardized into a base of 100 while proportions are standardized to a base of 1