Exploring Data

Exploring Data

  • The chapter focuses on exploring data before building a model to ensure the data is clean and useful.
  • The goal is to use summary statistics and visualization to explore data, identify problems, and address issues during data exploration.

Example Scenario

  • Predicting which customers lack health insurance using a dataset of customers with known health insurance status.
  • Predictive customer properties include age, employment status, income, residence information, and vehicle information.

Importance of Data Examination

  • Resist diving directly into modeling without examining the dataset first.
  • Datasets are often imperfect, containing missing, incorrect, inconsistent, or dirty data.
  • Examining data beforehand prevents redoing work due to discovering bad data fields or variables needing transformation.
  • In the worst case, failure to examine might lead to an incorrect model without a clear reason why.

Benefits of Addressing Data Issues Early

  • Saves unnecessary work and headaches.

Getting to Know Your Data

  • Understand customer demographics (age, affluence, location).
  • Knowing demographics helps build a better model with more specific ideas of what accurately predicts insurance coverage probability.

Techniques for Data Exploration

  • Combination of summary statistics (means, medians, variances, counts) and visualization (graphs).
  • Some problems can be spotted using summary statistics, while others are easier to find visually.

Using Summary Statistics

  • Use the summary() command in R to get an initial look at the data.
  • The goal is to understand if the data can potentially help predict health insurance coverage and if the data is of good quality.
Example
  • Using the setwd() command to specify the directory, followed by the readRDS() to read the file in.
  • Then use the summary(customer_data) to get the different values.

Organizing Data for Analysis

  • Data is assumed to be in a single data frame for most of the book.
  • Typically, data is stored in normalized form across multiple small tables in a database or across multiple log entries/sessions in log data.
  • These formats are optimized for adding/modifying data, but not for analysis.
  • Data can be joined into a single table using SQL or R commands like join (discussed in chapter 5).

Interpreting the summary() Command Output

  • The summary() command reports summary statistics on numerical columns and count statistics on categorical columns (if read in as factors).
  • Helps quickly spot potential problems like missing data or unlikely values.
  • Provides a rough idea of how categorical data is distributed.

Typical Problems Revealed by Data Summaries

  • Missing values
  • Invalid values and outliers
  • Data ranges that are too wide or too narrow
  • The units of the data
Missing Values
  • A few missing values may not be a problem, but largely unpopulated data fields should be repaired.
  • Many modeling algorithms in R quietly drop rows with missing values by default.
  • If a data field is largely unpopulated, determine why; sometimes the fact that a value is missing is informative.
  • Decide on the most appropriate action: include the variable, drop rows with missing values, or convert missing values to 0 or an additional category.
  • Missing values will likely be encountered during model scoring, so they should be dealt with during model training.
Invalid Values and Outliers
  • Check that the values make sense, including invalid values or outliers.
  • Examples of invalid values: negative values in non-negative numeric data fields (age, income) or text where numbers are expected.
  • Outliers are data points that fall well outside the expected range.
  • Invalid values and outliers might be bad data input, sentinel values, or valid but unusual data points.
  • Decide on the most appropriate action: drop the data field, drop the data points, or convert the bad data to a useful value.
  • Consider omitting outliers from model construction if they interfere with the model-fitting process.
  • The goal of modeling is to make good predictions on typical cases, so a model skewed to predict rare cases may not be the best overall.
Data Range
  • Pay attention to how much the values in the data vary.
  • Ensure there is enough variation in predictive variables (age, income) to see relationships.
  • Data that ranges over several orders of magnitude can be a problem for some modeling methods.
  • Data can also be too narrow; if all customers are within a small age range, that range won't be a good predictor.
  • A rough rule of thumb relates to the ratio of the standard deviation to the mean; if that ratio is very small, the data isn’t varying much.
Units
  • Consider the unit of measurement: hourly vs. yearly wages, kilometers vs. miles, dollars vs. thousands of dollars.
  • Check data definitions in data dictionaries or documentation to catch unit errors.
  • Spotting when measurements are in unexpected units by looking over the value ranges of variables.

Spotting Problems Using Graphics and Visualization

  • Pictures are better than text for some data characteristics.
  • Visualization helps to absorb information like the peak age of distribution, the range of the data, and the presence of outliers more easily.

Principles for Scientific Visualization

  • A graphic should display as much information as possible with the lowest cognitive strain to the viewer.
  • Strive for clarity, making the data stand out.
  • Avoid too many superimposed elements.
  • Find the right aspect ratio and scaling to bring out details.
  • Avoid having the data skewed to one side of the graph.
  • Visualization is an iterative process intended to answer questions about the data.
  • Different graphics are best suited for answering different questions.

Visualizations and Graphics Using ggplot2

  • The book demonstrates visualizations using the R graphing package ggplot2, as well as some prepackaged ggplot2 visualizations from the package WVPlots.

Key Points to Understand for ggplot2

  • Graphs in ggplot2 can only be defined on data frames.
  • Variables in a graph are called aesthetics and are declared using the aes function.
  • The ggplot() function declares the graph object and can include the data frame of interest and the aesthetics.
  • Layers produce plots and plot transformations, added to a given graph object using the + operator.
  • Each layer can also take a data frame and aesthetics as arguments, in addition to plot-specific parameters.
  • Examples of layers are geom_point (for a scatter plot) or geom_line (for a line plot).

Visually Checking Distributions for a Single Variable

  • Histograms
  • Density plots
  • Bar charts
  • Dot plots

Questions to Answer

  • What is the peak value of the distribution?
  • How many peaks are there in the distribution (unimodality versus bimodality)?
  • How normal (or lognormal) is the data?
  • How much does the data vary? Is it concentrated in a certain interval or in a certain category?

Examining the Shape of Data Distribution

  • Check for unimodality in data.
  • A unimodal distribution corresponds to one population of subjects.
  • Multimodal distributions suggest data comes from multiple populations.
Histograms
  • Bins a variable into fixed-width buckets and returns the number of data points that fall into each bucket as a height.
Density Plots
  • A continuous histogram of a variable, where the area under the density plot is rescaled to equal one.
  • More interested in the overall shape of the curve than actual values on the y-axis.
Bar Charts and Dotplots
  • A bar chart is a histogram for discrete data, recording the frequency of every value of a categorical variable.

Visually Checking Relationships Between Two Variables

Questions to Answer
  • Is there a relationship between the two inputs age and income in my data
  • If so, what kind of relationship, and how strong?
  • Is there a relationship between the input marital status and the output health insurance? How strong?
Visualizations
  • Line plots and scatter plots for comparing two continuous variables
  • Smoothing curves and hexbin plots for comparing two continuous variables at high volume
  • Different types of bar charts for comparing two discrete variables
  • Variations on histograms and density plots for comparing a continuous and discrete variable
Line Plots
  • Work best when the relationship between two variables is relatively clean: each x value has a unique (or nearly unique) y value
Scatter Plots and Smoothing Curves
  • The appropriate summary statistic is the correlation
  • Gets correlation of Resulting correlation is age and income positive but nearly zero.
Hexbin Plots
  • A hexbin plot is like a two-dimensional histogram
  • The data is divided into bins, and the number of data points in each bin is represented by color or shading.
Bar Charts for Two Categorical Variables
  • Stacked bar chart, side-by-side bar chart or shadow plot
Comparing a Continuous and Categorical Variable
  • Overlaid density plots give you good information about distribution shape: where populations are dense and where they are sparse, whether the populations are separated or overlap

Summary

  • Visualization is an iterative process and helps answer questions about the data.
  • Information you learn from one visualization may lead to more questions— that you might try to answer with another visualization.
  • If one visualization doesn't work, try another. Time spent here is time not wasted during the mod- eling process.