Knowing demographics helps build a better model with more specific ideas of what accurately predicts insurance coverage probability.
Techniques for Data Exploration
Combination of summary statistics (means, medians, variances, counts) and visualization (graphs).
Some problems can be spotted using summary statistics, while others are easier to find visually.
Using Summary Statistics
Use the summary() command in R to get an initial look at the data.
The goal is to understand if the data can potentially help predict health insurance coverage and if the data is of good quality.
Example
Using the setwd() command to specify the directory, followed by the readRDS() to read the file in.
Then use the summary(customer_data) to get the different values.
Organizing Data for Analysis
Data is assumed to be in a single data frame for most of the book.
Typically, data is stored in normalized form across multiple small tables in a database or across multiple log entries/sessions in log data.
These formats are optimized for adding/modifying data, but not for analysis.
Data can be joined into a single table using SQL or R commands like join (discussed in chapter 5).
Interpreting the summary() Command Output
The summary() command reports summary statistics on numerical columns and count statistics on categorical columns (if read in as factors).
Helps quickly spot potential problems like missing data or unlikely values.
Provides a rough idea of how categorical data is distributed.
Typical Problems Revealed by Data Summaries
Missing values
Invalid values and outliers
Data ranges that are too wide or too narrow
The units of the data
Missing Values
A few missing values may not be a problem, but largely unpopulated data fields should be repaired.
Many modeling algorithms in R quietly drop rows with missing values by default.
If a data field is largely unpopulated, determine why; sometimes the fact that a value is missing is informative.
Decide on the most appropriate action: include the variable, drop rows with missing values, or convert missing values to 0 or an additional category.
Missing values will likely be encountered during model scoring, so they should be dealt with during model training.
Invalid Values and Outliers
Check that the values make sense, including invalid values or outliers.
Examples of invalid values: negative values in non-negative numeric data fields (age, income) or text where numbers are expected.
Outliers are data points that fall well outside the expected range.
Invalid values and outliers might be bad data input, sentinel values, or valid but unusual data points.
Decide on the most appropriate action: drop the data field, drop the data points, or convert the bad data to a useful value.
Consider omitting outliers from model construction if they interfere with the model-fitting process.
The goal of modeling is to make good predictions on typical cases, so a model skewed to predict rare cases may not be the best overall.
Data Range
Pay attention to how much the values in the data vary.
Ensure there is enough variation in predictive variables (age, income) to see relationships.
Data that ranges over several orders of magnitude can be a problem for some modeling methods.
Data can also be too narrow; if all customers are within a small age range, that range won't be a good predictor.
A rough rule of thumb relates to the ratio of the standard deviation to the mean; if that ratio is very small, the data isn’t varying much.
Units
Consider the unit of measurement: hourly vs. yearly wages, kilometers vs. miles, dollars vs. thousands of dollars.
Check data definitions in data dictionaries or documentation to catch unit errors.
Spotting when measurements are in unexpected units by looking over the value ranges of variables.
Spotting Problems Using Graphics and Visualization
Pictures are better than text for some data characteristics.
Visualization helps to absorb information like the peak age of distribution, the range of the data, and the presence of outliers more easily.
Principles for Scientific Visualization
A graphic should display as much information as possible with the lowest cognitive strain to the viewer.
Strive for clarity, making the data stand out.
Avoid too many superimposed elements.
Find the right aspect ratio and scaling to bring out details.
Avoid having the data skewed to one side of the graph.
Visualization is an iterative process intended to answer questions about the data.
Different graphics are best suited for answering different questions.
Visualizations and Graphics Using ggplot2
The book demonstrates visualizations using the R graphing package ggplot2, as well as some prepackaged ggplot2 visualizations from the package WVPlots.
Key Points to Understand for ggplot2
Graphs in ggplot2 can only be defined on data frames.
Variables in a graph are called aesthetics and are declared using the aes function.
The ggplot() function declares the graph object and can include the data frame of interest and the aesthetics.
Layers produce plots and plot transformations, added to a given graph object using the + operator.
Each layer can also take a data frame and aesthetics as arguments, in addition to plot-specific parameters.
Examples of layers are geom_point (for a scatter plot) or geom_line (for a line plot).
Visually Checking Distributions for a Single Variable
Histograms
Density plots
Bar charts
Dot plots
Questions to Answer
What is the peak value of the distribution?
How many peaks are there in the distribution (unimodality versus bimodality)?
How normal (or lognormal) is the data?
How much does the data vary? Is it concentrated in a certain interval or in a certain category?
Examining the Shape of Data Distribution
Check for unimodality in data.
A unimodal distribution corresponds to one population of subjects.
Multimodal distributions suggest data comes from multiple populations.
Histograms
Bins a variable into fixed-width buckets and returns the number of data points that fall into each bucket as a height.
Density Plots
A continuous histogram of a variable, where the area under the density plot is rescaled to equal one.
More interested in the overall shape of the curve than actual values on the y-axis.
Bar Charts and Dotplots
A bar chart is a histogram for discrete data, recording the frequency of every value of a categorical variable.
Visually Checking Relationships Between Two Variables
Questions to Answer
Is there a relationship between the two inputs age and income in my data
If so, what kind of relationship, and how strong?
Is there a relationship between the input marital status and the output health insurance? How strong?
Visualizations
Line plots and scatter plots for comparing two continuous variables
Smoothing curves and hexbin plots for comparing two continuous variables at high volume
Different types of bar charts for comparing two discrete variables
Variations on histograms and density plots for comparing a continuous and discrete variable
Line Plots
Work best when the relationship between two variables is relatively clean: each x value has a unique (or nearly unique) y value
Scatter Plots and Smoothing Curves
The appropriate summary statistic is the correlation
Gets correlation of Resulting correlation is age and income positive but nearly zero.
Hexbin Plots
A hexbin plot is like a two-dimensional histogram
The data is divided into bins, and the number of data points in each bin is represented by color or shading.
Bar Charts for Two Categorical Variables
Stacked bar chart, side-by-side bar chart or shadow plot
Comparing a Continuous and Categorical Variable
Overlaid density plots give you good information about distribution shape: where populations are dense and where they are sparse, whether the populations are separated or overlap
Summary
Visualization is an iterative process and helps answer questions about the data.
Information you learn from one visualization may lead to more questions— that you might try to answer with another visualization.
If one visualization doesn't work, try another. Time spent here is time not wasted during the mod- eling process.