2.1 )Exploratory Data Analysis for Univariate Quantitative Data

Data Cleaning and Survey Data Preparation

  • Data cleaning is the critical first step before conducting any statistical analysis.

  • Examples of data issues identified in raw student survey data (22302230 total initial responses gathered):

    • Missing values: Respondents failed to complete all survey questions (missing observations were dropped).

    • Logical ambiguities: Responses indicating 18+18+ semesters until graduation (e.g., whether students counted spring and summer terms).

    • Impossible physical measurements: Heights reported as 5.9 in5.9\,\text{in} (filtered out; only heights satisfying Height≥48 in\text{Height} \ge 48\,\text{in} were retained).

    • Excessive course loads: Enrollment reported as 25 credit hours25\,\text{credit hours} (filtered out; only credit loads satisfying Credit Hours≤18\text{Credit Hours} \le 18 were retained).

    • Implausible age entries: Claims of being 25 years old25\,\text{years old} when discovering Santa Claus isn't real.

    • Outrageous tuition costs: Semester tuition reported as $25000\$25000.

    • Unrealistic weekly commitments: Homework time reported as 120 hours/week120\,\text{hours/week}.

    • Factually incorrect historical dates: Claims that the internet was invented in the year 20102010.

    • Incorrect university founding dates: Year Brigham Young University was established entered as 16771677 or 20002000.

    • Data entry scale mismatches: Desired salary entered as 8080 (intended to represent $80000\$80000).

  • Summary of dataset cleaning outcomes:

    • Initial total responses: 22302230

    • Discarded responses: 467467 (thrown out due to incomplete or impossible answers)

    • Cleaned sample size (nn): 17631763 individuals

    • Total variables recorded: 3434 variables

  • Clean dataset variables include: SemestersTillGrad, AttendDevos, InRelationship, UsedDatingApp, AppleOrAndroid.

  • Sample cleaned response records:

    • Individual 1: 66 semesters till graduation, No devotionals, No relationship, No dating app, Apple

    • Individual 2: 77 semesters till graduation, Yes devotionals, Yes relationship, No dating app, Apple

    • Individual 3: 44 semesters till graduation, Yes devotionals, No relationship, No dating app, Apple

The Framework of Statistical Analysis

  • The formal four-step process of statistical analysis consists of:

    1. Identify the target population and parameter of interest.

    2. Collect sample data from the population.

    3. Posit a statistical model based on information contained in the sample.

    4. Draw inference about the target population using the statistical model.

Univariate Research Objectives and Variable Types

  • Specific Study Setup and Research Questions:

    • Research Question: What is the average age at which BYU students find out Santa Claus isn't real? Specifically, is the mean age at which BYU students discover Santa Claus isn't real greater than 88?

    • Target Population: All Brigham Young University students.

    • Population Parameter: The mean age at which all BYU students find out Santa Claus isn't real, denoted by the Greek letter μ\mu (pronounced "mew").

    • Sample: A convenience sample of 17631763 BYU students enrolled in Stat 121 who completed the online student survey.

  • Variable Definitions and Classifications:

    • Response Variable (yy): The age at which a student discovered Santa Claus isn't real.

    • Continuous Quantitative Variable: A numeric variable that can take any value within a range, including fractions and decimals (e.g., age at discovery 8.3918.391).

    • Discrete Quantitative Variable: A numeric variable that can only take values from a fixed set of countable numbers (e.g., number of bedrooms in a house).

    • Categorical Variable: A non-numeric variable that places an observation into one of a fixed set of categories (e.g., college of major, device brand like Apple or Android).

    • Explanatory Variable (xx): None for univariate quantitative exploratory data analysis.

Exploratory Data Analysis (EDA) Overview

  • Exploratory Data Analysis (EDA) is the foundational first step of any statistical analysis, using sample data to understand the broader population.

  • Primary Methods: Understanding data through graphical visual representations (pictures) and numerical summaries.

  • Essential Core Concept - Distribution: The distribution of a variable defines what values the variable can take and how frequently it takes those values.

Graphical EDA: Histograms

  • Step-by-Step Procedure to Construct a Histogram:

    1. Divide the continuous range of numerical data into bins (number ranges, e.g., 0 to 10\text{ to }1, 1.01 to 21.01\text{ to }2, 2.01 to 32.01\text{ to }3).

    2. Count the number of data observations falling inside each bin.

    3. Plot the numerical bin ranges along the horizontal x-axis and the counts along the vertical y-axis.

Histogram of Age Santa Claus
  • Describing Histograms: Every histogram must be comprehensively evaluated using three fundamental features:

    • Shape: The visual geometry of the distribution curve.

    • Normal: Symmetric and bell-shaped with a single central peak.

    • Left Skewed: The tail of the distribution extends toward smaller/lower values on the left.

    • Right Skewed: The tail of the distribution extends toward larger/higher values on the right.

    • Uniform: The distribution is flat, with equal frequencies across all bin ranges.

    • Bimodal: The distribution displays two prominent distinct peaks.

    • Center: The middle point of the data distribution.

    • Spread: The range or variability spanning from the smallest to the largest observed value.

Histogram Shapes: Bimodal, Left Skewed, Normal, Right Skewed, UniformHistograms with Center IndicatedHistograms with Spread Indicated
  • Critical Limitation of Histograms:

    • Visual representation depends heavily on the chosen bin size and number of bins.

    • Changing bin resolution (e.g., 55 bins vs. 1010 bins vs. 2525 bins) alters the perceived shape and level of detail.

Effect of Bin Selection on Histograms

Graphical EDA: Density Plots

  • Definition: A density plot is a smoothed, continuous mathematical estimation of a histogram.

  • Fundamental Mathematical Principle: The total area under any density curve is equal to 11.

Density Plot of Age Santa Claus

Software Tooling for Exploratory Data Analysis

  • Navigation in the Stat 121 Analysis Tool (rconnect.byu.edu/Stat121V2):

    • Step 1: Select Dataset under "1) Dataset Selection" (e.g., pre-existing dataset 401K Returns with sample size n=40n = 40, or upload a custom file).

    • Step 2: Select Variable under "2) Select Variable" (e.g., Year or AgeSantaClaus).

    • Step 3: Select Plot under "3) Graphical Exploratory Data Analysis" (choose desired plot type).

Stat 121 Analysis Tool - Graphical EDA Interface
  • Step 4: Calculate numerical metrics under "4) Numerical Exploratory Data Analysis" by selecting summary statistics from the dropdown menu.

Stat 121 Analysis Tool - Numerical EDA Interface

Numerical EDA: Measures of Center

  • Sample Mean (yˉ\bar{y}):

    • Represents the traditional numerical average and acts as the physical balancing point of the histogram.

    • Mathematical Formula:

yˉ=1n∑i=1nyi=y1+y2+⋯+ynn\bar{y} = \frac{1}{n} \sum_{i=1}^{n} y_i = \frac{y_1 + y_2 + \dots + y_n}{n}

  • Sample Median (MM):

    • The exact middle data value when observations are sorted in order.

    • Cuts the distribution exactly in half (50%50\% of observations lie below MM and 50%50\% lie above MM).

  • Mode:

    • The most frequently occurring individual data value, corresponding to the highest peak of the density curve.

  • Sample Numerical Values for AgeSantaClaus Dataset:

    • Sample Mean: yˉ=8.391\bar{y} = 8.391

    • Median: M=8M = 8

    • Mode: Mode=7.953\text{Mode} = 7.953

Density Plot of Age Santa Claus with Mean, Median, and Mode

Distinction Between Sample Statistics and Population Parameters

  • Key Conceptual Relationship:

    • A sample statistic (such as sample mean yˉ\bar{y}) is computed directly from sample data.

    • A population parameter (such as population mean μ\mu) is a fixed, generally unknown theoretical value describing the whole population.

  • Inequality Principle:

yˉ≠μ\bar{y} \neq \mu

  • Analytical Implications: Observing a sample mean yˉ=8.391>8\bar{y} = 8.391 > 8 does not eliminate the need for further analysis. Sample statistics vary from sample to sample, requiring formal statistical inference to determine whether μ>8\mu > 8 for the population.

Numerical EDA: The Five-Number Summary and Percentiles

  • Components of the Five-Number Summary:

    1. Minimum (Min\text{Min}): The smallest individual value in the sample dataset.

    2. First Quartile (Q1Q_1): The 25th25\text{th} percentile (25%25\% of sample data falls below this value).

    3. Second Quartile (Q2=MQ_2 = M): The median or 50th50\text{th} percentile (50%50\% of sample data falls below this value).

    4. Third Quartile (Q3Q_3): The 75th75\text{th} percentile (75%75\% of sample data falls below this value).

    5. Maximum (Max\text{Max}): The largest individual value in the sample dataset.

  • Definition of pthp\text{th} Percentile: The specific numerical value such that p%p\% of the data observations fall strictly below it.

Graphical EDA: Boxplots

  • Definition: A boxplot is a direct visual display of the five-number summary.

  • Visual Structure:

    • Central box spans from Q1Q_1 to Q3Q_3 (representing the middle 50%50\% of observations).

    • A solid vertical line inside the box marks the median (MM).

    • Whiskers extend outwards from the box to the minimum and maximum non-outlier values.

Boxplot of Age Santa Claus

Numerical EDA: Measures of Spread

  • Standard Deviation (ss):

    • Conceptually measures the average distance of individual observations from the sample mean yˉ\bar{y}.

    • Mathematical Formula:

s=∑i=1n(yi−yˉ)2n−1s = \sqrt{\frac{\sum_{i=1}^{n} (y_i - \bar{y})^2}{n - 1}}

  • Interquartile Range (IQR\text{IQR}):

    • Measures the total spread of the middle 50%50\% of data observations.

    • Mathematical Formula:

IQR=Q3−Q1\text{IQR} = Q_3 - Q_1

  • Range:

    • Measures the absolute distance between extreme observations.

    • Mathematical Formula:

Range=Max−Min\text{Range} = \text{Max} - \text{Min}

  • Calculated Spread Metrics for AgeSantaClaus Dataset:

    • Standard Deviation: s=2.684s = 2.684

    • Interquartile Range: IQR=3\text{IQR} = 3

    • Range: Range=23\text{Range} = 23

Density Plot with Spread Statistics

Properties of Measures of Spread

  • Non-negativity Rule: Standard deviation (ss), Range, and Interquartile Range (IQR\text{IQR}) are strictly non-negative:

s≥0s \ge 0

Range≥0\text{Range} \ge 0

IQR≥0\text{IQR} \ge 0

  • Zero Variability Rule: If s=0s = 0 or Range=0\text{Range} = 0, every single observation in the dataset holds the exact same numerical value.

  • Dimensional Units Rule: Standard deviation (ss), Range, and Interquartile Range (IQR\text{IQR}) always retain the exact same physical measurement units as the original raw data.

Numerical EDA: Measures of Shape and Skewness

  • Numerical Domain of Skewness: Skewness values range continuously from negative infinity to positive infinity:

−∞<skewness<∞-\infty < \text{skewness} < \infty

  • Directional Interpretation:

    • If skewness<0\text{skewness} < 0: The distribution is left-skewed.

    • If skewness>0\text{skewness} > 0: The distribution is right-skewed.

  • Normality Threshold Rule of Thumb: If skewness satisfies the inequality:

−0.5<skewness<0.5-0.5 < \text{skewness} < 0.5

the distribution is classified as close enough to normal or symmetric.

Density Plots with Skewness Values

Impact of Distribution Shape on Center and Spread

  • Influence of Skewness and Outliers:

    • The sample mean is pulled strongly toward the long tail of a skewed distribution.

    • Standard deviation inflates when extreme values or long tails are present.

    • Median and IQR\text{IQR} remain resistant to skewness and extreme values.

  • Positional Hierarchy of Center Measures across Distribution Shapes:

    • Right-Skewed Distribution:

Mode<Median<Mean\text{Mode} < \text{Median} < \text{Mean}

  • Symmetric Distribution:

Mean=Median=Mode\text{Mean} = \text{Median} = \text{Mode}

  • Left-Skewed Distribution:

Mean<Median<Mode\text{Mean} < \text{Median} < \text{Mode}

Comparison of Mean, Median, and Mode for Skewed and Symmetric Distributions

Outlier Detection and the 1.5 IQR Rule

  • Outlier Definition: An observation that lies an extreme distance away from the primary cluster of data.

  • The 1.5×IQR1.5 \times \text{IQR} Outlier Flagging Rule: An observation yy is formally classified as an outlier if it satisfies either inequality condition:

y>Q3+1.5×IQRy > Q_3 + 1.5 \times \text{IQR}

y<Q1−1.5×IQRy < Q_1 - 1.5 \times \text{IQR}

  • Representation on Boxplots: Flagged outliers are drawn as individual isolated points beyond the boxplot whiskers.

Rules of Thumb for Selecting Summary Statistics

  • Case 1: Skewed Distributions or Datasets with Numerous Outliers:

    • Report Median for measure of center.

    • Report Interquartile Range (IQR\text{IQR}) for measure of spread.

  • Case 2: Approximately Normal / Symmetric Distributions without Major Outliers:

    • Report Mean (yˉ\bar{y}) for measure of center.

    • Report Standard Deviation (ss) for measure of spread.

Applied Homework Options and Key Vocabulary

  • Unit 2 Homework Options:

    1. 401(k) Returns

    2. Calories in Cereal

    3. Sunshine Duration

  • Key Terminology List:

    • Continuous quantitative variable

    • Discrete quantitative variable

    • Exploratory data analysis (EDA)

    • Distribution of a variable

    • Histograms

    • Density plots

    • Shapes (Normal, Left Skewed, Right Skewed, Uniform, Bimodal)

    • Measures of center (Mean, Median, Mode)

    • Measures of spread (Standard deviation, Interquartile Range, Range)

    • Measures of shape (Skewness)

    • Outliers and outlier inflation effects

    • 1.5×IQR1.5 \times \text{IQR} Rule for detecting outliers