Descriptive Statistics Study Guide

Data Fundamentals and Structure

• Tabular Data Representation:

  • Data is organized into rows and columns, similar to spreadsheets in Excel or Google Sheets.

  • Observations ($i$): Represented by individual rows in a dataset. An observation is also referred to as an experimental unit.

  • Variables ($x_i$): Represented by individual columns in a dataset, corresponding to specific attributes or features measured across observations.

• Celebrity Dataset Example:

  • The dataset contains $4$ observations ($i = 1, 2, 3, 4$) and $5$ variables:

    • Observation $1$: Celebrity Name = Tom Cruise, Height = $67$\,inches, Married = N, Sex = M, Age = $60$

    • Observation $2$: Celebrity Name = Zendaya, Height = $70$\,inches, Married = N, Sex = F, Age = $26$

    • Observation $3$: Celebrity Name = Tom Holland, Height = $68$\,inches, Married = N, Sex = M, Age = $26$

    • Observation $4$: Celebrity Name = Natalie Portman, Height = $63$\,inches, Married = Y, Sex = F, Age = $41$

• Statistic Definition:

  • A statistic summarizes a variable in data across each observation.

  • Example: Calculating the average height across the four celebrities yields $\frac{67 + 70 + 68 + 63}{4} = 67$\,inches.

Branches of Statistics

• Descriptive Statistics:

  • Focuses on describing the "what" within the data ("in-sample").

  • Summarizes data using meaningful mathematical measures, summary tables, and charts.

• Inferential Statistics:

  • Focuses on making claims or predictions based on data ("out-of-sample").

  • Draws inferences from a sample to an entire underlying population.

  • Requires probability models and deeper statistical theory.

Visualizations and Summaries for Data Description

• Summary Tables: Summarize statistics across categorized dimensions.

• Graphical Charts and Visualization Types:

  • Bar Plot: Used to summarize counts or percentages for discrete groups or categories.

  • Pie Chart: Used to summarize the percentage of total belonging to each category within a single variable.

  • Histogram: Displays frequency distributions for continuous variables. Serves as the primary visualization method when analyzing a single quantitative variable.

  • Line Chart: Used to track variables over time (time-series analysis).

• Key Properties of Data Distributions:

  • Domain: The span of input values along the horizontal axis ($X$ axis).

  • Range: The span of frequencies or counts along the vertical axis ($Y$ axis).

  • Central Tendency (Center): Middle points such as median or mode.

  • Variability / Dispersion: Spread around the center, such as variance or standard deviation.

  • Shape: Symmetry, peakness, and skewness of the distribution.

• R Code Example for Data Generation and Plotting:

set.seed(123)
data <- rnorm(n=1000, mean=0, sd=1)
hist(data)

*   *Properties of generated dataset:* $X$ varies from $-3$ to $3$ (domain), $Y$ varies from $0$ to $200$ (range), median and mode are centered at $0$, skewness is approximately $0$ (roughly equal data on left and right sides).

Measures of Location (Central Tendency)

Measures of location identify the center or typical value of a dataset. There are five main measures of location:

1. Midrange:

   • Definition: The midpoint value halfway between the maximum and minimum observations.

   • Formula: $\text{Midrange} = \frac{\text{max} + \text{min}}{2}$

   • R Functions: (min(x) + max(x)) / 2

   • Pros: Simple to compute; does not require the complete dataset.

   • Cons: Utilizes only two observations; highly sensitive to extreme values or outliers.

2. Mode:

   • Definition: The most frequently occurring value in the dataset.

   • R Function: mfv(x) from the modeest package.

   • Pros: Intuitive to interpret; completely unaffected by extreme outliers.

   • Cons: A unique mode may not exist, or multiple modes may exist; may not accurately reflect the center of continuous distributions.

3. Median:

   • Definition: The middlemost observation when data points are arranged in ascending order.

   • R Function: median(x)

   • Pros: Insensitive to extreme outliers; represents the typical experience.

   • Cons: Ignores specific values of observations other than the middle positioning.

4. Mean:

   • Definition: The arithmetic average of all observations in the sample or population.

   • Formulas:

     • Sample Mean: \bar{x} = \frac{\text{sum}(x_i)}{n} = \frac{\text{\sum} x_i}{n}

     • Population Mean: \text{\mu} = \frac{\text{\sum} x_i}{N}

   • R Function: mean(x)

   • Pros: The most widely used measure of central tendency; incorporates all data points.

   • Cons: Highly sensitive to extreme values and skewness.

5. Trimmed Mean:

   • Definition: The mean calculated after discarding a specified percentage of extreme values from both the top and bottom of the sorted dataset.

   • R Function: mean(x, trim = 0.2) (e.g., dropping the outer $20\%$ on each side to keep the central $60\%$; dropping top/bottom $5\%$ to keep middle $90\%$ is standard in practice).

   • Pros: Retains the comprehensive nature of the mean while eliminating the distorting impact of outliers.

• Impact of Skewed Data on Central Tendency:

  • Extreme outliers pull the mean away from the median toward the tail.

  • Real-World Example: If Michael Jordan (net worth \text{\\$3.5\,billion}) enters a room of Geography graduates from UNC-Chapel Hill, the mean graduating salary increases dramatically, falsely implying an extremely high typical earnings level, whereas the median remains stable and accurate.

• Detailed Example — Target Daily Sales ($n = 11$ days):

  • Dataset ($x_i$ in thousands of dollars): $17, 23, 29, 33, 37, 37, 37, 40, 42, 48, 49$

  • Midrange Calculation: $\frac{49 + 17}{2} = \frac{66}{2} = 33$

  • Mode Calculation: Most frequent value is $37$

  • Median Calculation: $6^{\text{th}}$ observation in ordered sequence = $37$

  • Mean Calculation: \text{\sum} x_i = 392 \n    \rightarrow \bar{x} = \frac{392}{11} = 35.63636

  • Trimmed Mean Calculation ($20\%$ trim): Removes lower $20\%$ ($2$ values: $17, 23$) and upper $20\%$ ($2$ values: $48, 49$). Mean of remaining $7$ values ($29, 33, 37, 37, 37, 40, 42$) is $\frac{255}{7} = 36.42857$.

• R Execution Script and Output for Target Daily Sales:

dailysls <- c(17, 23, 29, 33, 37, 37, 37, 40, 42, 48, 49)
library(modeest)
(min(dailysls) + max(dailysls)) / 2  # [1] 33
mfv(dailysls)                       # [1] 37
median(dailysls)                    # [1] 37
mean(dailysls)                      # [1] 35.63636
mean(dailysls, trim = 0.2)           # [1] 36.42857

• Practice Problem — Employee Training Hours:

  • Dataset ($n=7$ reports): $80, 50, 200, 45, 40, 85, 50$

  • Ordered Dataset: $40, 45, 50, 50, 80, 85, 200$

  • Midrange: $\frac{200 + 40}{2} = 120$

  • Mode: $50$

  • Median: $50$

  • Mean: $\frac{550}{7} = 78.57$

  • Trimmed Mean ($15\%$ trim / dropping top and bottom observation): $\frac{45 + 50 + 50 + 80 + 85}{5} = \frac{310}{5} = 62$

  • Evaluation: The outlier ($200$\,hours) severely inflates the midrange ($120$) and mean ($78.57$). The median ($50$\,hours) or trimmed mean ($62$\,hours) best describes the true center of employee training hours.

Measures of Variability (Dispersion)

Measures of variability quantify the spread, dispersion, or scatter of observations around the center. There are five primary measures:

1. Range:

   • Definition: Total distance between maximum and minimum observations.

   • Formula: $\text{Range} = \text{max}(x) - \text{min}(x)$

   • Pros: Straightforward to calculate.

   • Cons: Uses only two observations; extremely vulnerable to outliers.

2. Interquartile Range (IQR):

   • Definition: Distance between upper quartile ($Q_U$ or $75^{\text{th}}$ percentile) and lower quartile ($Q_L$ or $25^{\text{th}}$ percentile).

   • Formula: $\text{IQR} = Q_U - Q_L$

   • R Functions: quantile(x, 0.75) - quantile(x, 0.25)

   • Pros: Measures spread of the middle $50\%$ of data; unaffected by extreme outliers.

   • Cons: Discards half of the dataset information.

3. Variance:

   • Definition: The average squared deviation of observations from their mean.

   • Population Variance Formula: \text{\sigma}^2 = \frac{\text{\sum} (x_i - \text{\mu})^2}{N}

   • Sample Variance Formula: s^2 = \frac{\text{\sum} (x_i - \bar{x})^2}{n - 1}

   • R Function: var(x)

   • Pros: Mathematically tractable, utilizes all observations.

   • Cons: Measured in squared units of the original variable, making direct intuitive interpretation difficult.

4. Standard Deviation:

   • Definition: The standard/average distance that data points deviate from the mean.

   • Population Standard Deviation Formula: \text{\sigma} = \text{\sqrt{\sigma^2}} = \text{\sqrt{\frac{\sum (x_i - \mu)^2}{N}}}

   • Sample Standard Deviation Formula: s = \text{\sqrt{s^2}} = \text{\sqrt{\frac{\sum (x_i - \bar{x})^2}{n - 1}}}

   • Sample Calculation Shortcut Formula: s = \text{\sqrt{\frac{\sum x_i^2 - \frac{(\sum x_i)^2}{n}}{n - 1}}}

   • R Function: sd(x)

   • Pros: Expressed in the exact same measurement units as the raw data; standard metric in statistics.

   • Cons: Sensitive to extreme outliers.

5. Coefficient of Variation (CV):

   • Definition: Ratio of standard deviation to the mean, measuring relative variability as a percentage.

   • Formulas:

     • Sample CV: $\text{CV} = \frac{s}{\bar{x}}$

     • Population CV: \text{CV} = \frac{\text{\sigma}}{\text{\mu}}

   • Pros: Unitless measure; allows direct comparison of variability across variables measured in different units or with drastically different mean magnitudes.

   • Cons: Sensitive to extreme outliers; meaningless if mean is zero or close to zero.

• Step-by-Step Hand Calculation Table for Target Daily Sales ($n = 11$):

  • Dataset (\text{thous}\): $17, 23, 29, 33, 37, 37, 37, 40, 42, 48, 49$

  • Mean: $\bar{x} = 35.63636364$

• Sample Variance Computation:     $s^2 = \frac{954.5454545}{11 - 1} = \frac{954.5454545}{10} = 95.45454545$

• Sample Standard Deviation Computation:     s = \text{\sqrt{95.45454545}} = 9.770084

• Shortcut Formula Verification:     s = \text{\sqrt{\frac{14924 - \frac{(392)^2}{11}}{10}}} = \text{\sqrt{\frac{14924 - 13969.454545}{10}}} = \text{\sqrt{\frac{954.5454545}{10}}} = 9.770084

• Verbal Descriptions of Dispersion Measures:

  • Standard Deviation in words: Target daily sales for $n = 11$\,days varies an average of $9.770$\,thousands of dollars away from the mean of $35.636$\,thousands of dollars.

  • Coefficient of Variation Computation: $\text{CV} = \frac{9.770084}{35.63636} = 0.2741605 \rightarrow 27.416\%$

  • Coefficient of Variation in words: Target daily sales for a sample of $n = 11$\,days varies an average of $27.416\%$ away from the mean of $35.63636$\,thousands of dollars.

• R Script and Output for Variability Measures:

dailysls <- c(17, 23, 29, 33, 37, 37, 37, 40, 42, 48, 49)
rang <- max(dailysls) - min(dailysls) # [1] 32
quantile(dailysls, 0.75)             # 75% : 41
quantile(dailysls, 0.25)             # 25% : 31
IQR <- quantile(dailysls, 0.75) - quantile(dailysls, 0.25) # 10
var(dailysls)                        # [1] 95.45455
sd(dailysls)                         # [1] 9.770084
cv <- sd(dailysls) / mean(dailysls)  # [1] 0.2741605
hist(dailysls, breaks = 3)          # Shows slight left-skewness

• Practice Problem — Sales Territory Comparison (Metro vs. Rural):

  • Metropolitan Areas ($n = 150$): $s = \\text{\$206}$, $\bar{x} = \\text{\$417}$

  • Rural Areas ($n = 150$): $s = \\text{\$94}$, $\bar{x} = \\text{\$121}$

  • Metro Standard Deviation in words: Metro area sales vary an average of $\\text{\$206}$\,thousand away from the mean sales of $\\text{\$417}$\,thousand.

  • Metro Coefficient of Variation: $\text{CV}_{\text{Metro}} = \frac{206}{417} = 0.4940 \rightarrow 49.40\%$

  • Rural Coefficient of Variation: $\text{CV}_{\text{Rural}} = \frac{94}{121} = 0.7769 \rightarrow 77.69\%$

  • Metro Coefficient of Variation in words: Metro area sales vary an average of $49.40\%$ away from the mean sales of $\\text{\$417}$\,thousand.

  • Relative Variability Evaluation: Metropolitan territories have lower relative variability ($49.40\%$ vs. $77.69\%$), even though their absolute standard deviation is higher ($\\text{\$206}$ vs. $\\text{\$94}$).

Distribution Shape and Skewness

• Skewness Types:

  • Left-Skewed (Negative Skew): Tail extends to the left toward smaller values; mean is pulled below the median.

  • Symmetric (No Skew): Left and right halves are mirror images; mean equals median equals mode.

  • Right-Skewed (Positive Skew): Tail extends to the right toward larger values; mean is pulled above the median.

• Analytical Implications of Skewness:

  • When skewness or extreme outliers exist, metrics like the mean and standard deviation become distorted.

  • In skewed distributions, reporting median and interquartile range (IQR) or using a trimmed mean is preferred.

• Adjusting Histogram Binning in R:

  • Default plot command: hist(dailysls)

  • Specifying bin intervals: hist(dailysls, breaks = 3)

Measures of Relative Standing and Data Standardization

• Quartiles:

  • $Q_L$ ($25^{\text{th}}$ percentile) and $Q_U$ ($75^{\text{th}}$ percentile) act as boundary thresholds dividing the data into quarters.

  • Can be visually represented using box plots.

• $z$-Scores (Standardization):

  • Definition: Measures how many standard deviations an observation $x_i$ lies above or below the mean.

  • Population Formula: z = \frac{x_i - \text{\mu}}{\text{\sigma}}

  • Sample Formula: $z = \frac{x_i - \bar{x}}{s}$

  • Example Calculation: Given height with mean \text{\mu} = 68\,inches and standard deviation \text{\sigma} = 3\,inches, the $z$-score for a height of $60$\,inches is $z = \frac{60 - 68}{3} = \frac{-8}{3} = -2.66667$. This height is $2.67$ standard deviations below the average.

• Empirical Rule (68-95-99.7 Rule):

  • Applies strictly to symmetric, bell-shaped (mound-shaped) distributions.

  • $\approx 68\%$ of observations lie within $1$ standard deviation of the mean (\text{\mu} \pm 1\text{\sigma} or $z \in [-1, 1]$).

  • $\approx 95\%$ of observations lie within $2$ standard deviations of the mean (\text{\mu} \pm 2\text{\sigma} or $z \in [-2, 2]$).

  • $\approx 99.7\%$ of observations lie within $3$ standard deviations of the mean (\text{\mu} \pm 3\text{\sigma} or $z \in [-3, 3]$).

• Detecting Outliers:

  • Outliers are extreme data points inconsistent with the main body of data.

  • $z$-Score Threshold Rules:

    • $|z| > 2$ ($z > 2$ or $z < -2$): Considered a possible outlier.

    • $|z| > 3$ ($z > 3$ or $z < -3$): Considered a definite outlier.

  • Example Evaluation: A $z$-score of $-2.6667$ is classified as a possible outlier, but not a definite outlier.

  • Outliers can also be identified visually via box plots.

Data Importation Procedures in RStudio

• GUI / Manual Method:

  1. Navigate to the Environment Pane in RStudio.

  2. Click Import Dataset.

  3. For .csv files: Select From Text (base).

  4. For .xls or .xlsx files: Select From Excel.

• Programmatic / Script Method:

  • Set working directory dynamically to script path:

setwd(dirname(rstudioapi::getActiveDocumentContext()$path))

*   Importing `.csv` file:

mydata <- read.csv("filename.csv")

*   Importing `.xlsx` file:

library(readxl)
mydata <- read_excel("filename.xlsx")