Introduction to Statistical Variation, Quartiles, and Standardization

Review of Measures of Central Tendency and Sample Variance

• Recap of Last Lecture: The discussion began with a review of mean, median, and mode, as well as the initial introduction to measuring variation through range and standard deviation.

• Calculating Sample Variance (Method 1):

  • Step 1: Find the Sample Mean ($x_{bar}$): In an example dataset containing 6 data points, the sample mean is identified as $50$.

  • Step 2: Calculate Deviations: For every data point ($x_i$), subtract the sample mean. These are known as the deviations: $(x_i - x_{bar})$. Note that subscript $i$ refers to the number's position in the dataset list, not an interval.

  • Step 3: Calculate Squared Deviations: Multiply each deviation by itself to obtain the squared deviation: $(x_i - x_{bar})^2$.

  • Step 4: Summation: Get the total sum of all squared deviations: $\sum (x_i - x_{bar})^2$.

  • Step 5: Variance ($s^2$): Divide the summation of squared deviations by $n - 1$ (where $n$ is the sample size).

  • Step 6: Standard Deviation ($s$): The standard deviation is the positive square root of the variance.

• Alternative Formula for Variation:

  • A second way to calculate the sum of squared deviations is:     $\sum (x_i - x_{bar})^2 = \sum x_i^2 - n \times (x_{bar})^2$

  • This formula can also be expressed as:     $\sum x_i^2 - \frac{(\sum x_i)^2}{n}$

• Interpreting Variation:

  • Standard deviation is the most widely used measure of variation.

  • The more the data is spread out, the larger the standard deviation will be.

  • City Comparison Example: In a comparison of two cities, City B ($S_B$) has a much larger standard deviation than City A ($S_A$). This implies that the data points for City B are distributed more widely than those in City A.

Linear Transformations of Data

• Effect on Mean:

  • If a constant $b$ is added to every data point in a dataset ($y_i = x_i + b$), the new sample mean ($y_{bar}$) will increase by that same value $b$.

• Effect on Variance and Standard Deviation:

  • Additive Shifts: Shifting data points to the left or right by adding or subtracting a number does not change the spread or variation. Therefore, adding a constant does not affect the variance or standard deviation.

  • Multiplicative Scaling: Multiplying the data by a number does affect the variation. If you multiply by a large number (e.g., $2$), the data becomes more spread out. If you multiply by a very small number, the data points become more clustered.

Quartiles and Interquartile Range (IQR)

• Process for Finding Quartiles:

  • Step 1: Order the Data: Arrange all values in increasing order.

  • Step 2: Find the Median ($Q_2$): Determine the middle value of the dataset. If there is an even number of data points (e.g., $12$), the median is the average of the two middle numbers.

  • Step 3: Divide into Halves: Split the ordered dataset into a bottom half and a top half.

    • If there is an odd number of data points (e.g., $7$), exclude the median from both halves.

    • Example: For data points $3, 4, 5, 5, 6, 7, 8$, the median is $5$. The bottom half is $3, 4, 5$ and the top half is $6, 7, 8$.

  • Step 4: Find $Q_1$: Calculate the median of the bottom half of the data.

  • Step 5: Find $Q_3$: Calculate the median of the top half of the data.

• The Interquartile Range (IQR):

  • The IQR is a measure of variation that is less affected by outliers than the range.

  • Formula: $IQR = Q_3 - Q_1$

Identifying Potential Outliers

• Definition: Outliers are observations that fall far outside the overall pattern of the data, being either significantly larger or smaller than the rest of the points.

• The 1.5 IQR Rule (Fences):

  • Potential outliers can be identified by building two "fences" (lower and upper limits).

  • Lower Limit: $Q_1 - 1.5 \times IQR$

  • Upper Limit: $Q_3 + 1.5 \times IQR$

  • Any data point falling outside these two limits is considered a potential outlier.

• Example Calculations:

  • HILD Temperature Dataset:

    • Median ($Q_2$): $70.5$

    • $Q_1$: Approximately $70$

    • $Q_3$: $71.5$

    • $IQR$: $71.5 - 70 = 1.5$

    • $1.5 \times IQR = 2.25$ (Note: Transcript mentions $3.35$, possibly a calculation variant or error).

  • Second Exercise:

    • Median ($Q_2$): $278$

    • $Q_1$: $268$

    • $Q_3$: $301$

    • Data points such as $341$ may be checked against limits to see if they are potential outliers.

Five-Number Summary and Box Plots

• Five-Number Summary:

  • The dataset is described using five specific values: Minimum, $Q_1$, Median ($Q_2$), $Q_3$, and Maximum.

  • These numbers divide the data into four quarters, each containing $25\%$ of the observations.

• Box Plots:

  • A visual tool to display the five-number summary and the variation within the data.

  • It is particularly useful for comparing several different samples or datasets.

  • Modified Box Plot: In a modified version, outliers are indicated with stars ($*$). Whiskers extend to the largest and smallest values that are not outliers.

Measures for Population Data

• Statistics vs. Parameters:

  • Statistics: Measures calculated from a sample (e.g., sample mean $x_{bar}$, sample variance $s^2$).

  • Parameters: Measures calculated from the entire population (e.g., population mean $\mu$, population variance $\sigma^2$).

• Notation Differences:

  • Mean: Sample is $x_{bar}$; Population is $\mu$ (mu).

  • Variance: Sample is $s^2$; Population is $\sigma^2$ (sigma squared).

  • Standard Deviation: Sample is $s$; Population is $\sigma$ (sigma).

  • Size: Sample size is $n$; Population size is $N$.

• Formula Difference for Variance:

  • For sample variance, the sum of squared deviations is divided by $n - 1$.

  • For population variance, the sum of squared deviations is divided by the total population size $N$.

Standardization and Z-Scores

• Standardized Variable ($z$):

  • A standardized version of a variable $x$ is created to allow for fair comparisons between datasets with different units (e.g., meters vs. inches).

  • Formula: $z = \frac{x - \mu}{\sigma}$

  • Centering: Subtracting the mean ($\mu$) ensures the new mean of the standardized data is $0$.

  • Scaling: Dividing by the standard deviation ($\sigma$) ensures the new standard deviation of the standardized data is $1$.

• Interpreting Z-Scores:

  • If $z = 0$, the value is equal to the mean ($x = \mu$).

  • If $z = -2$, the value is two standard deviations below the mean.

  • Example Calculation: Given $\mu = 3$ and $\sigma = 2$:

    • For a value $x$: $\text{z-score} = \frac{x - 3}{2}$.

    • If the score is $0$, $x$ is the mean.

    • Negative scores indicate the number of deviations below the mean.