Unit 2: Examining Relationships Study Guide

Examining Relationships in Statistics

• Comparing Populations vs. Examining Relationships:

  • In statistics, researchers often wish to compare two or more different populations with respect to the same variable. Side-by-side boxplots are commonly used tools for making these comparisons between samples.

  • However, the focus often shifts to examining relationships between several variables for the same population.

  • When examining the relationship between two variables, two primary situations arise:

    • The researcher is simply interested in the nature of the relationship.

    • One of the variables is thought to explain or predict the other.

Explanatory and Response Variables

• Definitions:

  • Response Variable: Denoted by $Y$, it is a variable that takes values representing the outcome of a study.

  • Explanatory Variable: Denoted by $X$, it is a variable that helps explain the outcome of the study.

• Example 1 (Caffeine and Sleep):

  • Study: Researchers interviewed 300 adults to see if caffeine in coffee helps keep people awake. They asked how many cups of coffee the adults drink on an average day and how many hours of sleep they get at night.

  • Response Variable ($Y$): Hours of sleep.

  • Explanatory Variable ($X$): Number of cups of coffee per day.

• Example 2 (English and Math Performance):

  • Study: A psychology professor located 450 students at a large university who took the same introductory English and Math courses. The professor compared their percentage grades at the end of the semester.

  • Relationship: In this case, there is no defined explanatory or response variable. The researcher is simply interested in the nature of the relationship (e.g., are people strictly left- or right-brained, or do they excel in both?).

Scatterplots

• Utility: Scatterplots are the best way to display the relationship between two quantitative variables. They display values measured on the same individuals.

• Plotting Rules:

  • Each individual’s data appears as a single point on the plot.

  • If an explanatory and response variable exist, the explanatory variable is plotted on the x-axis and the response variable on the y-axis.

  • If no such distinction exists, the choice of axes is arbitrary.

• Example Case Study: Classes Missed vs. Exam Score:

  • Data ($n=8$):

    • Student 1: 5 classes missed, 60 exam score.

    • Student 2: 2 classes missed, 95 exam score.

    • Student 3: 6 classes missed, 73 exam score.

    • Student 4: 10 classes missed, 56 exam score.

    • Student 5: 1 classes missed, 81 exam score.

    • Student 6: 8 classes missed, 45 exam score.

    • Student 7: 4 classes missed, 82 exam score.

    • Student 8: 2 classes missed, 78 exam score.

Examining Scatterplots: Four Key Characteristics

1. Direction:

   • Negative Association: An above-average value of one variable tends to be accompanied by a below-average value of the other (e.g., more classes missed leads to lower exam scores). The points slope downward from left to right.

   • Positive Association: The pattern of points slopes upward from left to right, meaning above-average values of one variable accompany above-average values of the other.

2. Form:

   • Refers to the general shape of the relationship.

   • Linear Relationship: When a straight line serves as a good approximation for the relationship.

   • Other Forms: Relationships can also be quadratic, logarithmic, or exponential.

3. Strength:

   • Determined by how closely the points lie to a simple form like a straight line.

   • If all points fall close to the line, the relationship is strong.

   • If points appear "randomly scattered" and fall far from the approximating line, the linear relationship is weak.

4. Outliers:

   • Observations can be outlying in the x-direction, the y-direction, or both.

   • A bivariate outlier occurs when an observation falls outside the general pattern of points, even if it is not an extreme value for either variable alone.

Assessing Strength with Correlation

• Visual Limitations: Scatterplots can be misleading based on the scale used. The same data can appear stronger or weaker depending on the values used on the x- and y-axes.

• Definition of Correlation Coefficient ($r$): A numerical measure that describes the direction and strength of a linear relationship between two quantitative variables measured for $n$ individuals.

• Formulas:

  • Standard formula: $r = \frac{1}{n-1} \times \frac{\sum (x_i - \bar{x})(y_i - \bar{y})}{s_x s_y}$

  • Alternative notation: $r = \frac{1}{n-1} \times \textstyle\sum \left( \frac{x_i - \bar{x}}{s_x} \right) \left( \frac{y_i - \bar{y}}{s_y} \right)$

• Steps for Calculation:

  1. Calculate the standard deviations $s_x$ and $s_y$.

  2. Calculate the deviations $(x_i - \bar{x})$ and $(y_i - \bar{y})$ for each observation.

  3. Multiply corresponding deviations for $x$ and $y$: $(x_i - \bar{x})(y_i - \bar{y})$.

  4. Sum these products: $\sum (x_i - \bar{x})(y_i - \bar{y})$.

  5. Divide the result by $(n - 1)s_x s_y$.

• Example Calculation Snippet (Classes Missed vs. Exam Score):

  • Sum of products $\sum (x_i - \bar{x})(y_i - \bar{y}) = -294.5$

  • Result: $r = -0.8169$

Properties of Correlation

• Range: $r$ always falls between $-1$ and $1$, inclusive.

• Direction Icons: Positive $r$ indicates positive association; negative $r$ indicates negative association.

• Extreme Values: $r = 1$ or $r = -1$ occur only in perfect linear relationships where all points lie exactly on a line.

• Units: Correlation has no units; it is a dimensionless number.

• Roles of Variables: Correlation makes no distinction between explanatory ($X$) and response ($Y$) variables; the result is the same regardless of which is which.

• Scale Invariance: Changing the units of measurement (e.g., feet to meters) does not change the correlation.

• Form Specificity: $r$ measures only the strength of linear relationships. It is useless for non-linear forms.

• Sensitivity: Correlation is not robust; it is strongly affected by outliers.

Linear Regression

• Definition: A regression line is a straight line describing how a response variable $Y$ changes as an explanatory variable $X$ changes. It is used to predict values of $Y$ for a given $X$.

• Model Equation: $\hat{y} = b_0 + b_1 x$

  • $\hat{y}$: The predicted value of $Y$ for a given $X$.

  • $b_1$: The slope of the line.

  • $b_0$: The intercept of the line.

• The Best Line: The "best" line is found using the least squares method.

• Least Squares Regression Line (LSRL): The line that minimizes the sum of squared deviations in the vertical direction (residuals).

  • $\min \sum (y_i - \hat{y}_i)^2$

• Formulas for Coefficients:

  • Slope: $b_1 = r \times \frac{s_y}{s_x}$

    • Definition: The predicted increase in $y$ when $x$ increases by one unit.

  • Intercept: $b_0 = \bar{y} - b_1 \bar{x}$

    • Definition: The predicted value of $y$ when $x = 0$.

Coefficient of Determination ($r^2$)

• Definition: The value $r^2$ is the fraction (or percentage) of the variation in the values of $Y$ that is accounted for by the least squares regression on $X$.

• Interpreting $r^2$:

  • If $r = 1$ or $-1$, then $r^2 = 1$, meaning regression on $X$ accounts for 100% of the variation in $Y$.

  • If $r = 0$, then $r^2 = 0$, meaning regression on $X$ tells us nothing about $Y$.

• Calculating $r$ from $r^2$: If software provides only $r^2$, $r$ is the positive square root if there is a positive association, and the negative square root if there is a negative association.

Regression Example: Apartment Rent and Size

• Data: Sample of 10 apartments measuring Size $X$ in square feet and Monthly Rent $Y$ in dollars.

  • Data pairs (Size, Rent): (770, 1270), (650, 990), (925, 2230), (850, 1295), (575, 860), (860, 1925), (800, 1575), (1000, 1790), (730, 1580), (900, 1550).

• Calculated Values:

  • Slope ($b_1$): $2.60$

  • Intercept ($b_0$): $-589.10$

  • Equation: $\hat{y} = -589.10 + 2.60x$

• Interpretations:

  • Slope: For every one square foot increase in size, the predicted monthly rent increases by $\$2.60$.

  • Intercept: In this context, the intercept is statistically meaningless because an apartment cannot be 0 square feet, and rent cannot be negative.

  • Determination ($r^2$): If $r = 0.8031$, then $r^2 = 0.645$. Thus, 64.5% of the variation in monthly rent is accounted for by the regression on size.

• Prediction Example: For an 860 square foot apartment:

  • $\hat{y} = -589.10 + 2.60 \times 860 = 1646.90$

Residuals and Extrapolation

• Residuals: The difference between an observed value of the response variable and the value predicted by the regression line.

  • Formula: $\text{Residual} = y_i - \hat{y}_i$

  • A positive residual indicates the observed point falls above the regression line.

  • A negative residual indicates the observed point falls below the regression line.

  • Example: For the 860 sq ft apartment (actual rent $\$1925$), the residual is $1925 - 1646.90 = 278.10$.

• Extrapolation: The process of predicting a value of $Y$ for a value of $X$ that is outside the range of the observed data.

  • Danger: We have no evidence that the linear relationship continues to hold outside the observed range (e.g., predicting rent for a 1250 sq ft apartment when data only goes up to 1000 sq ft).

  • Guideline: Extrapolation should be avoided whenever possible.

Outliers and Influential Observations

• Types of Outliers:

  1. Outlier in the Y-direction: Points far from the line vertically. Generally has little effect on the position of the regression line.

  2. Bivariate Outlier: Points that fall outside the general pattern but are not extreme in $X$ or $Y$ alone. Generally has little effect on the regression line.

  3. Outlier in the X-direction: Points that are extreme horizontally. Often have a strong effect on the regression line.

• Influential Observation: An observation is influential if removing it from the dataset would dramatically alter the position of the regression line and the value of $r^2$. Outliers in the x-direction are frequently influential.

• Example of Influence: If a 1500 sq ft apartment with rent $\$1790$ were added to the dataset, the value of $r^2$ could drop significantly (e.g., from $0.645$ to $0.316$).

Association vs. Causation

• The Golden Rule: Association does not imply causation.

• Lurking Variables: A variable that helps explain the relationship between variables in a study, but which is not itself included in the study.

  • Apartment Example: The neighborhood (nicer parts of the city) could be a lurking variable explaining both larger sizes and higher rents.

  • Student Example: A student's effort is a lurking variable that explains both missing fewer classes and obtaining higher exam scores.

• Drug Study Case:

  • A study finds a correlation of $r = 0.85$ between the percentage of teenagers trying marijuana ($X$) and hard drugs ($Y$) across American cities.

  • Lurking Variable: The availability of drugs in different cities. High availability increases the likelihood of trying both marijuana and hard drugs.

• Establishing Causation: The best way to avoid lurking variables and establish causation is through an Experiment rather than an Observational Study.

  • In an experiment, explanatory variables are randomly assigned to units. Random assignment "diversifies away" the effects of lurking variables across the groups.

Categorical Variables in Scatterplots

• Sometimes a scatterplot can display two or more distinct relationships within the same plot.

• Example (Golfers): A plot of Average Driving Distance ($X$) vs. Average Score ($Y$) might appear non-linear initially.

• Sub-groups: When the data points are identified by a categorical variable (e.g., Male vs. Female golfers), the scatterplot may reveal two distinct linear relationships.

• Analysis Recommendation: In such cases, a separate regression line should be fit to each distinct population to provide an accurate description of the relationship.