Relationships Between Variables and Least Squares Regression

Categorical Variables: Contingency Tables and Proportions

• Contingency tables are used to summarize the relationship between two categorical variables by displaying the counts of every possible outcome combination.

• Using the email data set ($3921$ emails), we analyze the relationship between the variable spam and the variable number (which identifies if an email contains none, small numbers under 1 million, or big numbers over 1 million).

• A contingency table (Table 2.24) summarizes these outcomes:

  • Spam and No number: $149$

  • Spam and Small number: $168$

  • Spam and Big number: $50$

  • Total Spam: $367$

  • Not spam and No number: $400$

  • Not spam and Small number: $2659$

  • Not spam and Big number: $495$

  • Total Not spam: $3554$

  • Column Totals: None ($549$), Small ($2827$), Big ($545$), Grand Total ($3921$).

• Row and column proportions facilitate comparisons:

  • Row Proportions: Calculated by dividing a cell count by its row total. For example, the proportion of spam emails with no numbers is $\frac{149}{367} = 0.406$.

  • Column Proportions: Calculated by dividing a cell count by its column total. For example, the proportion of emails with no numbers that are spam is $\frac{149}{549} = 0.271$.

• Comparing column proportions provides evidence of association. Because the spam rate varies across levels ($27.1\%$ for none, $5.9\%$ for small, and $9.2\%$ for big), the variables are associated.

• When choosing between row and column proportions, prioritize the one that answers the specific research question. To classify emails, column proportions are more useful because they show how the proportion of spam changes within each category of the explanatory variable (e.g., format or number).

• In the case of spam and format (plain text vs. HTML):

  • Plain text spam rate: $\frac{209}{1195} = 17.5\%$

  • HTML spam rate: $\frac{158}{2726} = 5.8\%$

Visualizing Qualitative Relationships: Stacked Bar Plots

• A stacked bar plot is a graphical representation of the information in a contingency table.

• In a standardized stacked bar plot, each bar is of equal height (representing $100\%$ or $1.0$), and the internal segments show the proportion of each outcome.

• Standardized plots are often clearer for identifying associations because they allow for a direct comparison of proportions across groups regardless of total counts.

• Variables are dependent if the proportions of one variable change across the groups of another; they are independent if the proportions remain constant.

Comparing Qualitative and Quantitative Variables: Box Plots and Histograms

• Relationships between a categorical and a quantitative variable are investigated by splitting the population into groups based on the categorical variable and comparing plots for each group.

• Side-by-side box plots are traditional tools for comparing centers (median) and spreads (IQR). For example, comparing median household income between counties with a population gain versus those with no gain:

  • Counties with population gain: Median income approx. $\$45000$, with higher variability.

  • Counties with no gain: Median income approx. $\$40000$, with lower variability.

• Hollow histograms use the outlines of histograms for each group on the same scale, which is useful for comparing distribution shapes, skewness, and identifying modes.

• Note: Observed relationships between variables in observational data, such as income and population gain, cannot be interpreted as causal connections.

Relationships Between Two Quantitative Variables: Scatterplots

• Scatterplots display the relationship between two numerical variables. Each point represents a single case.

• Associated variables (also called dependent variables) show a discernible pattern in the scatterplot.

• Independent variables show no evident relationship.

• Types of Association:

  • Positive Association: Points have a trend going up and to the right; a high value in one variable is associated with a high value in the other (e.g., poverty rate and federal spending per capita).

  • Negative Association: Points have a downward trend; a high value in one variable is associated with a low value in the other (e.g., homeownership vs. multi-unit structures).

• Nonlinear Relationships: Trends can be nonlinear, such as car price versus weight, which may show a curved "horseshoe" or decaying trend.

• Transformations: To model data with clusters near zero or non-constant trends, transformations (like log base e) are applied to improve the linearity and model fit.

The Correlation Coefficient $(r)$

• Correlation measures the strength of the linear relationship between two variables and ranges from $-1$ to $1$.

• Let $(x_1, y_1), ..., (x_n, y_n)$ be observations. The correlation is calculated following quadrant splitting around the sample means $(\bar{x}, \bar{y})$.

• Points in the upper-right or lower-left quadrants contribute a positive score to the relationship, while points in the other two quadrants contribute a negative score.

• The Sample Covariance is defined as:   $cov(x, y) = \frac{1}{n-1} \sum_{i=1}^{n} (x_i - \bar{x})(y_i - \bar{y})$

• Because covariance changes if units of measure change, we standardize it using standard deviations $s_x$ and $s_y$ to find Correlation $(r)$:   $r = \frac{1}{n-1} \sum_{i=1}^{n} \frac{(x_i - \bar{x})}{s_x} \frac{(y_i - \bar{y})}{s_y} = \frac{cov(x, y)}{s_x s_y}$

• Interpretations of $r$:

  • $r = 1$ or $-1$: Perfect linear relationship.

  • $r \approx 1$: Strong positive linear trend.

  • $r \approx -1$: Strong negative linear trend.

  • $r \approx 0$: No linear association (though a strong nonlinear relationship may still exist).

Special Visualizations: Time Plots and Maps

• Time Plots: Scatterplots where the x-axis represents time, often with points connected by a continuous line to show evolution over time (e.g., unemployment rates).

• Intensity Maps: Used for geographic data; different colors represent ranges of values for variables like poverty rates or federal spending per capita across counties.

  • These maps are useful for seeing geographic trends (e.g., the oil boom near the Canadian border or high poverty in the deep south).

Linear Combinations of Variables

• A linear combination of two variables $X$ and $Y$ is defined as $ax + by + c$, where $a$, $b$, and $c$ are fixed and known constants.

• Average of a Linear Combination:   The sample average for the combination is found by plugging in the sample averages for each variable:   $\bar{aX + bY + c} = a\bar{x} + b\bar{y} + c$

• Variability of Linear Combinations:

  • For independent variables:     $s^2_{ax+by+c} = a^2 \times s^2_x + b^2 \times s^2_y$

  • For dependent variables, we must include the covariance term:     $s^2_{ax+by+c} = a^2 \times s^2_x + b^2 \times s^2_y + 2abcov(x, y)$     In terms of correlation $r$, this is:     $s^2_{ax+by+c} = a^2 \times s^2_x + b^2 \times s^2_y + 2abrs_x s_y$

• Covariance Property:   $cov(a + bx + cy, z) = bcov(x, z) + ccov(y, z)$

• Correlation Property:   $r(a + bx, z) = sign(b)r(x, z)$. This implies the absolute value of correlation is invariant under linear transformations.

The Least Squares Regression Line

• Explanatory vs. Response Variables:

  • The response variable ($Y$), also called dependent or explorative, is the variable we wish to predict.

  • The explanatory variable ($X$), also called independent, is the variable used to predict the outcome.

• Prediction and $\hat{y}$:   Estimates for the response variable are denoted with a "hat" symbol: $\hat{y} = a + bx$.

• Residuals:   The residual ($e_i$) is the difference between the observed value and the predicted value:   $e_i = y_i - \hat{y}_i$

  • If the model underestimates an observation, the residual is positive.

  • If it overestimates, the residual is negative.

• Best Fit Criterion:   The least squares line minimizes the sum of the squared residuals ($e^2_1 + e^2_2 + ... + e^2_n$). This method is preferred because it is standard, easier to calculate than absolute magnitudes, and penalizes large errors more heavily via squaring.

Estimating and Interpreting Regression Parameters

• Slope calculation ($b$):   $b = r \frac{s_y}{s_x}$

• Intercept calculation ($a$):   Since the point $(\bar{x}, \bar{y})$ always lies on the least squares line:   $a = \bar{y} - b\bar{x}$

• Parameter Interpretations:

  • Slope ($b$): Represents the estimated average change in the response variable for every one-unit increase in the explanatory variable.

  • Intercept ($a$): Represents the estimated average value of the response variable if the explanatory variable is zero.

• ANOVA and Determination Coefficient $(R^2)$:

  • Total variations ($s^2_y$) can be split into variation explained by the model ($s^2_{\hat{y}}$) and residual variation ($s^2_e$):     $s^2_y = s^2_{\hat{y}} + s^2_e$

  • The Coefficient of Determination ($R^2$) indicates the proportion of total variation explained by the linear model:     $R^2 = \frac{s^2_{\hat{y}}}{s^2_y}$

  • Successfully, $R^2$ is also the square of the correlation coefficient ($r$).

Limitations of Linear Models

• Robustness: The least squares line is not robust against outliers because its components (means, standard deviations, correlation) are not robust.

• Extrapolation: Applying model estimates outside the range of the original data is dangerous and unreliable. For example, using an income-based gift aid model to predict aid for a family earning $\$1$ million might result in impossible negative aid values ($-\$18800$).

• Structural Mis-specification: Linear models are unsuitable for relationships that are clearly decaying or curvilinear, though transformations for $Y$ or $X$ (like $z = x^2$) can sometimes turn non-linear relationships into manageable simple linear models.