MAT 3522: Linear Models and Design of Experiments - Review Flashcards

These vocabulary flashcards cover random vectors and matrices, simple and multiple linear regression, goodness of fit measures, regression assumptions, and statistical inference as presented in the MAT 3522 lecture notes.

Random Vector

A vector whose components are random variables, such as $X = \begin{pmatrix} X_1 \\ X_2 \end{pmatrix}$.

Mean Vector

For a random vector $X$, the mean vector is defined as $E(X) = \mu = \begin{pmatrix} E(X_1) \\ E(X_2) \\ \vdots \\ E(X_p) \end{pmatrix}$.

Covariance

A measure of the direction of the linear relationship between two random variables $X$ and $Y$, defined by $Cov(X, Y) = E[(X - \mu_X)(Y - \mu_Y)]$ or $E(XY) - E(X)E(Y)$.

Positive Covariance ($Cov(X, Y) > 0$)

Indicates that the variables $X$ and $Y$ tend to increase together.

Negative Covariance ($Cov(X, Y) < 0$)

Indicates that one variable tends to increase while the other variable decreases.

Zero Covariance ($Cov(X, Y) = 0$)

Indicates that there is no linear relationship between the two variables.

Variance-Covariance Matrix ($\Sigma$)

A symmetric matrix where the diagonal elements are variances ($\sigma_{ii} = Var(X_i)$) and off-diagonal elements are covariances ($\sigma_{ij} = Cov(X_i, X_j)$).

Positive Semi-definite

A mathematical property that all covariance matrices possess.

Simple Linear Regression

A statistical technique used to model the relationship between one response variable and one explanatory variable using a straight-line population model: $Y_i = \beta_0 + \beta_1 X_i + \varepsilon_i$.

Intercept ($\beta_0$)

The predicted value of the response variable $Y$ when the explanatory variable $X$ is zero.

Slope ($\beta_1$)

The amount the predicted value of $Y$ increases for every one-unit increase in the explanatory variable $X$.

Estimated Regression Equation

The fitted model using sample data given by $\hat{Y} = b_0 + b_1 X$, where $b_0$ and $b_1$ are the estimated intercept and slope.

Homoscedasticity

The assumption for regression that the variance of the error terms remains constant, mathematically expressed as $Var(\varepsilon_i) = \sigma^2$.

Least Squares Estimation

A method of estimating regression coefficients by minimizing the sum of squared residuals ($SSE = \sum_{i=1}^{n} (Y_i - \hat{Y}_i)^2$).

Residual ($e_i$)

The difference between the observed value and the predicted value, defined as $e_i = Y_i - \hat{Y}_i$. It measures the prediction error.

Multiple Linear Regression

A statistical technique used to model the relationship between one response variable and two or more explanatory variables using the model $Y_i = \beta_0 + \beta_1 X_{1i} + \beta_2 X_{2i} + \dots + \beta_k X_{ki} + \varepsilon_i$.

Perfect Multicollinearity

A situation where explanatory variables are perfectly linearly related (e.g., $X_2 = 2X_1$), making it impossible to uniquely estimate regression coefficients.

Design Matrix ($X$)

In the matrix form $Y = X\beta + \varepsilon$, a matrix where the first column is ones for the intercept term and the remaining columns contain the explanatory variable data.

Least Squares Estimator (Matrix Form)

The formula used to estimate all regression coefficients simultaneously: $\hat{\beta} = (X'X)^{-1} X'Y$.

Goodness of Fit

The extent to which the fitted regression equation adequately describes the relationship between the response and explanatory variables.

Total Sum of Squares (SST)

Measures the total variation present in the response variable, defined as $SST = \sum_{i=1}^{n} (Y_i - \bar{Y})^2$.

Regression Sum of Squares (SSR)

Measures the variation in the response variable that is explained by the regression model, defined as $SSR = \sum_{i=1}^{n} (\hat{Y}_i - \bar{Y})^2$.

Error Sum of Squares (SSE)

Measures variation not explained by the regression model, defined as $SSE = \sum_{i=1}^{n} (Y_i - \hat{Y}_i)^2$ or $\sum_{i=1}^{n} e_i^2$.

Coefficient of Determination ($R^2$)

The proportion of variation in the response variable explained by the regression model, calculated as $R^2 = \frac{SSR}{SST}$.

Adjusted Coefficient of Determination ($R^2_{adj}$)

A measure of model performance that rewards useful variables and penalizes unnecessary variables, defined as $R^2_{adj} = 1 - \left( \frac{SSE/(n - k - 1)}{SST/(n - 1)} \right)$.

Residual Plot

A graph of residuals ($e_i$) against fitted values ($\hat{Y}_i$) used as a diagnostic tool for checking regression assumptions.

Heteroscedasticity

A pattern in a residual plot exhibiting a funnel shape, indicating that the constant variance assumption has been violated.

t-Test for Slope

A hypothesis test where $H_0: \beta_1 = 0$ against $H_1: \beta_1 \neq 0$ to determine if an explanatory variable significantly affects the response.

Mean Square Error (MSE)

An estimator of error variance ($\sigma^2$), calculated as $MSE = \frac{SSE}{n - 2}$ in simple regression or $MSE = \frac{SSE}{n - k - 1}$ in multiple regression.

Overall F-Test

A test of the significance of the entire regression model where $H_0: \beta_1 = \beta_2 = \dots = \beta_k = 0$ against $H_1: \text{At least one } \beta_j \neq 0$, using the statistic $F = \frac{MSR}{MSE}$.

Confidence Interval

A range of plausible values for an unknown population parameter, constructed using the general form: $Estimate \pm (Critical Value) \times (Standard Error)$.

Partial Effect

An interpretation of a regression coefficient in multiple regression representing the effect of one variable while holding all other variables constant.