Terms and Definitions

Probability

A number from 0 to 1 for how likely an event is. 0 means impossible and 1 means certain

Event

Something that can happen, such as "a student passes."

Disjoint (mutually exclusive)

A and B cannot both happen, so P(A and B) = 0 and P(A or B) = P(A) + P(B)

Histogram

Bars showing how many values fall in each range. It shows the shape of the data: symmetric, skewed, or with several humps

Boxplot

The box is the middle 50% of the data, the line inside is the median, the whiskers show the typical range, and separate dots are outliers. It does not show the mean

Stem-and-leaf plot

A text version of a histogram that keeps the actual digits

Scatterplot

One point per observation, with x on the horizontal axis and y on the vertical axis

Pairs plot

A grid of scatterplots, one for every pair of variables

Mean

The average. Pulled towards outliers and toward the long tail of skewed data

Median

The middle value when the data is in order. It is resistant to outliers.

Skew

A lopsided distribution. Right skew has a long tail of large values, and the mean is above the median. Left skew is the reverse.

Standard Deviation

A typical distance of values from the mean. It is sensitive to outliers

Quartiles and IQR

Q1 and Q3 are the 25th and 75th percentiles, and IQR = Q3 − Q1 is the spread of the middle 50%. It is resistant to outliers.

Outlier

A point far from the rest.

Pearson correrlation

The amount of straight-line relationship, from -1 to 1. +1 is a perfect upward line, -1 is a perfect downward line and 0 means no straight-line relationship. A value near 0 does not rule out a curve

Rank

Each value replaced by its position from smallest (1) to largest

Spearman correlation

The Pearson correlation of the ranks. It measures whether y consistently rises or falls with x, and is resistant to outliers.

Function

A relationship that gives exactly one y for each x

Operator

A relationship that gives a distribution of y values for each x. The average at an x can fall where no data sits.

“Uninteresting” data

A featureless blob with no curve, clusters, trend, or outliers. The multivariate normal is the standard example.

Regression

Modeling how y depends on x. The basic form is y = E(Y given x) + ε

E (y|x)

The expected (average) value at y at a particular x. The vertical bar means “given.”

ε (epsilon)

Random error, the random scatter of an individual y around the average at its x.

Fitted value (model prediction)

The y the line predicts at a given x

Residual

Observed - model. Positive means the point is above the line, negative means it is below, and 0 means it is exactly on the line. Residuals are what you can measure of epsilon.

Taylor expansion idea

Any smooth curve looks like a straight line if you zoom in far enough, which is why a line is a reasonable local model.

Extrapolation

Predicting at x values outside the range of the data. It is risky because the line is only a local approximation

Anscombe’s example

Datasets with the same summary numbers and fitted line but very different pictures. The lesson is to plot the data and compare it with the program's output

Subject-matter Theory

What is already known about the topic. If it is solid, use it to predict what the output should look like, then check the data against that

Functional Modeling

Modeling a single y for each x. Linear and nonlinear fits are both functional

Linear

Gives one y per x, straight line

Nonlinear

Gives one y per x, curved line

Non-functional

One y per x does not describe the data. Two separate bands of y at the same. You need the distribution of y at each x.

Geometry of regression

The fitted values are a projection of y onto the space spanned by the x variables (including the intercept), and the residuals are what is left over.

Statistical Inference

Using data from a sample to learn about the population

Statistic

A number computed from a sample, such as the sample proportion p-hat or the sample mean x-bar

Parameter

The true value for the whole population. It is fixed but unknown

Point estimate

A statistic used as the best guess for a parameter. It will not match the parameter exactly

Sampling Distribution

The distribution of a statistic over many samples of the same size from the population

Standard Error

The standard deviation of a statistic, meaning how much it varies from sample to sample. A small SE means a precise estimate.

Central Limit Theorem

Averages (and sums) tend toward a normal distribution as the sample size grows, and the uncertainty of an average shrinks like 1 over the square root of n

Unbiased

The mean of the sampling distribution equals the parameter

Minimum Variance

The statistic has the smallest variance of all the statistic you might use. For the exponential distribution the median is biased relative to the mean.

Bootstrap

A way to estimate how much a statistic varies when you only have one sample

Bootstrap Sample

A random sample taken with replacement from the original sample, the same size as the original. It uses only original values, and some values repeat while others are left out.

Bootstrap Statistic

The statistic computed on one bootstrap sample

Bootstrap Distribution

The distribution of many bootstrap statistics. Its spread estimates the standard error.

Golden Rule of Bootstrapping

Bootstrap statistics are to the original sample statistic as the original sample statistic is to the population parameter.

Bootstrap for Regression

Resample the (x, y) pairs together, as whole rows, with replacement

Confidence Interval

A range of plausible values for a parameter. The parameter is fixed, while the statistic and the interval are random. 95% of 95% confidence intervals capture the truth.

95% CI rule of thumb

Statistic ± 2 × SE. The 2 × SE part is the margin of error.

Percentile Method

Take the 2.5th and 97.5th percentiles of the bootstrap distribution as the interval.

Pivotal Method

Flip the bootstrap quantiles around the statistic and scale by an estimate of the standard deviation, using a bootstrap of the bootstrap (about 200 resamples). At n = 10 it covered 93% in the slides' simulation, versus 85.5% for percentile and 86% for the t method.

Hypothesis testing

Assume the null hypothesis, then ask whether the statistic you observed would be rare under it. If it is rare, the result favors the alternative.

p-value (Pr(>|t|) in summary)

How rare your result would be if the true coefficient were 0. A small p-value means the coefficient is unlikely to be 0.

Cross Validation

Leave some data out of the fit, then predict it, to keep from fooling yourself. The slides' ethic is to bend over backwards to disprove yourself (Feynman).

Training set and test set

Fit the model on the training set, then predict the test set, which the model has not seen.

Overfitting

A model with extra, unneeded terms fits the training data but predicts new data worse, because the extra terms add noise.

K-fold cross validation

Split the data into k disjoint pieces. Each piece is the test set once while the rest is the training set. Add up the squared prediction errors. The split is random, so the answer changes a little each run.

Leave-one-out and PRESS

Set k = N so every point is left out once. There is no randomness. PRESS is the sum of the squared leave-one-out residuals, and smaller is better.

Leave-one-out residual

Residual ÷ (1 − leverage). This trick gives the answer without refitting N times.

Leverage (Hat Values)

How much each point contributes to its own prediction. It is the diagonal of the hat matrix H, and hat returns it.

Hat matrix H

The matrix that turns the observed y into the fitted y: fitted y = Hy.

Parsimony (KISS)

Prefer smaller models, because they are generally better predictors even when a larger one might be "correct."

R-squared

The fraction of variation the model explains. It never goes down when you add a variable, even pure noise, so it is the wrong tool for choosing variables.

Adjusted R-squared

R-squared with a penalty for extra variables. In the slides' noise-column example it dropped while R-squared rose.

Cp

A variable-selection statistic that compares a subset model with the full model. When the subset fits as well as the full model, Cp is close to p, the number of parameters in the subset.

Interaction (x1 * x2)

Lets the slope for one variable differ by the level of the other, such as a separate age slope for each gender.

Parallel slopes (x1 + x2)

The groups share one slope and differ only in height.

Smoothing parameter (lambda)

Controls how smooth a fitted curve is. gauss.reg1a chooses it with PRESS

Residual Standard Error

The typical size of the residuals, in the units of y.

Multicollinearity

Predictor columns that are nearly linear combinations of each other, so they carry almost the same information. The matrix (X-transpose X) inverse becomes unstable.

Symptoms of Multicollinearity

Big standard errors, few or no significant t-stats, yet the model still predicts well. Coefficients can be huge with opposite signs (the slides showed −15 and +14).

Eigenvalues ($eigen)

In the collinear case, some eigenvalues of the x covariance matrix become very small, which is the sign of near-redundant columns.

Dealing with collinearity

Combine variables (principal components), drop variables one at a time (t-stats can change a lot), or regularize. The best approach is to do both.

Regularization

Adding a penalty that keeps coefficients from growing, which stabilizes them.

Ridge regression

Minimizes squared error plus λ times the sum of the squared coefficients. It shrinks coefficients toward 0 and stabilizes them, but none reach exactly 0. In matrix form it uses (X-transpose X + kI) in place of (X-transpose X).

Lasso

Constrains the sum of the absolute values of the coefficients (the L1 norm) to be less than t. Variables drop out to exactly 0 as t shrinks, so it selects variables.

Ridge vs lasso

Ridge uses squared coefficients and only shrinks. Lasso uses absolute values and can remove variables.

LARS

Least Angle Regression, starts with the variable most correlated with y, nudges its coefficient up, and adds the next variable when it ties in correlation with the residuals. It computes the lasso path and gives df and Cp for choosing a stopping point.

glmnet

Minimizes squared error plus a mix of the ridge and lasso penalties. alpha = 0 is ridge, alpha = 1 is lasso, and values in between combine them.

Stein’s Theorem

In more than 2 dimensions you can construct a biased estimator with a smaller mean squared error than the unbiased one. This is why shrinking can help.

Biased Estimator

One whose average is not the true value. A little bias can buy a large drop in variance.

Random Variable

A numeric quantity that changes from trial to trial in a random process, such as the sum of two dice

Discrete Random Variable

Has a countable set of possible values (counting)

Continuous Random Variable

Takes values within an interval (measurement), such as the time to run a mile.

Probability function, p(x)

Assigns a probability to each value of a discrete random variable. The probabilities add to 1.

Binomial Random Variable

Counts successes in a fixed number n of independent trials, each with the same success probability p.

Binomial Coefficient

n! / (k! (n − k)!), the number of ways to arrange k successes among n trials. For 2 successes in 5 trials it is 10.

Binomial Probability

P(X = k) = (binomial coefficient) × p^k × (1 − p)^(n − k). For 3 sixes in 5 dice rolls it is about 0.0322. In R: dbinom, pbinom, rbinom.

Normal Distribution, X ~ N(μ, σ)

A symmetric bell-shaped curve. The mean μ is the center and the standard deviation σ is the spread. Areas under the curve are probabilities, found with pnorm (also dnorm, rnorm).

Rate (λ)

A rate per time or per area, so it scales up with the length of time or the size of area.

Binomail to Poisson

When n gets large with n × p held constant, the binomial turns into a Poisson.

R Function Letters

d gives the probability at a value, p gives the probability up to a value, and r generates random values

Binary (yes/no) response

An outcome with two values, such as default or no default, or a drought year or not.

GLM (Generalized Linear Model, glm)

Regression that is not limited to sums of squares. It only needs a base likelihood, and the family argument says what kind of response you have.

Logit Model

glm(y ~ x, family = binomial(link = logit)). It models the probability p = e^(xβ) / (1 + e^(xβ)), which stays between 0 and 1.

Probit Model

The same setup with link = probit, which uses the normal curve for the link instead.

Likelihood

How probable the observed data is under a model. GLMs fit by maximizing it.

Deviance

2 × (log likelihood of a base model minus log likelihood of your model). It lets you compare models directly. Lower is a better fit.