(L11/Ch9) T-Tests

Key Terms

Alpha-level (α-level)

The probability of making a Type I error (usually .05).

Alternative hypothesis (experimental hypothesis)

The prediction that there will be an effect, for example that your manipulation will have some effect or that variables will relate to each other.

Assumption of normality

It relates to the errors of the model (the residuals), not the observed data. If errors are not normal, estimates stay unbiased, but standard errors, confidence intervals and significance tests need normal errors or a large sample.

Beta-level (β-level)

The probability of making a Type II error. Cohen (1992) suggests a maximum of 0.2.

Binary variable/Dichotomous

A categorical variable with only two mutually exclusive categories.

Bootstrap

A technique for estimating the sampling distribution of a statistic by taking repeated samples, with replacement, from the data. The SD of the resulting distribution estimates the standard error, from which confidence intervals and significance tests are computed.

Boxplot (box-whisker diagram)

Shows the median in the centre of a box that spans the middle 50% of scores (the interquartile range). Whiskers extend to the highest and lowest extreme scores.

Categorical variable

A variable made up of categories of objects or entities.

Central limit theorem

With large samples (above about 30) the sampling distribution is normal whatever the shape of the population. For small samples the t-distribution approximates it better. The standard error of the mean equals s divided by the square root of N.

Cohen's d

An effect size that expresses the difference between two means in standard deviation units, d = (X1 minus X2) divided by s.

Confidence interval

A range around a statistic that is believed to contain the true value in a certain proportion of samples (for example 95%). In the other samples it does not, and you cannot know which case you are in.

Degrees of freedom

Essentially the number of entities free to vary when estimating a parameter. They matter for significance tests and determine the exact form of the distribution of test statistics such as t.

Dummy variables

A way of recoding a categorical variable into dichotomous variables that take only the values 0 or 1. One group is chosen as the baseline (for example a control group) and coded 0.

Effect size

An objective and usually standardized measure of the magnitude of an observed effect. Examples are Cohen's d, Glass's g and Pearson's r.

Error bar chart

Plots the mean with its 95% confidence interval as a line. Error bars can also use the standard error or standard deviation.

Experimental research

Research in which variables are systematically manipulated to see their effect on an outcome variable. The data can support statements about cause and effect.

General linear model

The linear model can cover different designs, such as comparing means of categorical predictors (t-test, ANOVA) and including both categorical and continuous predictors (ANCOVA).

Heterogeneity of variance

The opposite of homogeneity of variance. The variance of one variable is different across levels of another variable.

Homogeneity of variance

The assumption that the variance of one variable is stable (relatively similar) at all levels of another variable.

Hypothesis

A proposed, theory-driven explanation for a narrow phenomenon. It cannot be tested directly, so it is turned into predictions about measurable variables.

Independence of errors/Independent errors

The assumption that one residual (prediction error) does not influence another, so for any two observations the residuals are uncorrelated. For people, one person's error does not influence another person's.

Independent design/Between-groups design/Between-subjects design

Different treatment conditions use different people, so the data are independent.

Independent t-test

A test using the t-statistic that establishes whether two means collected from independent samples differ significantly.

Levene's test

Tests the hypothesis that the variances in different groups are equal. A significant result means the variances differ. With large samples, small differences can be significant. The author does not recommend it.

Linear model

A model of the form outcome = b times predictor + error. The key is that its form is linear, so with a single predictor it is a straight line.

Long format data

Scores on an outcome variable are in a single column, and each row is a combination of attributes of a score, such as the entity or condition. Scores from one entity can appear over several rows.

Mann-Whitney test

A non-parametric test for differences between two independent samples. It is functionally the same as Wilcoxon's rank-sum test, and both are non-parametric equivalents of the independent t-test.

Non-parametric tests

Procedures that do not rely on the restrictive assumptions of parametric tests. In particular, they do not assume a normally distributed sampling distribution.

Normal distribution

A probability distribution that is perfectly symmetrical (skew of 0).

Null hypothesis

The reverse of the experimental hypothesis. It states that your prediction is wrong and the predicted effect does not exist.

One-tailed test

A test of a directional hypothesis. The book generally advises against it because of the temptation to interpret interesting effects in the opposite direction to that predicted.

Outcome variable/Dependent variable

A variable whose values we are trying to predict from one or more predictor variables. It is called dependent in experiments because it is not manipulated, so its value depends on the manipulated variables.

Outlier

An observation very different from most others. Outliers bias statistics such as the mean and their standard errors and confidence intervals.

Paired-samples t-test/Dependent t-test

A test using the t-statistic that establishes whether two means collected from the same sample (or related observations) differ significantly. Also called the matched-pairs t-test.

Parameter

Describes relations between variables in the population. We use sample data to estimate its likely value because we cannot access the population directly.

Parametric test

A test based on a known distribution, usually the normal. It needs four assumptions, a normal sampling distribution, homogeneity of variance, interval or ratio data, and independence.

Population

The collection of units (people, plants, cities and so on) to which we want to generalize a set of findings or a model.

Power

The ability of a test to detect an effect of a particular size (0.8 is a good level to aim for). It equals 1 minus the Type II error rate.

Predictor variable/Independent variable

A variable used to try to predict values of an outcome variable. It is called independent in experiments because the experimenter manipulates it.

Q-Q plot (see IMG6)

Short for quantile-quantile plot. It plots the quantiles of a variable against those of a distribution, often the normal. Points on the diagonal mean the same distribution, and deviations show departures from it.

Raincloud plot (see IMG16)

A visualization that combines a raw data plot, a boxplot and a probability density plot.

Repeated-measures design/Within-subject design

Different treatment conditions use the same people, so the data are related. Also called a related design.

Residual

The difference between the value a model predicts and the value observed in the data. Basically an error. The collection of residuals for all observations are the residuals.

Robust methods

Procedures that give unbiased estimates, confidence intervals and significance tests even when the normal assumptions of the statistic are not met.

Sample

A smaller (hopefully representative) collection of units from a population, used to find out about that population.

Sampling distribution

The probability distribution of a statistic. It is the distribution of values we would get if we took lots of samples from a population and calculated the statistic for each.

Sampling variation

The extent to which a statistic (the mean, median, t, F and so on) varies in samples taken from the same population.

Shapiro-Wilk test

Tests whether a distribution of scores differs significantly from a normal distribution. A significant value means a deviation from normality, but large samples make even small deviations significant.

Standard deviation

An estimate of the average spread of a set of data in the original units of measurement. It is the square root of the variance.

Standard error

The standard deviation of the sampling distribution of a statistic. It shows how much the statistic varies across samples from the same population. Large values mean a sample may not reflect the population accurately.

Standard error of differences

The standard deviation of the sampling distribution of differences between sample means. It measures how variable those differences are.

Standardization

Converting a variable into a standard unit of measurement, typically standard deviation units. It lets us compare data measured in different units.

Systematic variation

Variation due to a genuine effect. It can be explained by the model fitted to the data.

t-statistic

A test statistic with a known distribution (the t-distribution). In the linear model it tests whether a b-value differs from zero, and for two means it tests whether their difference differs from zero.

Test statistic

A statistic for which we know how frequently different values occur. Its observed value is typically used to test hypotheses.

Two-tailed test

A test of a non-directional hypothesis.

Type I error

Believing there is a genuine effect in the population when there is not.

Type II error

Believing there is no effect in the population when there is.

Unsystematic variation

Variation not due to the effect of interest, such as natural differences in intelligence or motivation. It cannot be explained by the model fitted to the data.

Variance

An estimate of the average spread of a set of data. It is the sum of squares divided by the number of values minus 1.

Variance ratio (Hartley's Fmax)

The ratio of the variance of the group with the biggest variance to that of the group with the smallest. It is compared to Hartley's critical values as a test of homogeneity of variance.

Variance sum law

The variance of a sum or of a difference between two independent random variables equals the sum of their variances.

Wide format data (see IMG5)

Scores from a single entity appear in one row, and the levels of the predictor are arranged over different columns.

Wilcoxon signed-rank test

A non-parametric test for differences between two related samples. It is the non-parametric equivalent of the related t-test.

Difference score (lecture and Ch 9)

A person's score in one condition minus their score in the other. The paired-samples t-test tests whether the mean of these scores (D-bar) is 0.

p-value (lecture, see IMG2, IMG3 and IMG4)

The probability of observing this test statistic or one more extreme if the null hypothesis is true. If it is lower than alpha, H0 is rejected.

Pooled standard error (lecture and Ch 9)

The standard error of the difference that uses the pooled variance for both groups, so it assumes equal variances.

Pooled variance (lecture and Ch 9)

A weighted average of the two group variances, weighted by degrees of freedom (n - 1). It is used when group sizes differ, and you do not need to memorise the formula.

Student t-test (Ch 9)

The standard independent t-test. It assumes equal variances and is slightly more powerful.

t-distribution (lecture, see IMG1)

The sampling distribution of t under H0. It is bell-shaped and centred on 0, with fatter tails for small n (few degrees of freedom), and it converges to the standard normal as n grows.

Unpooled standard error (lecture, see IMG6)

The standard error of the difference that uses each group's own variance, the square root of (s1 squared over n1 + s2 squared over n2).

T-test in terms of the linear model

All t-tests use the linear model (outcome = model + error) to compare means. The independent t-test is a linear model with one two-category predictor, and t tests whether b for group (the difference between means) differs from 0. The one-sample t-test tests the mean against a reference value. The paired t-test is a one-sample test on the difference scores.

Three types of t-tests

One-sample compares one group mean to a hypothesized value. Paired-samples compares two means from the same or related entities (within-subjects). Independent-samples compares two means from different entities (between-subjects).

Degrees of freedom for each t-test

One-sample and paired, df = n - 1 (n is the number of pairs for the paired test). Independent (Student), df = n1 + n2 - 2. Welch, lower than that. The df set the shape of the t-distribution used to get p.

T-statistic as a signal-to-noise ratio

t = (observed difference minus the difference expected under H0) divided by the standard error of the difference. The top is the effect (signal, systematic variance) and the bottom is the error (noise, unsystematic variance). A larger absolute t means a larger signal relative to sampling noise.

Standard error - role in the t-test logic

The SE shows how much sample means (or differences) vary by chance alone. A small SE means even a modest difference is unlikely under H0, and a large SE means big differences are plausible by chance. The observed difference is judged relative to the SE.

Two explanations for a larger-than-expected difference

(1) There is no effect and we were unlucky, so sampling variation produced very different means. (2) The samples come from different populations, meaning a genuine effect and H0 is false. The bigger the difference relative to the SE, the more plausible the second.

What b0 and b1 represent

b0 (the intercept) is the mean of the baseline group coded 0. b1 is the difference between the two group means, so the t-test of b1 tests whether that difference is 0.

Median splits - why to avoid them

Dichotomizing a continuous variable at the median (1) distorts information, since similar people end up in opposite groups and dissimilar people in the same group, (2) shrinks effect sizes, and (3) increases the chance of spurious effects. It is only justified with a clear theoretical break point, such as a clinical diagnosis.

Variable types needed for a t-test

The outcome is a quantitative (continuous) variable, because t-tests compare means. In the independent t-test the predictor is categorical with exactly two groups. The paired t-test needs two measurements of the same outcome per participant. The one-sample t-test compares the outcome mean with a numerical reference value.

Postulated value

A value that is assumed or hypothesized rather than observed. H0 uses the postulated value as its model (for example model = 120), whereas HA uses the observed data (model = x-bar).

One-sided p-values

Only one tail counts, in the direction of HA. If t falls in the opposite direction to HA, p is large, always greater than 0.5.

Cohen's benchmarks for d

Small about .3, medium about .5 and large about .8 on the slide. They are only rough guides, so report the value and not just the label.

Paired t-test - what must be normal

The residuals, meaning the difference scores, should be roughly normal, especially with small samples. The raw scores in each condition do not need to be. Compute the differences and check them with a Q-Q plot. Two very non-normal measures can still give normally distributed differences.

Why repeated measures are more powerful

Using the same people removes individual differences such as IQ and motivation. Unsystematic (error) variance drops, so the SE is smaller and confidence intervals are narrower, which makes an effect easier to detect.

Paired design - advantage and trade-off

The advantage is more power, because stable individual differences are removed from the comparison. The trade-off is possible practice and boredom effects from repeating tasks, which the design can address with counterbalancing.

Counterbalancing

Systematically varying the order in which conditions are done. With two conditions, half the participants do A then B and the other half do B then A. It aims to remove systematic bias caused by practice effects or boredom effects.

Practice effect

Participants' performance on a task may be influenced (positively or negatively) if they repeat it, because of familiarity with the experimental situation and the measures used.

Boredom effect

Performance on a task may be influenced (assumed to be negatively) by boredom or lack of concentration when there are many tasks or the task goes on for a long time.

Independent t-test - what you need to calculate it

Only the group means, standard deviations (or variances) and sample sizes. You do not need the raw data.

Assumptions of all t-tests (lecture)

(1) Normally distributed residuals (model errors). (2) Random samples. (3) Independence, meaning observations are independent across participants or pairs and participants do not systematically influence one another. In a paired test the two scores within a person are deliberately dependent, but the pairs should be independent.

Extra assumption of the independent t-test

Equality (homogeneity) of variance, meaning both groups have equal variances. Assess it by comparing the observed SDs or variances, or with Levene's test. If it is violated, use Welch's t-test.

Levene's and Shapiro-Wilk tests - warning

Both are heavily influenced by sample size. In large samples tiny, harmless deviations become significant, and in small samples real problems can be missed. Do not treat them as pass or fail checks. Inspect variances and Q-Q plots, and prefer Welch where appropriate. A variance ratio above 2 is problematic.

Welch's t-test (lecture and Ch 9, see IMG6)

A version of the t-test that is robust to unequal variances, which bias the sampling distribution of t. It (1) uses the unpooled SE, the square root of (s1 squared over n1 + s2 squared over n2), and (2) lowers the df, more so the more unequal the variances are.

Student vs Welch rows

Student assumes equal variances and Welch does not. Field recommends reading the Welch row. The rows differ more when group sizes and SDs differ.

Practical default for an independent-samples test

The Welch t-test, because it is more robust when variances differ. It changes the SE (unpooled) and the df. It can have slightly less power when variances are truly equal, but it gives safer inference when equality of variance is doubtful.

Assessing the normality assumption

Use a Q-Q plot, where points should lie along the diagonal (not exact). A Shapiro-Wilk test is possible, where p below alpha means the assumption is violated, but with the same caution as Levene's test. Normality matters less as n grows (about 30).

Handling assumption violations

Violations distort the sampling distribution and the Type I and Type II error rates. Unequal variances call for Welch. Problematic non-normality calls for a non-parametric test, the Wilcoxon signed-rank test for paired samples or the Mann-Whitney test for independent samples. Bootstrapping is another option.

General procedure for a t-test

Explore the data, check for outliers, normality and homogeneity (boxplots, histograms, descriptives), run the t-test (or a non-parametric test if assumptions fail), and compute an effect size.

Key terms from Chapter 9

Dependent t-test, dummy variables, independent t-test, paired-samples t-test, standard error of differences and variance sum law.