lecture 5: effect size and p-value

Effect Sizes & P-Values

The Big Picture

Describing Data (Weeks 1 & 2)
- Types of data variable
- Distribution
- Normal Distribution
- Histogram
- Mean
- Standard Deviation
- Skewness
- Samples & Populations
- Sampling distribution of the mean
- Standard error of the mean
- 95% confidence interval
Testing Hypotheses (Weeks 3 & 4)
- Hypothesis
- Null hypothesis
- Test statistics
- One-sample T-Test
- Independent samples T-Test
- Paired samples T-Test
- Within and Between subject designs
- One/Two-tailed tests
Interpreting & Reporting (Weeks 3, 4 & 5)
- Levene’s Test
- Shapiro-Wilks Test
- Degrees of freedom
- Assigning significance – p-values
- Effect sizes
- Reporting T-Tests

Outline & Objectives

P-Values
- Null hypothesis significance tests
Effect Sizes
- T-Values combine the size of a difference with the precision of the estimate.
- Cohen’s D provides a ‘pure’ measure of the size of a difference.
Comparing means with T-Tests
- A big picture summary

Test Statistics

A test statistic quantifies how far the observed data are from what we would expect to see from the null hypothesis.
It is a summary of the whole dataset and is specifically tuned to address a specific hypothesis.
The world is noisy, and we may have been lucky (or unlucky) in our data sample.
Test statistics are continuous – how are we supposed to make a categorical decision?

Null Hypothesis Significance Testing

The process of hypothesis testing:
1. Hypothesis – what question are we asking of our data
2. Test statistic – summary of the dataset relating to the hypothesis
3. Null model – expected variation due to uncertainty in the data
4. p-value – probability of observing our test statistic if the null were true
5. Decision – accept or reject the null hypothesis
The probability of a result can be computed from the sampling distribution of our test, assuming that the null hypothesis is true.
If our data are normally distributed, we can compute the sampling distribution straightforwardly.

The T-Distribution as a Null Model

Our null model describes the distribution of t-values that we might expect to see, just due to random noise, if there were no true difference in our data.
It tells us what t-values we might reasonably see by chance in our experiment.
This is all conditional on our parametric assumptions being true!

Degrees of Freedom

The shape of the null model depends on the number of observations.
We see more extreme values with smaller sample sizes.
This is specified by the ‘degrees of freedom’ of the analysis.
We use DF rather than the number of observations, as we must make an adjustment account for the number of values we’re estimating from the data.
If we estimate the mean from 100 data points, we have 99 degrees of freedom.
One-Sample T-Test: $DF = N – 1$
Independent-Sample T-Test: $DF = N1 + N2 – 2$
Paired Sample T-Test: $DF = N - 1$

P-Values

P stands for Probability.
The p-value is the probability of observing a result at least as extreme as the one from the data.
By convention, we typically consider a result ‘significant’ if we have a less than 5% chance of observing a result the same size or larger by pure chance.
This is typically written as a decimal 0.05

Interpreting p-values

The p-value tells us the probability of obtaining test results at least as extreme as the result actually observed, under the assumption that the null hypothesis is correct.
A p-value of 0.4 means there is a 40% chance of observing a result at least as extreme as the one we have observed under the assumption that the null is true.
A p-value of 0.012 means there is a 1.2% chance of the same.
A p-value is the probability under a specified statistical model that a statistical summary of the data (e.g., the sample mean difference between two compared groups) would be equal to or more extreme than its observed value.
P-values are not an indicator of the truth of the null hypothesis.
P-values don’t allow us to make claims with certainty – any claim could be wrong, even if it would be a 1-in-a-million chance.
No single experiment can prove something, only replications can do this with certainty.

Interpreting p-values: Over-the-top Descriptions

Significant finding: We claim there is a non-zero effect, while acknowledging that if scientists make claims using this methodological procedure, they will be misled, in the long run, at most alpha % of the time, which we deem acceptable. We will, for the foreseeable future, and until new data or information emerges that proves us wrong, assume this claim is correct. Alpha = Type 1 error (Incorrect rejection of null)
Non-significant finding: We cannot claim there is a non-zero effect, while acknowledging that if scientists refrain from making claims using this methodological procedure, they will be misled, in the long run, at most beta % of the time, which we deem acceptable. Beta = Type 2 error (Incorrect acceptance of null)

Null Hypothesis Significance Testing Summary

Experimental Hypothesis: Drinking coffee extends participant’s attention spans
Null Hypothesis: Drinking coffee does nothing to participants attention spans
Test statistic: Independent samples t-test summarizing dataset in terms of the hypothesis. $t(36) = 5.21$
Null Model: Distribution of t-values arising from random variation in data sampling if drinking coffee does nothing
p-value: $p = 0.034$ , the probability of observing a t-value at least as large as 5.21 if drinking coffee does nothing
Decision: Threshold is 0.05, we can tolerate being mislead 5% of the time in the long run – Reject the null hypothesis

Misuse of P-Values

The p-value is not the probability that the null hypothesis is true, or the probability that the alternative hypothesis is false.
- A p-value can indicate the degree of compatibility between a dataset and a particular hypothetical explanation (such as a null hypothesis).
- Specifically, the p-value can be taken as the probability of obtaining an effect that is at least as extreme as the observed effect, given that the null hypothesis is true.
The p-value is not the probability that the observed effects were produced by random chance alone.
- The p-value is computed under the assumption that a certain model, usually the null hypothesis, is true.
- This means that the p-value is a statement about the relation of the data to that hypothesis and that hypothesis alone.
- P-values assume that the null hypothesis is true and tell us the probability of our data observations, they do not address whether our hypothesis is true given our data observations.
The 0.05 significance level is merely a convention.
- The 0.05 significance level (alpha level) is often used as the boundary between a statistically significant and a statistically non-significant p-value.
- However, this does not imply that there is generally a scientific reason to consider results on opposite sides of any threshold as qualitatively different.
The p-value does not indicate the size or importance of the observed effect.
- A small p-value can be observed for an effect that is not meaningful or important.
- In fact, the larger the sample size, the smaller the minimum effect needed to produce a statistically significant p-value (see effect size).

Reporting P-Values

Report p-values after test statistics
Specify the degrees of freedom of the test
Report exact p values to two or three decimals (e.g., p = .006, p = .03).
However, report p values less than .001 as “p < .001.”
Specify the significance threshold used
- Example: $t(28) = 1.13; p = 0.32$

Review: What a p-value is and is not

What a p-value is:
- The probability of a chance result being at least as extreme as the one observed in your data under the assumption that the null hypothesis is true
- Conventionally considered significant at p > 0.05
What a p-value is not:
- The probability that the null hypothesis is false
- The probability that the experimental hypothesis is true
- A statistically significant result does not necessarily mean that the result is practically significant, or useful.
- The p-value is a probability, not a certainty.

How Big is the Effect?

Sometimes it is more appropriate to focus on estimating the magnitude of an effect rather than going through the (potentially flawed) process of hypothesis testing
A statistically significant result does not necessarily mean that the result is practically significant, or useful.
There are many reasons that a practically inconsequential finding could be statistically robust, particularly at large sample sizes.

Effect Sizes

Test Statistics
- t-statistics
  - A blend of effect magnitude and our confidence in the estimate
Effect Sizes
- Cohen’s D
  - Purely the size of the difference between groups.
  - No information about confidence in the estimate
We can compute a ‘pure’ measure of how large a difference using an effect size

Cohen's D

Cohen’s D is NOT strongly affected by sample size (apart from at very small samples)
In an extreme case, a test statistic can nearly always be ‘significant’ if the sample size is large enough.
This is as it blends the size of the difference with the precision of the estimate, and precision always increases with sample size
The effect size measures the strength (or magnitude) of the apparent difference, irrespective of how significant or probable that effect may be.

Standardized Values According to Cohen

A medium effect of 0.5 is visible to the naked eye of a careful observer.
A small effect of 0.2 is noticeably smaller than medium but not so small as to be trivial.
A large effect of 0.8 is the same distance above the medium as small is below it.
These values are often used in the literature but must be used with caution – they are arbitrary labels, not absolute truths.

Reporting Effect Sizes

Report effect sizes after p-values
Report exact effect sizes to two decimals (e.g., p = .61, p = .032).
Report the specific effect size measure used.
- Example: $t(28) = 1.13; p = 0.32$ , Cohen’s d = 0.15

Comparing Analyses and Results

174 analyst teams worked on 2 datasets
- Blue tit dataset: Compare sibling number and nestling growth
- Eucalyptus dataset: Compare grass cover and tree seedling recruitment
Blue tit: Most analysts concluded that there was a negative relationship between measures of sibling competition and nestling growth, though half the teams expressed qualifications or described effects as mixed or absent.
Eucalyptus: There was a broader spread of conclusions with at least one analyst team providing conclusions consistent with each conclusion category. The most common conclusion for the Eucalyptus dataset was that there was no relationship between grass cover.

Worked Example

A significant difference was found between the average amazon review rating between female (M=4.29, SD = 1.47) and male (M=4.22, SD=1.58) reviewers, t_{123003903} = 302.88, p>0.001, Cohen’s d = 0.05
- Hugely ‘significant’ effect in terms of the hypothesis test
- Very small effect size which indicates that the result has limited practical impact…

Four Factors

The significance level (i.e., your p-value), frequently set at $\alpha =.05$
Alpha: the probability of correctly rejecting the null hypothesis (1- type 1 error) for a given effect size and sample size
Power: size of the relationship/difference between two variables
Effect size: the number of observations in your study.
Sample size
These factors are interrelated, from any 3 we can compute the fourth
Low ‘Power’ is a major problem, it would mean that a replication has a poor chance of correctly detecting the effect if it is there.

Worked Example: Sample Size

We choose:
- alpha = 0.05
- Power = 0.8
We estimate:
- Effect size = 0.05
We compute:
- Sample size = 8407 participants per group!

Two Examples

t(1,476) = 8.32, p<0.0001, Cohen’s d = 0.02
- Very large sample
- Strong evidence for a difference in the means
- Very low probability of obtaining a result this large by chance, if the null were true
- But very small effect size, though robust this difference is not likely to be of practical importance
$t(14) = 1.23 p=0.312$ , Cohen’s d = 0.65
- Small sample
- Little evidence for a difference in the means
- Moderate probability of obtaining a result this size by chance, if the null were true
- But a moderately large effect size. Possibly an indication that there is an effect but we just can’t tell from this sample. Need to replicate or extend.

Revision

Research question & hypothesis
Study design & experiment
Data type & distributions
Test statistics & T-Tests
Assumption Checks & Alternatives
Student or Welch?
Interpreting T-values
Interpreting Effect sizes
One- or Two-Tails
Interpreting p-values
Reporting results

Overview

Shapiro-Wilk test: Normal Distribution
Levene’s test: Homogeneity of variance
Data to be analyzed
- Compare one sample to reference: One-Sample t-test. Consider non-parametric alternative if the assumption of normality is violated (Wilcoxon Rank Test).
- Compare two samples:
  - Independent Samples: Student’s t-test / Welch’s t-test. Consider Welch’s t-test if groups do not have comparable variance (Mann-Whitney U)
  - Dependent Samples: Paired t-test. Checking normality of the paired-difference, not the data (Wilcoxon Rank Test).