Lecture 1 - Null Hypothesis Significance Testing

What are the five steps underpinning the logic of NHST?

1. Pose null hypothesis

2. Build sampling error

3. Run statistical significance test (based on central limit theorem)

4. Compare cut-off to the alpha value

5. Declare not-significant or significant result

What does the NHST calculate?

It calculates the probability of your observed result occurring if the null hypothesis were true.

• known as the significance or p-value

What does a p-value of .91 mean?

if the null hypothesis were true, multiple tests with the sam sample size will achieve this score or greater 9.1% of the time.

What does a significance value tell us?

With some degree of certainty, that some difference exists - that the difference is not nothing.

What does a significance value not tell us?

The magnitude or size of an effect

What are the 3 main things that dictate whether you get a significant result?

1. Pre determined critical significance level (typically set at .05)

2. Magnitude of effect (size or strength of relationship)

3. Sample size

Why are significance values so sensitive to sample size?

Sampling error!!!!

Large samples = have less sampling error

• almost anything can be significant but may be trivial/small in effect size

Small samples = more sampling error

• may miss a significant result

What does power mean?

The probability of correctly concluding that there is a significant difference if it exists

What are the 6 limitations of NHST?

1. Dichotomous reject & non-reject

2. No focus on magnitude of effect

3. practical significance of effects

4. steady accumulation of knowledge

5. faulty premise - that there can be no difference in a population sample (there’s probs some difference

6. Overly sensitive to sample size

What are the 3 pros to NHST?

1. Easy to learn & do

2. Universally applied

3. Reduces decision making to a black and white dichotomy

Why is replication and reproducibility important?

• single study results may be due to sampling error

• replication builds confidence in the existence and magnitude of an effect

• Replication allows you to uncover common difference amongst means

• CI’s used to compare multiple studies

What is a confidence interval?

An interval estimation of a central tendency (e.g. mean)

• takes into account sampling error

• Captures uncertainty

• Assigns practical meaning to studies

What does a 95% CI mean?

if the experiment were repeated many times, 95% of the CIs/samples would contain the true population mean

What is the margin of error?

the likely greatest error in that point estimate

• quantifies the uncertainty in confidence intervals

e.g. 53 ± 2% —> 51%-55%

Why are CI’s good for comparing studies?

You can visually compare them!

• studies that might be inconsistent with NHST could be more consistent when using CI’s

What are the 7 steps for estimation based thinking?

1. Questions framed like “how much?” or “to what extent?”

2. identify most appropriate measure for the question

3. Design a study that gives us good point and interval estimates

4. Make a figure examine the data, and calculate the point and interval estimates

5. Interpret these using judgement in the research context.

6. Report the study in its entirety

7. Report in context of previous studies and consider replication