IPR - WEEK 7 - statistical testing theory

- Hypothesis testing -

What is a hypothesis?

A clear, testable statement or prediction about what you expect to find i.e. about the variables or the outcomes

proposes a potential explanation or effect that can be examined through your experiment / analysis

Key features of a hypothesis?

1. testable - it must be possible to test this prediction using the experiment / observation (i.e. the variables you selected)

2. Specific: Should clearly define the variables and expected effect or relationships

3. falsifiable: should have a way to prove the hypothesis wrong if its incorrect (i.e., testable and realistic)

What are the types of hypothesis?

• Null Hypothesis (H₀)

• Alternative Hypothesis (H₁)

  • One tailed

  • Two-tailed

Null hypothesis

There will be no effect or relationship

e.g. no difference in wellbeing scores between the placebo and treatment group

Alternative hypthesis?

• There will be an effect or relationship

• one-tailed = effect is expected direction is specified e.g. students in the treatment group wull have higher wellbeing scores

• two-tailed = an effect is ecpected but direction not specified e.g. wellbeing scores will differ between the treatment and placebo group

When should ur hypothesis be one or two-tailed?

• one tailed = rare, when there is specific direction or strong theoretical justification. only used if an opposite direction is impossible or irrelevant.

• two-tailed = when you predict a difference but not the direction. Direction could go wither way

To test a hypothesis, we need to analyses, what do most analyses test provide?

• Test statistic e.g. Z-scores - summarise s how dar samole result is from H₀ expectations

• ie.e., differences between your observed data (your effect) and what is expected under the Null Hypothesis (there beong no effect)

• Each statistic has an associated p-value

Whats a p-value?

Probavility of observing data as extreme (or more extreme) assuming there is no effect (if the null hypothesis is true)

it determined the strength of evidence against the Null Hypothesis

• the compared to a set criterion: the significance alpha level (a) to determine if you can reject the Null Hypothesis

P levels compared to the sognifocance alpha level (a), to determine if you can reject the Null Hypothesis?

• If p < a → reject H₀ (result is statistically significant

• If p ≥ a → fail to reject H₀

If there is a small p value?

data are rare under H₀ → reject H₀

strong evidence against H₀ - EFFECT LIKELY TO EXIST

does not tell you your Alternative Hypothesis is true

Phrasinf of null hypothesis in write ups:

If p < α → “We reject the null hypothesis” (evidence suggests there is
some difference)
If p ≥ α → “We fail to reject the null hypothesis” (insufficient evidence to
conclude there is a difference)

What is the significance / alpha level ( a) ?

Set threshold / criteria that quantifies the strength of evidence against the Null hypothesis, i.e. threshold of deciding whether to reject H₀

What is the typical alpha level (a)

.05 (other values acceptable, .01 annd .001

what does the a value represent?

The maximum (acceptable) probability of rejecting the null hypothesis (H₀) when the null hypothesis is actually true

i.e. if you accept up to a 5% risk of claiming there is an effect when there is not

p value > .05

Not rare under null hypothesis → fail to reject null hypothesis ( H₀)

P-value < .05

rare under null hypothesis → reject H₀

Significance level for two tailed distribution?

The alpha (.05) is divided equally between the 2 of them

a/2 = 0.25

still interpreted as p < 0.5 = reject h₀

significance level for one tailed?

critical region (tail) is on one specific side

if p < .05 = reject h₀

_{will ignore anything extreme on the other side}

When a = .05 and p < a, the result falls in the critical region (5% tail) suggesting:

• Our result is rare and under the assumption of no diffference and inconsistent eith the middle 95% of values we would expect

• if the null hypothesis were true (no difference), there is a 5% probability of getting this extreme (rare)

• This suggests the result would be very unlikely if H₀ were true. We have evidence against H₀ and reject it.

What is a TYPE I ERROR?

When there is NO effect and we say '“we reject the null hypothesis”

no effect or difference exists, but say there is

probability of making a Type I error is the a level

i.e., fales positive

What is a TYPE II ERROR?

When there is an effect and we say “We fail to reject the null hypothesis”

An effect or difference exists, which we miss

probability of making a Type II erro ris denoted as B (beta)

i.e., false negative

Statistical power

• if an effect truly exists in the population, power is the likelihood your test will detect it

• so power is about correctly rejecting the null hypothesis when the alternative hyothesis is true

• probability of avoiding a TYPE II error

• Power = 1 - B

whats a commonly used statisitcal power

.80 (80%)

Fsctors that affect power?

• small effect between conditions (larger effects → higher power)

• sample sie is too small (bigger samples → higher power)

• alpha level threshold is strict (e.g., .01 - > lower power)

• variability in the data ( more noise → lower power)

• Test type/hypothesis: one-tailed tests → higher power than two tailed

effect size

an idea of the sixe of the effect we found