SSR - Lecture 3

Empirical cycle

• observation

• induction → theory

• deduction → null hypothesis, alternative hypothesis

• testing → set alpha, choose power

• evaluation → making decision

sampling distribution

• tells us how likely our data are under certain hypothesis (e.g. null hypothesis)

test statistic

• statistic that summarizes data and is used for hypothesis testing, because we know how it’s distributed under different hypotheses → e.g. number of heads, sum of dice, t-statistic, F-statistic, …

What does alpha determine?

• how strict we are in our decision to reject null hypothesis (historically set to 5%)

Type 1 error

• rejecting null hypothesis even though it’s actually true → = alpha

Type 2 error

• not rejecting H0 even though it’s actually false → = beta

P-value

• conditional probability of observed test statistic or more extreme assuming null hypothesis is true

When do we reject H0?

• when p-value </= alpha

What does alpha determine?

• how willingly we reject null hypothesis

Which effects does it have if you increase/decrease alpha?

• Increase alpha = reject null hypothesis more often

• Increases Type I error rate

• Decreases Type II error rate

Rejection region

• dictated by alpha → The stricter the alpha, the further into the tail the region starts

What are common misconceptions about p-value? (3)

• significant result means effect is important → significance = effect size + sample size

• non-significant result means null hypothesis is true

• significant result means that null hypothesis is false\

What can non-significance mean? (2)

• there is no effect in reality

• sample size was not high enough to detect effect

evidence of absence

• evidence means that there is no effect

absence of evidence

• we have not enough/inconclusive data to make inference about whether effect exists or not

Describe the decision table

What is the best way to keep both alpha and beta low?

• high sample size

What is power?

• correctly reject H0 → True positive

• Power = 1 - beta

• depends on sample size

What is 1-alpha?

• correctly accept H0 → true negative