Hypothesis Testing, P-Values, Type I and II errors

Stages of the Scientific Method

1. Question Identified
2. Hypothesis formed
3. Research Plan
4. Data Collected
5. Results Analyzed ... then between this and conclusions, new questions arise
6. Conclusions

what is the goal of inferential statistics?

Goal is to infer that measurements obtained from a sample (statistics) are reasonable estimates of these same values in the population (parameters)

most research in EXSS involves quantification of a difference between some measure (i.e. mean)

i.e. difference between groups, difference across time

hypothesis testing

Procedure used to determine if the findings of an investigation are meaningful

*** i.e. if they have statistical significance

*** Is the difference between an observed statistic and an estimated parameter large enough to reject the null hypothesis?

4 steps of hypothesis testing

1. State the hypothesis
2. Select criterion/criteria for rejecting Ho
3. Compute the test statistic
4. Accept or reject the null hypothesis

a hypothesis

a tentative explanation or prediction of the outcome of a research investigation

Research /alternative hypothesis (Ha)

investigator's prediction based on previous literature and experience;
Not assessed directly via hypothesis testing

true/false --> Ha is assessed directly via hypothesis testing

False

Ha is NOT assessed directly via hypothesis testing

null/statistical hypothesis (Ho)

Hypothesis of no relationship or no difference

Directly assessed via hypothesis testing

a priori from your research question, come up with a ___ and ___ hypothesis

null and alternative

null hypothesis (Ho)

Ho: m1 = m2
(no difference)

alternative hypothesis (Ha)

Ha: most often it's just one choice...

There is a difference: Ha: m1 ≠ m2 (not equal)

for t-tests there are 3 choices for alternate / Ha

Ha: m1 ≠ m2 (not equal, 2 tailed test)
Ha: m1 < m2 (one-tailed test)
Ha: m1 > m2 (one-tailed test)

what are the two components that are contained by a difference between means?

- a "real difference"
- a meaningful difference

always select the _____ before collecting data

alpha level

what is the usual alpha level / what does it mean?

0.05

Reflects the probability that chance alone produced the difference observed between groups

We are comfortable with 5% uncertainty

real difference is due to the ______

treatment

sampling error is where difference is due to _______ or ________

chance or random effects

(true/false) sampling error is always present

True

reminder --> hypothesis testing involves 4 steps!

1. State the hypothesis
Select
2. criterion/criteria for rejecting Ho (alpha)

(do before collecting data!!!)
***********************
Compute the test statistic
Accept or reject the null hypothesis

we always come up with Ho and Ha _______ we conduct our study (a priori)

we always come up with Ho and Ha before we conduct our study (a priori)

We calculate a test statistic and p-value from our study using our sample based on a prior _____, ____, and _____

Ho, Ha, and selected alpha level (0.05) -- our level of significance

two options with our test statistic related to P

P is less than alpha (p<0.05)

P is greater than/equal to alpha (p > = 0.05)

a significant p-value

< 0.05 means the probability that chance produced the difference in our groups is small, tiny

we reject the null!

a non-significant p-value

>= 0.05 means the probability that chance produced the difference in our groups is NOT small

Type I error

alpha

OOPS, I rejected the null when it was True!!

I said "there IS a difference in the groups" when in TRUTH there wasn't!!

Type II error

beta

OOPS, I accepted the null when it was False!!
I said "there is NO difference in the groups" when in TRUTH there was!

type I or Type II is seen to have more serious consequences for research purposes?

type I
(5% when alpha level is .05)

decreasing the chance of alpha error increases the chance of _______ error?

beta error

to reduce the probability of a type II error, you can increase what two things?

- increase your alpha
- increase number of subjects

what can you do to reduce the probability of type I error?

decrease your alpha

as sample size increases, what happens to statistical power?

as sample size increases, statistical power increases

However, a very large sample size can inflate statistical power such that a very small difference could be statistically significant

The question that must be addressed is "What is a meaningful difference?"

think about statistical difference first, then look at _________ difference

meaningful difference

hypothesis testing is the basis for __________

statistical inference