Module 6

What is a Type I Error?

The null hypothesis (H0) is true, but we reject it and conclude that an effect exists ("false positive")

Why does type 1 error occur

Due to chance factors and fluctuations in sampling

Why is a Type I Error a major concern in research

It communicates false information, such as claiming a drug or intervention works when it actually has no effect

What is the probability of committing a Type I Error?

p(Type I Error) = alpha (α), typically set at .05 or 5% in the long run

What is a Type II Error

The null hypothesis is false, but we fail to reject it and conclude that no effect exists (i.e., false negative)

What is the probability of committing a Type II Error

p(Type II Error) = beta (β), which is difficult to calculate because true population parameters are usually unknown

Why does a Type II Error occur

It occurs due to random or chance fluctuations, small effect sizes that are hard to detect, or research design/measurement issues

What is Statistical Power

The probability of correctly rejecting a false null hypothesis (ability to detect a real effect when it exists); calculated as 1 - beta (1 - β)

What is a common benchmark value for Statistical Power in research

Power = 0.80 (80%), corresponding to a beta (β) value of 0.20 (20%)

What is the probability of NOT making a Type I Error

1 - alpha (1 - α), which equals .95 or 95% when alpha is set to .05.

In a real-world testing scenario (e.g., anti-doping), what represents a Type I vs Type II error

Type I Error: Falsely accusing a clean athlete of doping. Type II Error: Failing to catch an athlete who actually doped.

How do you directly decrease the probability of a Type I Error?

Lower the alpha level (e.g., from .05 to .01) to make it harder to reject H0.

What is the primary trade-off of lowering the alpha level (e.g., from .05 to .01)?

It increases the probability of a Type II Error (beta) and decreases statistical power.

• The more difficult we make it to reject H0, the more likely we are to miss effects that actually exist

What are the 5 main ways to increase Statistical Power (1 - beta) and decrease Type II Error?

1. Increase sample size (n).

2. Increase effect size.

3. Raise alpha level.

4. Use a one-tailed test.

5. Decrease within-group variability.

How does increasing sample size (n) affect standard error, power, and critical values?

It decreases standard error (due to increase in df), increases statistical power, and lowers the required critical value threshold in a t-test.

Why can a large sample size (n) be a double-edged sword in NHST?

With extremely large sample sizes, even trivial/tiny differences that have no practical importance can become statistically significant.

What are the influences of sample size on NHST

Sample size influences our decision to reject H0, and affects the conclusions we draw

The larger the sample size, the more likely we are to reject H0, even if the means of the samples are practically the same

We want large sample sizes to be precise and confident in our conclusions, but it would be helpful to have a gauge not reliant on sample size.

Why calculate effect size?

1. Unaffected by sample size

2. Allows for standardized comparisons across samples and research studies

3. Aids in interpretation of results

• A significant result may still be of small “practical significance”

• Non-significant results can be better understood with effect size comparisons

Interpretation of Cohen’s d?

Tells you how far the sample mean is from the population means in terms of standard deviation units

What are the formulas for Cohen's d in a One-Sample z-test vs. One-Sample t-test?

One-Sample z-test: d = (M - μ) / σ | One-Sample t-test: d = (M - μ) / s

How do you interpret Cohen's d values based on standard benchmarks?

d = 0.20 is Small (below 0.20 is trivial); d = 0.50 is Medium; d = 0.80+ is Large.

For anything in between its for example medium-large effect (d = 60) and these number are absolute value so it can be negative or positive but it doesn't matter.

How does increasing between-group variability affect Type II Error and Power?

It increases the effect size by pushing H1 further from H0, decreasing Type II Error and increasing statistical power.

How does raising alpha affect type II error and power

Increase in power, decrease type II error

How does standardizing testing procedures or using a controlled lab environment impact power?

It decreases within-group variability (unexplained error variance), which increases statistical power.

Why are researchers usually hesitant to raise alpha or use one-tailed tests just to increase power?

Raising alpha increases Type I Error risk and wants to reduce type I error more than type II error.

One-tailed tests examine only one direction of the distribution and are rarely justified unless replicating previous research. Using one tailed tests decrease the type 1 error

What happens when you decrease within-group variability?

Minimize the error variance (STD and variance that cannot be explained or accounted for) within a group by increasing the sample size and standardizing test procedures.

Think about who we sample, the particular study whether that’s controlled lab or real word environment.