Week 10 – Hypothesis Testing, Confidence Intervals, Effect Sizes & Statistical Power

Criticisms of hypothesis testing (p < .05)

The significance approach (set H0 and H1, then check whether p < .05) is the most popular approach in published health articles, but it has valid criticisms: p < .05 is an arbitrary threshold; researchers are often unclear what p actually means; and p values can be abused (p-hacking).

p-hacking

Running a lot of different analyses and then only reporting the ones that are significant (collecting or selecting data or analyses until nonsignificant results become significant). Head et al. (2015, PLOS Biology) concluded it is probably common, but its effect seems weak relative to the real effect sizes being measured.

What did the 2019 Nature editorial ('It's time to talk about ditching statistical significance') suggest researchers do instead?

Educate themselves about statistical misconceptions and consider uncertainty from multiple angles in every study. Logic, background knowledge and experimental design should be considered alongside p values and similar metrics.

Two additional values commonly reported alongside the p value

Confidence intervals and effect sizes. Researchers include them to give a more accurate understanding of the data and what it means.

Point estimate

An exact value (a single number) that represents some aspect of the population, e.g., the estimated difference between two means. It is an educated guess of the real population value based on the sample data.

Confidence interval

An estimated range of values for a population parameter. The size of the interval is expressed as a percentage (e.g., 95%).

How is a 95% confidence interval for the difference between means interpreted?

It contains the true difference between the means 95% of the time.

Point estimate vs confidence interval in a results table

The mean difference (M) is the point estimate; the 95% CI is the range around that point estimate (from the mandala study table).

What does the width of a confidence interval tell you?

How uncertain we are about the unknown parameter: a wide interval means we are unsure, a narrow interval means we are fairly sure.

Doctor example for confidence intervals (how long an illness will last)

'On average 9 weeks' = point estimate. 'Between 3 weeks and 15 weeks' = confidence interval (wide, unsure). 'Between 8 weeks and 10 weeks' = confidence interval (narrow, fairly sure).

Why are confidence intervals more informative than a simple hypothesis test result?

They give a range of plausible values for the unknown parameter, not just a point estimate or a reject/retain decision.

How do confidence intervals relate to the p value?

They are consistent. p < .05 means a pattern like ours is unlikely (less than a 5% chance) if H0 is true, and the 95% CI should tell the same story about whether there is a difference/effect.

Confidence interval and significance in an independent samples t-test

If the difference between groups is significant (p smaller than .05), the 95% CI for the difference should not include 0. If the CI includes 0, the difference is not significant.

Example: independent t-test with p = .080 (episodic memory)

t(58) = 1.78, p = .080; mean difference = 1.46, SE = 0.818, 95% CI [−0.180, 3.10]. The CI includes 0, so the difference is not significant.

Example: independent t-test with p < .001 (aerobic fitness)

t(58) = 4.29, p < .001; mean difference = 5.86, SE = 1.37, 95% CI [3.13, 8.60]. The CI does not include 0, so the difference is significant.

Example: paired samples t-test in jamovi (working memory)

t(59) = −7.01, p < .001; mean difference = −2.45, SE difference = 0.350, 95% CI [−3.15, −1.75]. The CI does not include 0, consistent with p < .001.

How do you get confidence intervals in jamovi t-tests?

In the t-test dialog, under Additional Statistics tick Mean difference and Confidence interval (95%). The output then shows the mean difference, SE difference and the lower and upper limits of the CI.

Example: CI and p in the mandala study table

Mindfulness, breathing group: mean difference 1.22, 95% CI [−0.67, 3.10], p = .194 (CI includes 0, not significant). Anxiety, mandala group: mean difference −1.93, 95% CI [−2.92, −0.93], p < .001 (CI excludes 0, significant).

Where do you see confidence intervals in everyday life?

Particularly in reports about survey data, or polls about politics, as reported in news articles.

Everyday example: tourism survey estimate

About 310,000 interstate holiday visitors to the Northern Territory was reported as somewhere between 240,000 and 380,000 (an interval of 22% either side). The limits are 95% confidence intervals: if the survey was run 100 times, the true number would fall in the interval about 95 times.

Everyday example: error bars in survey reports

Survey reports show 95% confidence intervals as error bars on charts, e.g., reasons for selecting an ACT public school, and business performance expectations by industry.

Statistical significance vs practical significance

Statistical significance: the result is unlikely to be due to chance (e.g., p < .05). Practical significance: the difference/effect is big enough to be meaningful. A finding can be statistically significant without having any practical significance.

Why can a very large N produce significant but meaningless results?

Bigger samples reduce inferential uncertainty (they are better), but if N is large enough even very small, meaningless differences will be statistically significant.

Lecture examples where an effect could be significant but not practically significant

Gender differences in mathematical ability; effectiveness of treatments for pain.

Effect size

A measure of the size of an effect that ignores sample size. It is a measure of practical significance: how big is it?

Examples of effect size measures

η² (eta-squared), r² and Cohen's d (different measures are used for different tests).

What does effect size (Cohen's d) tell you in a t-test?

Whether the difference between means is large compared to the amount of dispersion (spread) in the data.

Formula for Cohen's d

d = (μ1 − μ2) / s: the difference between the population means divided by the standard deviation of either population.

Does Cohen's d depend on sample size?

No. d does not depend on sample size (N is not included in the calculation).

How is the effect size approach used alongside the significance level approach?

As a complement: people normally report the effect size as well as the p value. jamovi can calculate d and other effect sizes for you.

Cohen's (1992) guidelines for judging d

Small effect ≥ .20; medium effect ≥ .50; large effect ≥ .80.

Why do effect sizes allow results to be compared across studies?

N (the number of people in the study) is not included in the calculation of effect size.

How do you get Cohen's d in jamovi?

In the t-test dialog, tick Effect size under Additional Statistics. The output then includes a Cohen's d column.

Example: Cohen's d for a paired samples t-test

t(59) = −7.01, p < .001, Cohen's d = −0.905 (a large effect by Cohen's guidelines).

Example: Cohen's d for an independent samples t-test with p = .080

t(58) = 1.78, p = .080, Cohen's d = 0.460 (above the small cut-off of .20 but below medium at .50, even though the result is not significant).

Example: reporting d in a paper (mandala study)

The mandala group reported significantly lower anxiety after the activity, t(27) = −3.98, p < .001, d = −0.67. The paper reports the 95% CI for each mean difference in its table and d in the text.

Example: effect sizes in an exercise study table

Table 2 gives Cohen's d for HIIT vs CON (wait-list control), MOD (moderate intensity continuous training) vs CON, and HIIT vs MOD. Bold values signify moderate or large effect sizes (e.g., aerobic fitness, VO2peak, HIIT vs CON: d = 1.28).

Example: meta-analysis reporting effect sizes with 95% CIs (Northey et al.)

Exercise and cognitive function in adults over 50 (estimate mean with 95% CI): aerobic 0.24 (0.10 to 0.37); resistance training 0.29 (0.13 to 0.44); multicomponent training 0.33 (0.14 to 0.53); tai chi 0.52 (0.32 to 0.71); yoga 0.13 (−0.10 to 0.36). Only yoga's CI includes 0.

Which hypothesis testing concepts does the 'pulling it all together' section link up?

Statistical hypotheses (H0 and H1), one-tailed vs two-tailed tests, the probability or p value and alpha (α), and Type I and Type II errors.

What is inferential statistics about, in terms of probability?

Calculating the probability of getting a given pattern in the data if the null hypothesis is true, i.e., if nothing is going on (no real difference between groups, or the sample is just from the normal population).

Why do we reject H0 when the likelihood of our data is low?

If the likelihood of data like ours is sufficiently low (usually set at 5%), it is unlikely to occur if H0 is true, so we reject H0.

Alpha (α) / significance level

The cut-off we use for the p value. Alpha .05 means a 5% cut-off; alpha .01 means a 1% cut-off. We could use values other than 5%.

Decision rule using alpha

Your p value must be smaller than the alpha level you chose in order to reject the null hypothesis.

What do rejection regions show (alpha in visual terms)?

Alpha describes the rejection regions of the distribution. If our t value (or other test statistic) falls into a rejection region, beyond the critical value, we reject the null hypothesis.

Alpha in a two-tailed test

Alpha (the rejection region) is split between the two ends of the curve, α/2 in each tail. The t value can be in either direction as long as it is big enough to reject H0.

Alpha in a one-tailed test

Alpha (the rejection region) is all at one end of the curve. The t value has to be big enough in that direction to reject H0.

Type I error

Incorrectly rejecting a true null hypothesis (you reject H0 but you are wrong). Its probability is alpha (α).

Type II error

Incorrectly retaining a false null hypothesis (you retain H0 but you are wrong). Its probability is beta (β).

If alpha = .05, how often will you wrongly reject a true null hypothesis?

5 times out of 100 (1 time out of 20). The pattern was just due to chance, which is a Type I error.

Decision table: research conclusion vs reality

If H0 is really true: retaining H0 is accurate (1 − α) and rejecting H0 is a Type I error (α). If H1 is really true: retaining H0 is a Type II error (β) and rejecting H0 is accurate (1 − β = power).

How are alpha and beta related?

Inversely: as one gets larger the other gets smaller. Making the chance of wrongly rejecting H0 (α) smaller makes the chance of wrongly retaining H0 (β) bigger.

Power

The probability of correctly rejecting a false H0. Power = 1 − β.

Beta (β)

The probability of a Type II error: failing to reject the null hypothesis when it is false.

What does increased power mean, and why does it matter?

You are more likely to find a significant result if one actually exists. Power tells us how likely we are to find a difference/relationship/effect if one is really there.

Interpreting power values (.80 vs .50)

Power .80 = an 80% chance of finding a difference between the groups if one is really there. Power .50 = only a 50% chance of correctly finding a significant effect.

Factors that affect power

Alpha (higher alpha increases power); effect size (larger effects are easier to find); sample size (larger N increases power); sample variance (smaller variance means more power); one- vs two-tailed test (a one-tailed test has more power to detect a difference in one direction only).

How does alpha affect power?

Alpha is the probability of a Type I error; a higher alpha increases power.

How does a one-tailed test affect power?

A one-tailed test has more power to detect a difference in one direction ONLY.

Power analysis

A calculation (using programs such as G*Power) done BEFORE a study to work out how many people you need to have a certain level of power.

Example power analysis question

How many people are needed for a power of .80, given alpha = .05, looking for a small effect size, and conducting a between-groups t-test?

When might you be asked to report a power calculation?

When applying for a grant or for ethical approval, so you can be clear how many people you will need in your study.

Example: a priori power analysis in a paper (Wang et al., climate change study)

They used G*Power. Assuming an effect size of 0.29, a sample of 111 would give a power level of 0.99 at α = 0.05. A total of 218 Australian adults (114 female) took part.

Why do hypothesis tests focus on populations rather than samples?

We are more interested in populations than samples: we don't mainly care about the 20 people in our sample but what it means for the whole population. Our hypotheses are about the underlying populations, not the particular samples.

H0 for t-tests in terms of populations

No difference between the two groups: they are drawn from the same population, so the population mean for the first group/condition is the same as for the second.

One-tailed vs two-tailed hypotheses for t-tests

One-tailed (directional): H0: µ1 = µ2, H1: µ1 > µ2. Two-tailed (non-directional): H0: µ1 = µ2, H1: µ1 ≠ µ2.

H0 for correlations in terms of populations

No correlation between the two variables in the population: out in the real world there is no relationship between them.

One-tailed vs two-tailed hypotheses for correlations

One-tailed (directional): H0: r = 0, H1: r > 0. Two-tailed (non-directional): H0: r = 0, H1: r ≠ 0.

Mandala study: What happened to mindfulness scores across the three activities?

Mindfulness improved significantly after Mandala (mean difference 1.75; 95% CI [0.44, 3.06], p = .011) and free drawing (1.17; 95% CI [0.16, 2.18], p = .025). The breathing group was not significant (1.22; 95% CI [−0.67, 3.10], p = .194).

Mandala study: What happened to anxiety scores across the three activities?

Anxiety decreased significantly in all three groups: Mandala (mean difference −1.93; 95% CI [−2.92, −0.93], p < .001), free drawing (−1.12; [−2.18, −0.05], p = .041), and breathing (−2.73; [−3.97, −1.49], p < .001).

Mandala study: What happened to positive affect?

No significant pre-to-post change was found in any group: Mandala p = .328 (95% CI [−1.21, 3.49]), free drawing p = .201 ([−0.63, 2.86]), and breathing p = .370 ([−4.11, 1.59]). All three confidence intervals include 0.

Mandala study: What happened to negative affect?

Negative affect decreased significantly in all three groups: Mandala (mean difference −1.79; 95% CI [−2.76, −0.82], p = .001), free drawing (−1.92; [−3.19, −0.66], p = .004), and breathing (−2.83; [−4.71, −0.94], p = .004).

What do CON, HIIT and MOD stand for in the exercise-intervention effect-size table?

CON = wait-list control; HIIT = high-intensity interval training; MOD = moderate-intensity continuous training.

Which exercise-intervention comparisons showed moderate or large effect sizes in Table 2?

Bold Cohen’s d values indicate moderate or large effects. The listed values were: episodic memory, HIIT vs CON 0.76 and MOD vs CON 0.66; executive-function errors, HIIT vs CON 0.75 and HIIT vs MOD 0.55; working memory, HIIT vs CON 0.81 and HIIT vs MOD 1.41; resting middle cerebral artery mean velocity, HIIT vs CON 0.86; cerebrovascular reactivity, HIIT vs CON 0.72 and HIIT vs MOD 0.54; VO₂peak, HIIT vs CON 1.28 and MOD vs CON 0.72.

What did the exercise study report about adherence and exercise intensity?

Adherence was similar in HIIT and MOD (78.7 ± 13.2% vs 79.4 ± 12.0% of sessions; p = .93), and no adverse events were recorded. HIIT had a higher relative heart rate (93.9 ± 5.7% vs 84.1 ± 5.2%; p < .001) and RPE (14.5 ± 2.7 vs 12.3 ± 1.6; p < .01) than MOD immediately before warm-down.

What did the business-performance survey show about average expectations and industry confidence intervals?

Businesses reported an overall average future-performance expectation of 7.26 out of 10. Average industry scores ranged from 8.18 to 6.39; the slide describes very little variation between industries because the results were within the 95% confidence intervals.