(10) Power analysis and meta-analysis

What is effect size?

The size of the relationship between two variables

• the extent to which one can predict the value of one variable given the value of the other variable

What do statistical tests tell us about results?

They indicate whether an effect is likely due to chance, not whether it is meaningful

Does effect size imply causation?

No, it reflects the strength of a relationship, not causality

Why can the Pearson correlation be used as an effect size?

It measures the strength of the relationship and is standardised across studies

• we can derive it from other tests such as t-tets if its not reported

What is Cohen’s d?

A standardised measure of the difference between two means

How is effect size expressed in Cohen’s d?

In standard deviation units

What is a problem with relying only on significance testing?

Significant results may not be meaningful and depend heavily on sample size

• e.g. a significantly large sample size will always produce a significant result

Why is the null hypothesis rarely true?

Because real-world differences almost always exist to some degree

What is a Type I error?

Finding a significant effect when none exists

• false positive

What is a Type II error?

Failing to detect a real effect

• false negative - most likely due to lack of statistical power

What is statistical power?

The probability of detecting a real effect when it exists

e.g. a study with a power of 0.8 has a 80% chance of finding a sig effect when there is one, also a 20% chance of not finding one

Why are low-powered studies problematic?

They are unlikely to detect real effects and increase Type II errors

What factors affect statistical power?

Effect size, sample size, and alpha level

• the bigger the effecti size, the higher the likelyhood it is detected

• the bigger the sample the greater the power

• a more leinent alpha will increase the power

What is the recommended level of statistical power?

Around 0.80 (commonly accepted standard)

• but power should definitely be above .5

Why is power analysis important in research planning?

To ensure studies have a reasonable chance of detecting effects

What are Cohen’s benchmarks for effect size?

0.2 = small, 0.5 = medium, 0.8 = large

How can researchers estimate effect size before a study?

Using pilot data, previous research, or meta-analysis

What is meta-analysis?

A method of combining effect sizes across studies to estimate an overall effect

• average effect size

how is meta analysis conducted

• define variables of intrest

• plan database search

• calculate effect size

• combine effect size

• compare effect sizes from different study types

Why convert correlations to z-scores in meta-analysis?

To avoid distortion when averaging effect sizes

How can statistical power be increased?

Increase sample size, relax alpha, reduce noise, standardise procedures, and use reliable measures and repeated measures design

how does reducing ‘noise’ increase effect size

it reduces within condition variability which increases the power by increasing the effect size

What is the role of falsification in scientific discovery?

Theories should be tested by trying to disprove them rather than confirm them