Lecture 2 - Effect Size & Power Analysis

Define effect size measures

Statistics that estimate the strength or magnitude of the relationship between variables or the difference between two distributions of scores.

How long have effect size measures been around for?

Over 90 years

All Analyses are essentially about what?

the nature of relationships

Why can you use any type of statistical test (GLM)?

regardless of:

• the nature of the design

• the hypothesis

• whether it is an intervention or not

is essentially exploring the nature of the relationships amongst variables.

What are 3 conceptualisations of relationships?

1. difference

2. correlation

3. predictor

What information does cohen’s d provide?

an estimate of the magnitude of the difference between two means.

standardised —> which allows for comparison between variables of different scales

What is the most commonly used effect size measure and why?

Cohen’s d

1. easy to conceptualise

2. easy to calculate

3. many questions are two group comparisons

What are the two families of ES measures?

1. Standardised mean differences / d family

2. Proportion of overlapping or shared variability / r family

What are the characteristics of the first family of ES measures?

• Shows the difference between two means in standard units (similar to Z-score)

• Commonly used and well understood

• Can be positive or negative and has no theoretical upper limit

• Want to assess magnitude of an effect

What are the characteristics of the second family of ES measures?

• measures shared variability

• R squared for correlation & regression

• eta squared for ANOVA

• Cramer’s V for chi-square

What is the relationship Between R2 and d

mathematically related concepts

not directly comparable as they are measured along different scales

must be converted first!

Which type of ES measure to choose?

choice comes down to the communication aspect of your research

1. Is it about the differences?

2. Or is it about the shared commonality?

List 4 things good about ES measures

1. Provide useful information

2. make sense of significant, but trivial results

3. lifeline to non-significant results

4. both families are unaffected by sample size

Why is it taking so long for ES measures to become universal?

1. Software packages don’t provide ES

2. ES make like more difficult for researchers (not black or white)

3. Interpretation of ES less clear than p-value

4. no universally accepted index of effect

Origin and use of interpreting ES criteria

the criteria used by Cohen is derived from lab-based studies

• should not be used in an absolute sense; but in a comparative sense

What is a type 1 error?

False positive

• Occurs when we reject a true Null hypothesis

• Finds a significant effect when the effect does not exist

• denoted by α

What is a type 2 error?

False negative

• Occurs when we fail to reject a false null hypothesis

• Finds an insignificant effect when the effect does exist

• Denoted by β

What is statistical power?

The probability of rejecting a false null hypothesis, 1 - β

• is based on the assumption that the null hypothesis is false (we should reject it)

What is the conventional desirable statistical power?

80%

What are 3 reasons why power is important?

• give your study a fighting chance

• avoid unnecessary waste of resources (over and under powering)

• minimise exposure to risk

What are the 3 things that influence power?

1. Effect size - stronger effect has a better chance of finding it

2. Significance level - alpha value (should not manipulate though)

3. Sample size - bigger sample has less sampling error

What does altering significance levels do to power?

a higher significance (α) level increases statistical power

• it makes a false Null hypothesis more likely to be rejected

• (increase likelihood a type 1 error)

ALTERING SIGNIFICANCE LEVELS SHOULD BE AVOIDED

What are the two major types of statistical power analyses?

1. A priori power analysis

• what sample size do I require?

• What power can I expect with what I have? (assumes fixed sample size)

2. Post hoc power analysis

What are the 3 parameters needed for an a priori power analysis?

1. Effect size in the population

2. α

3. Sample size

What is the process for a priori power analysis?

1. Select your design and proposed method of analysis.

2. Decide on α.

3. Estimate the population effect (effect size).

4. Settle on desired power (.8, 80%).

5. Run the analysis and obtain required sample size to meet these conditions.

What does power = .8 mean?

you have an 80% chance of correctly deciding that null hypothesis is false and that something is going on.

How do you estimate an effect size for an a priori power analysis?

1. Pilot data

2. Previous research

3. Theoretical thinking

4. Reliability and validity of measures

5. Research design