Chapter 4: Significance, Effect Size and Power

Statistical significance

The probability that an observed outcome has occurred by chance

Effect size

Describes the strength of the relationship is relation to the sample size and average variation

Statistical power

Describes the extent to which the data are robust enough to find that effect

What statistical power should researcher aim for?

At least 80% of occasions to correctly reject the null, to avoid getting Type II errors

Experimental/alternative hypothesis

Predict that observed differences in a outcome between groups of people was due to the factor that we're examining

Null hypothesis

States there are no differences or that observed differences were due to chance

One tailed hypothesis

A specific prediction regarding the direction of an outcome, stating how the variables will differ

Two tailed hypothesis

A non specific prediction just stating there will a difference/relationship

Type I error

Where the null hypothesis is rejected when it should have been accepted

Type II error

Where we fail to reject the null hypothesis when we should have done so

Significance with one tailed tests

Usually set the significance level at 5%, the outcome should reside in the outer 5% of one end of the distribution

In a one tailed test, if Y is greater than X, the significance should be...

The significance should be in the lower 5% of the distribution

In a one tailed test, if X is greater than Y, the significance should be...

The significance should be in the upper 5% of the distribution

Significance with two tailed tests

Significance is set to 5% but we have to share that between two tails, so the level at either end is 2.5%

Reasons to get a Type I error

• significance level is too high

• too many analyses

• biased data/analyses

• biased data/analyses

Reasons to get a Type II error

• not enough samples

• too many outliers relative to the sample size

• the design might lead to inconsistent responses

Replication

Useful to reduce type I and II errors, if other researches achieve similar results, it strengthens the claim

Variance

The extent that scores vary around the mean

Standard deviation

The average variation of scores, in relation to the sample mean

Standard error

The average/estimated variation of scores in a sampling distribution/population

Sampling distribution

A theoretical calculation of all possible samples in a population

Standard error of difference

An estimate of the SD in the sampling distribution representing differences between two samples/conditions

Confidence intervals

An estimate of the range of values likely to be included within a given proportion of a sampling distribution

Confidence intervals of difference

An estimate of the range of values within a given proportion that represents differences between two samples/conditions

Parametric tests

Base outcomes on mean scores; significance often focuses on how mean scores differ between groups/conditions

Significance in non parametric tests

Focus on median scores and how ranked scores differ between groups/conditions

Pearson's r effect size

Focuses on associations between samples dn is often used in correlation

Cohen's d effect size

Examining differences relative to sample sized and pooled standard deviation