09. Reliability and Validity

Reliability

Consistency and dependability of a measure, to produce the same or similar results when repeated

Internal reliability

The extent to which a measure is consistent within itself

Split-half reliability

Compares the results of one half of a test with the other half

Cronbach’s Alpha

A test that splits items equally in every way possible, correlates all halves with all other halves, should be >.70.

External Reliability

The extent to which a measure varies from one use to another

Test-retest Reliability

The stability of a test over time, a good test is consistently reliable

Inter-rater reliability

Used in observational studies, the degree to which different raters give consistent estimates of the same behaviour

Improving reliability

• Improve quality of items, clear and unambiguous

• Increase/decrease number of items

• Increase sample size and control individual differences

• Choose appropriate sample

• Control conditions

Sensitivity

Detecting even a small effect of the IV on the DV

Factors when selecting a sample

• Choose an appropriate and large sample

• Find the right demographic for your research question

• Control conditions

• Choose tasks and measures to ensure a range of scores

Validity

Whether a test measures what it aims to measure

Face Validity

Whether the test appears to measure what it claims to

Content Validity

Does it cover the full range of symptoms/facets of a construct

Construct Validity

The degree to which a test measures the construct/psychological concept at which it is aimed

Convergent Validity

The degree to which it correlates with other measurements assessing the same construct

Divergent/Discriminant Validity

The degree to which it does not correlate with other measurements assessing different concepts

Criterion Validity

Whether a test reflects a certain set of abilities, the degree to which a measurement can accurately predict specific criterion variables

Concurrent Validity

How well a test correlates with a previously validated measure, given at the same time

Predictive Validity

How well a test predicts future performance

External Validity

The extent to which the results of a study can be generalised to different populations, settings and conditions

Ways to ensure high external validity

• Extend to other populations, new people and situations

• High construct validity

• Use a representative sample

• Replicate using a new group

Internal Validity

When we can be confident that manipulating the IV affects the DV, a causal relationship

Internal Validity: Causation Criteria

• Show correlation/co-variation

• Show time-order relationships

• Eliminate all other possible causes

Threats to Internal Validity

• Testing intact groups

• Order/practice/fatigue/transfer effects

• Presence of extraneous variables

• Unequal loss across groups

• Expectancy effects/demand characteristics

Ecological Validity

How much a method of research resembles ‘real life’ and can therefore be generalised across different settings