Chapter 5: Identifying good measures

All behavioral research involves

measurement of some biological, social, cognitive element

Measures should be

1. reliable

2. accurate

Three types of measures

1. observational

2. self-report

3. physiological

Observational measure

recording observable behaviors based on operationalized variable

Self-report measures

operationalizes by recording people’s answers to questions (typically about self) in a questionnaire or interview

Physiological Measures

Operationalizes by recording biological data

4 scales of measurement

1. nominal

2. ordinal

3. interval

4. ratio

type of scale determines which statistics allowed to use

nominal scales

numbers serve only as labels

• no numerical meaning

• describes categories

ex: yes=1 no=0, brunette=1 blonde=2 ginger=3

Ordinal scale

• Rank ordering of data

• Differences between values are not equal

Interval Scale

• ordered scale

• equal intervals

• no natural zero

Ratio Scale

• ordered scale

• equal intervals

• natural zero

natural/true zero

total abscence of value.

Reliability

Consistency of measuring technique

Two reasons for people to score differently

1. Actual Difference

2. Measurement error

Measurement Error

Factors that distort a score

Sources of measurement error

1. Transient states of participant (hunger level)

2. Stable attributes of participant (stage fright)

3. Situational (freezing temperature)

4. Characteristics of measurement (unfamiliar wording)

5. experimentor error (mistype)

3 types of measurement reliability tests

1. test-retest

2. inter-rater

3. internal/interitem

Benchmark acceptable reliability score

>0.70 or >70%

Test-retest

reliability test: consistent responses on the same measure over time

(good for stable measures, bad for transient)

Internal/interitem reliability

Degree that all items/questions in a scale assess the same construct

correlation of answers across questions

Cronbach’s alpha coefficient measures internal reliability

Cronbach’s alpha coefficient

correlation based statistic that measures a scales internal reliability

(How correlated are items in scale)

Inter-rater reliability

the degree to which two or more coders/observers give consistent ratings of a set of targets

Two ways to assess Inter-rater reliability

1. percent of agreement between raters

2. correlation between rates

How to increase reliability

1. standardize measure administration

2. clarify instructions and questions

3. Train observers

4. minimize coding errors

5 types of measurement validity for construct validity

1. Face (subjective)

2. Content (subjective)

3. Criterion (empirical)

4. Convergent (empirical)

5. Discriminant (empirical)

Face Validity

Does the measure appear to measure what its supposed to be?

(Face validity is not actual validity) but increases confidence in results

True/False some tests have intentionally low face validity

True

Content Validity

Does the measure capture all parts of the conceptual definition?

Criterion-Related Validity

Does the measure allow you to distinguish people based on a particular behavioral outcome?

Compare new measure to behavior

Researchers identify behaviors that the measure should be related to, then measure their correlation

Two types of Criterion-Related Validity

1. Concurrent: measure and behavior outcome administered at the same time

2. Predictive: measure administered days, weeks, months, years, before behavioral outcome measure is taken

Convergent Validity

Does the measure strongly positively/negativaly correlate with other pre-existing measure assesing similar/opposite constructs

Discriminant Validity

Does the measure weakly relate to other pre-established measures with little to nothing to do with the construct of interest?

A measure can be reliable but not valid, HOWEVER

a measure cannot be valid without being reliable