Identifying Good Measurements Part 1 - Chapter 5

There are three ways to operationalize each variable in a study

1. WHAT

2. WHAT

3. WHAT

There are three ways to operationalize each variable in a study

1. Self-reports

2. Observational

3. Physiological

Operationalizing “Happiness”

• Defined “happiness” as WHAT - the conceptual definition 

• Asked people to respond to WHAT items about their satisfaction with life using a WHAT - operational definition 

• Another operational definition of happiness is the WHAT

Operationalizing “Happiness”

• Defined “happiness” as SUBJECTIVE WELL-BEING - the conceptual definition 

• Asked people to respond to FIVE items about their satisfaction with life using a 7-POINT SCALE - operational definition 

• Another operational definition of happiness is the LADDER OF LIFE SCALE

Operationalizing Other Conceptual Variables 

• Start by stating a definition of their WHAT 

• Second, create an WHAT

Operationalizing Other Conceptual Variables 

• Start by stating a definition of their CONSTRUCT 

• Second, create an OPERATIONAL DEFINITION

Self-report measures

Recording people’s answers to questions about themselves in a questionnaire or interview 

Observational measure/ behavioural measures  

Recording observable behaviours or physical traces of behaviours

Psychological measures

Recording biological data, such as brain activity or volume, hormone levels, eye movement or heart rate 

Categorical variables or nominal variables

A variable whose levels are categories (eg, languages)

Quantitative variables or continuous variables

A variable whose values can be recorded as meaningful numbers.(eg, height or weight) 

The three types of quantitative variables are

1. WHAT

2. WHAT

3. WHAT

The three types of quantitative variables are

1. Ordinal scale

2. Interval scale

3. Ratio scale

Ordinal scale

levels represent a ranked order, and in which distances between levels are not equal (e.g., order of finishers in a race).

Interval scale

has no “true zero,” (zero degrees does not mean no temperature) and in which the numerals represent equal intervals (distances) between levels

Ratio scale

The numerals of a quantitative variable have equal intervals and when the values of zero truly means “none” of the variable is being measured

The construct validity of a measure has two aspects

1. WHAT

2. WHAT

The construct validity of a measure has two aspects

1. Reliability

2. Validity

Reliability

How consistent the results of a measure are

Validity

Whether it’s actually measuring the construct it’s supposed to measure

If an operationalization is reliable, it will yield a WHAT pattern of scores every time 

If an operationalization is reliable, it will yield a CONSISTENT pattern of scores every time 

What are the three types of reliability (which all involve consistency)

• WHAT

• WHAT

• WHAT

What are the three types of reliability (which all involve consistency)

• Test-retest reliability

• Interrater reliability

• Internal reliability / internal consistency

Test-restest reliability

Study participants will get the same score each time they are tested with it

Interrater reliability

Consistent scores are obtained no matter who is rating the variable

Internal reliability / Internal consistency

Participant gives a consistent pattern of answers, no matter how the researchers phrase the question

For a measure to be reliable in should be a linear WHAT slope on a scatterplot 

For a measure to be reliable in should be a linear POSITIVE slope on a scatterplot 

Correlation Coefficient or r 

A single number, ranging from –1.0 to 1.0, that indicates the strength and direction (slope) of an association between two variables.

The spread of dots in a scatterplot corresponds to the WHAT of the relationship 

The spread of dots in a scatterplot corresponds to the STRENGTH of the relationship 

The relationship is strong when the dots are close to the WHAT and weak when the dots are WHAT

The relationship is strong when the dots are close to the LINE and weak when the dots are SPREAD OUT

When the slope is positive r is WHAT and when the slope is negative r is WHAT 

When the slope is positive r is POSITIVE and when the slope is negative r is NEGATIVE 

r falls anywhere between WHAT to WHAT

r falls anywhere between -1.0 to 1.0 

• The stronger the relationship is the closer r is to WHAT

• The weaker the relationship the closer r is to WHAT

• The stronger the relationship is the closer r is to -1.0 or 1.0 

• The weaker the relationship the closer r is to 0 

Test-Retest Reliability 

• For constructs that are supposed to stay WHAT over time 

• Used when researchers are measuring constructs that are theoretically WHAT over time (adult height) 

Test-Retest Reliability 

• For constructs that are supposed to stay STABLE over time 

• Used when researchers are measuring constructs that are theoretically STABLE over time (adult height) 

Test-Retest Reliability 

• Can apply this reliability to WHAT, WHAT and WHAT measures 

Test-Retest Reliability 

• Can apply this reliability to SELF-REPORT, OBSERVATION and PHYSIOLOGICAL measures 

Test-Retest Reliability 

• Patterns should be WHAT. If r turns out to be WHAT and WHAT (0.5 or higher), the measure would have very good test-retest reliability 

Test-Retest Reliability 

• Patterns should be CONSISTENT. If r turns out to be POSITIVE and STRONG (0.5 or higher), the measure would have very good test-retest reliability 

Interrater Reliability 

• Important for WHAT measures 

• When WHAT coders or observers are WHAT behaviour 

Interrater Reliability 

• Important for OBSERVATION measures 

• When TWO coders or observers are RATING behaviour 

Interrater Reliability 

• If r is WHAT and WHAT (.7 or higher) we would have very good interrater reliability 

• A WHAT r would indicate a big problem 

Interrater Reliability 

• If r is POSITIVE and STRONG (.7 or higher) we would have very good interrater reliability 

• A NEGATIVE r would indicate a big problem 

Interrater Reliability - Cohen’s kappa

used when a observers rating a categorical variable (closer to 1 the better)

Internal Reliability 

• When a measure has WHAT items that all measure the same WHAT 

Internal Reliability 

• When a measure has MULTIPLE items that all measure the same CONSTRUCT 

Internal Reliability - Average interitem correlation (AIC)

The average of all these correlations (0.15- 0.50 means items go together) 

Internal Reliability - Cronbach’s alpha (coefficient alpha or a)

mathematically combines the AIC and the number of items in the scale (closer to 1 the better the internal reliability)