Chapter 5 - identifying good measurement (construct validity)

measuring variables

reliability needs to occur, but isnt sufficient for validity

Conceptual variable

• construct/theoretical 

• The abstract concept being studied. (Eg. happiness)

• Conceptual definition: researcher’s definition of the variable in question at a theoretical level

Operational variable

• The concrete concept being studied

• Operational definition: describes how the variable will be measured or manipulated 

Example: course enjoyment - using operational definitions

• Operational definition 1

  • Asking students “ are you enjoying the course; yes or no?”

• Operational definition 2

• Asking students “on a scale of 1 - 10, 10 being the highest, how much are you enjoying the course?”

• Operational definition 3

  • Asking students about the course and scoring responses as positive or negative 

3 common types of measures

1. Self-report measure

2. Observational measure

3. Physiological measures 

Self report measures

• Operationalize a variable by recording people’s answers about themselves in a questionnaire or interview

  • Ex.  operationalizing caffeine consumption by asking “how many caffeinated beverages have you consumed today?”

Observational measures

• Also called behavioural measures

• Operationalize a variable by recording observable behaviours or using physical traces of behaviours 

  • Ex. operationalizing allergies by observing how often someone sneezes

  • Ex. operationalizing cleanliness by quantifying how much dust someone has on their furniture 

Physiological measures

• Operationalize a variable by recording biological data (ex. Brain activity, hormone levels, heart rate)

  • Ex. operationalizing whether someone can distinguish between 2 different speech sounds by examining their brian response to those speech sounds 

Which measure is best?

• Depends on the concept and the conclusions you wish to draw

• Variables can be operationalized in many ways across all the types of measures

• Ideally, all types of measures should show similar patterns 

Scales of measurement 

• Variables by definition must have at least 2 levels to allow them to change 

  • However, levels can be coded using different scales of measurement 

    • Categorical variables

    • Quantitative variables 

      • Ordinal

      • Interval 

      • Ratio

Categorical (nominal) variables

 levels are qualitatively distinct categories 

• Ex. geographical region

Quantitative variables

levels are meaningful numerical values

• Ex. happiness on a scale of 1-10 

• 3 types:

  • Ordinal scale: numbers represent a ranked order, but internals are not equal (ex. 1st, 2nd, 3rd)

  • Interval scale: numbers represent equal distances between levels and there is no true zero (ex. IQ score - there is lowest and highest but IQ cant be 0, shoe sizes)

  • Ratio scale: numbers represent equal intervals and there is a true zero (ex. Heights or exam scores)

Reliability

• Looks at how consistent the results of the measure are 

3 types of reliability 

1. Test-retest reliability

2. Interrater reliability 

3. Internal reliability 

Test-retest reliability

• Consistent scores every time the measure is used 

  • Ex. IQ test one at the beginning of the term and another at the end of the term 

    • Scores should be relatively consistent across the two tests 

    • Points in scatter plot should follow the line of best fit 

Interrater reliability

• Consistent scores no matter who does the measuring 

  • Ex. two observers measure how often a child smiles during one hour at a daycare playground 

    • Counts should be consistent between both raters 

      • Observer data points in scatter plot should follow the line of best fit 

Internal reliability

• Internal consistency

• Consistent pattern of responses, regardless of how the researcher has phrased the question

  • Ex. researcher asks in several different ways about how lonely you feel 

    • Responses should be consistent with one another 

    • More likely to show in a comparison chart rather than scatterplot 

Scatter plots and correlation coefficients 

• Scatterplots are a good way to visualize reliability

• Correlation coefficient (r): statistical measure describing the strength of the linear relationship between two variables

  • Tells us how close the dots in put scatterplot fall to the line of best fit  

Correlation coefficient for reliability 

• Slope direction: tells us if the relationship is positive, negative, or zero

  • Absolute value: tells us how strong the relationship is 

r for test-retest reliability

• Looking for correlation between measure at time 1 and time 2

  • If r is strong and positive (0.05 or above), then it is good test-retest reliability

  • If r is positive but weak, sign of low test-retest reliability

    • You cannot have a negative r

r for interrater reliability

• Looking for correlation between measure by person 1 and person 2

• If r is + and strong (0.70 or higher), good interrater reliability 

• If r is + but weak (less than 0.70), do not have good interrater reliability 

  • If r is negative, terrible interrater reliability 

R for internal reliability 

• Looking for correlation between scores for two or more measures that are intended to quantify the same construct 

  • If items correlate with one another, you have strong internal reliability

    • Can average score of all those items to get a single overall score 

    • Usually are no absolute number cutoffs

• Cronhacb’s alpha (ɑ): statistics usually used to evaluate internal reliability

  • Values range between 0 and 1 (1 is no variance) 

  • 0 is not similar, 1 is the same 

• Calculate correlation between pairs of items, them calculate average inter-item correlation (AIC)

  • Are these items capturing the same concept?

Validity

• Looks at whether a measure is actually measuring the construct its supposed to be 

Face validity (subjective)

 it looks like what you want to measure

• Ex. can ask experts if a new measure seems to match existing standards for same construct → experts (people relevant who has experience in the field) should usually be present for face validity 

Content validity (subjective)

The measure contains all the parts that your theory says it should contain 

• Ex. IQ test consistent with 3-tier models of intelligence 

Criterion validity (empirical)

whether the measure is related to a concrete outcome that it should be related to 

• Ex. aptitude tests → asking what potential performance is (ex. Driving test)

Known-group paradigm (empirical)

whether scores on the measure can distinguish among a set of groups whose behaviour is already well understood (categories)

• Ex. among a group of depressed people (known group), scores can be categorized into sub-categories (ex. severe, moderate, mild, none)

Convergent validity (empirical)

a measure should correlate strongly with other measures of the same construction; similarity → ex. Two tests trying to show the same result should be similar in construct

Discriminant (divergent) validity (empirical)

a measure should correlate less strongly with measures of different constructs; there must be differences → two different concepts should have different constructs 

Relationship between reliability and validity

• Validity of measure is not the same as its reliability  

• A measure can be less valid than it is reliable, but cannot be more valid than it is reliable 

• Reliability is not necessary but not sufficient for validity