1/34
Ways of measuring variables, reliability and validity of measurement
What 2 definitions are used in measurement?
Conceptual definition/construct = the definition of the variable in question at a theoretical level
Start by stating the definition of their construct (conceptual variable) -> create operational definition
Operational definition = reps a researcher's specific decision about how to measure/manipulate the conceptual variable for their study
Psych research needs both definitions
What are 3 common types of measurement?
Self-reports
Observational/behavioural measures
Physiological measures
What are self-report measures?
Operationalise a variable by recording people’s answers to questions about themselves in a questionnaire or interview
In research on children, self-reports may be replaced with parent reports or teacher reports
What are observational/behavioural measurement?
Operationalizes a variable by recording observable behaviours or physical traces of behaviours
May record physical traces of behaviour (e.g. stress behaviours could be measured by counting the number of tooth marks left on a person’s pencil)
What are physiological measures?
Operationalizes a variable by recording biological data, such as brain activity or volume, hormone levels, eye movements, or heart rate
Require use of special equipment -> e.g. facial EMG used to detect happy facial expression
FMRIs
E.g. stress
Measure the amount of cortisol released in saliva because people under stress show higher levels of cortisol (Carlson, 2009)
Skin conductance → electronic recording of the activity in the sweat glands of the hands or feet People under more stress have more activity in these glands
Detecting electrical patterns in different brain regions near the scalp using EEG
Must be validated by using other measures
E.g. using fMRI to study brain activity relative to intelligence BUT PPTs were given an IQ test (OM) to know their intelligence (Dubois et al., 2018)
How are operational variables primarily classified?
Categorical (nominal)
Continuous (quantitative)
How can nominal data be described?
Using the mode
What are the 3 types of quantitative variables?
Ordinal scale
Interval scale
Ratio scale

What is an ordinal scale?
Measurement applies when the numerals of a quantitative variable represent a ranked order
May or may not be numeric
Can be described using the median, mode or range
What is an interval scale?
Requires two conditions:
Unlike ordinal scales, the numerals represent equal intervals (distances) between levels
There is no “true zero” → a person can get a score of 0, but the 0 does not literally mean “nothing”
Can be described using the mean, median, mode, range, or SD
What is a ratio scale?
Measurement applies when the numerals of a quantitative variable have equal intervals and when the value of 0 truly means “none” of the variable being measured
Can be described using the mean, median, mode, range, or SD
Scales of measurement diagram

What is reliability?
How consistent the results of a measure are
What are the ways of measuring reliability?
Test-retest
Interrater
Internal reliability/consistency
What is test-retest reliability?
A PPT will get pretty much the same score each time they are tested with it
Needs to be present when researchers are measuring constructs (such as intelligence or personality traits) that are theoretically stable over time
Can apply whether the operationalization is self-report, observational, or physiological
Depends on whether the variable is supposed to stay the same over time (e.g. intelligence vs allergies/stress)
What is interrater reliability?
Consistent scores are obtained no matter who is rating the variable
Important for observational measures
If r is positive and strong (according to many researchers, r = .70 or higher), we would have very good interrater reliability
When we’re assessing reliability, a negative correlation is rare and undesirable
r can be used to evaluate interrater reliability when the observers are rating a quantitative variable BUT Cohen’s kappa is used when the observers are rating a categorical variable
Measures the extent to which two raters place participants into the same categories
A kappa close to 1.0 (ideally, .80 or higher) means that the two raters agreed
What in internal reliability/connsistency?
A PPT gives a consistent pattern of answers, no matter how the researchers phrase the question
Relevant for measures that use multiple items or observations to get at the same construct
Applies when researchers combine multiple items or observations into one score
Steps:
Researchers ask a large sample of PPTs to answer all of the items
Compute the correlations between every item + every other item
Compute the average inter-item correlation (AIC - average of all these correlations)
An AIC between .15 and .50 means that the items go reasonably well together (Clark & Watson, 2019)
Might compute Cronbach’s alpha (coefficient α - mathematically combines the AIC + the number of items in the scale)
The closer Cronbach’s alpha is to 1.0, the better the internal reliability
For self-report measures, researchers are looking for α = .80 or higher (Clark & Watson, 2019)
If the AIC or Cronbach’s alpha is acceptable, the researchers can combine all the items together
If the AIC or Cronbach’s alpha is unacceptable, the researchers look carefully at the items → may revise or omit rogue items from combined score
What is used to visualise reliability?
Scatterplots used to visualise → spread of dots corresponds to the strength of the relationship
What is the correlation coefficient?
Single number that indicates the slope and spread of the dots on a scatterplot (-1-1)
Slope direction can be + - or 0
What does validity question?
Whether it’s actually measuring the construct it’s supposed to measure
What are the 3 types of validity?
Face validity
Content validity
Criterion validity
What is face validity?
A measure has face validity if it is subjectively considered to be a plausible operationalization of the conceptual variable in question
Researchers might check face validity by consulting experts
What is content validity?
Requires knowledge of the conceptual definition; a measure must capture all parts of a defined construct
What do face and content validity establish?
That the operationalizations look consistent with the conceptual definition
BUT the other forms of validity require data → we collect data to test that the measurement is associated with something it theoretically should be associated with
Both are subjective assessments of validity
What is criterion validity?
Evaluates whether the measure under consideration is associated with a concrete behavioural outcome that it should be associated with, according to the conceptual definition
Especially important for self-report measures because the correlation can indicate how well people’s self-reports predict their actual behaviour
Provides strong evidence for a measure’s construct validity
What is a testing example of criterion validity?
To assess criterion validity, your company would give both sales tests to all the current sales representatives and then find out each person’s sales figures as a measure of their selling performance. You would then compute two correlations: one between Sales Aptitude Test A and sales figures, and the other between Sales Aptitude Test B and sales figures
Future sales performance is correlated more strongly with scores on Test A than with scores on Test B. If the data looked like this, the company would conclude that Test A has better criterion validity as a measure of selling ability, so it would be better for hiring salespeople. In contrast, the other data show that scores on Test B do not predict future sales performance; it has poor criterion validity as a measure of sales aptitude
What is a way of gathering evidence for criterion validity?
Correlations
What is the known-groups paradigm and what can it be used for?
researchers see whether scores on the measure can discriminate among two or more groups whose behaviour is already confirmed
Way of gathering evidence for criterion validity
Can also be used to validate self-report measures
e.g. BDI → the average BDI score of the known group of depressed people was higher than the average score of the known group who were not depressed
What is convergent validity?
The patterns of correlations with measures of theoretically similar constructs
Convergent validity evidence also includes similar constructs, not just the same one
E.g. Segal (2008)
BDI scores were positively and strongly correlated with CES-D scores -> If the BDI really quantifies depression, the researchers reasoned, it should be correlated with (should converge with) other self-report measures of depression
This correlation between similar self-report measures of the same construct (depression) provided evidence for the convergent validity of the BDI
What is divergent (discriminant) validity?
The patterns of correlations with measures of theoretically dissimilar constructs
E.g. consider that many developmental disorders have similar symptoms, but diagnoses can vary. It might be important to specify, for instance, whether a child has autism or only a language delay. Therefore, a screening instrument for identifying autism should have discriminant validity; it should not accidentally diagnose that same child as having a language delay. Similarly, a scale that is supposed to diagnose learning disabilities should not be correlated with IQ because learning disabilities are not related to general intelligence.
What types of validity are usually evaluated together as a patter of correlations among self-report measures?
Convergent + divergent validity
How do convergent and divergent validity relate to measurement?
A measurement should have higher correlations (higher r values) with similar traits (convergent validity) than it does with dissimilar traits (discriminant validity)
What can convergent and divergent validity provide key evidence for?
Construct validity → the overall pattern of convergent and discriminant validity correlations helps researchers decide BUT it’s often even more important for a self-report measure to predict behaviour, as in criterion validity
Describe the relationship between reliability and validity
A measure can be less valid than it is reliable BUT can’t be more valid than it is reliable
Reliability has to do with how well a measure correlates with itself
Validity has to do with how well a measure is associated with something else
If a measure does not even correlate with itself, then how can it be more strongly associated with some other variable?)
Reliability is necessary (but not sufficient) for validity
Validity and reliability concept map
