Terms used interchangeably in the textbook: Measure; Measurement Procedure; Scale/Questionnaire/Inventory; Test; Instrument; Behavioral observation. All are ways of measuring a variable. They could be single-item or multiple-item.
Selecting a Measurement Procedure
Start by reviewing past literature.
If more than one procedure exists, consider which will best answer your question.
Especially consider how much sensitivity in measurement you will need.
Use the conventional way of measuring a variable so that your results can be compared to previous studies.
Constructs and Operational Definitions
Third step in the research process: Determining a method for defining and measuring the variables being studied.
Variables: characteristics or conditions that change or have different values for different individuals.
Constructs and Operational Definitions (continued)
Types of variables involved in research:
Well-defined, easily observed, and easily measured (e.g., height and weight).
Intangible, abstract attributes (e.g., motivation or self-esteem).
Measurement is more complicated for abstract attributes.
Theories and Constructs
Theory: a set of statements about mechanisms underlying a particular behavior.
Constructs (hypothetical constructs): hypothetical entities created from theory and speculation; cannot be seen, but are assumed to exist; help explain and predict behavior in a theory.
Operational Definitions
Methods of defining and measuring constructs; Variables that cannot be observed or measured directly.
Examples:
Intelligence can be operationally defined as a score on an IQ test.
“Hunger” can be operationally defined as the number of hours of food deprivation.
Reason we need to establish reliability and validity.
RELIABILITY VS. VALIDITY
Reliability: extent to which the measure is stable or consistent in producing similar scores.
Are you measuring it consistently?
Validity: extent to which the measure is assessing what it claims to measure.
Are you measuring what it’s supposed to measure?
Reliability and Validity (conceptual map)
Reliability: High / Validity: High → desirable combination
Reliability: Low / Validity: High → possible if a consistent but biased measure
Reliability: High / Validity: Low → consistently wrong
Test-retest reliability: If administered twice, are scores from the measure correlated?
Split-half reliability: If items in the measure are divided in half, are scores from the 2 halves correlated?
Inter-rater reliability: If 2 observers are rating the same behavior, are their scores correlated?
Example: marshmallow experiment
Diagrammatic example (conceptual): Time 1: Your Measure; Time 2: Your Measure; Observer 1 Ratings; Observer 2 Ratings; Your Measure: Even # Items; Your Measure: Odd # Items
CONSISTENCY OF A RELATIONSHIP
Demonstrating consistency: Two scores (one from the new measure and one from a well-established measure) are represented for each person in a scatter plot.
Positive relationship: the two measurements change together in the same direction.
Negative relationship: the two measures change in opposite directions.
Determine the consistency of a relationship by computing a correlation between the two measures.
Consistent positive relationship: a correlation near +1.00
Consistent negative relationship: a correlation near −1.00
Inconsistent relationship: a correlation near zero (≈ 0.00)
(Illustrative figures showing positive, negative, and no consistent relationship)
VALIDITY OF MEASUREMENT
To establish validity: Measurement procedure must accurately capture the variable that it is supposed to measure.
Example: Does an IQ test measure intelligence accurately?
TYPES OF VALIDITY
Face validity: Does the measure “superficially” appear to tap the underlying construct/variable? Some consider it unscientific.
Concurrent validity: Do scores from the measure correlate with different established measures of the same construct/variable?
Predictive validity: Do scores from the measure predict a behavior of interest?
Construct validity: Do scores from the measure assess what it is supposed to measure? Established if many studies using the same measure give similar results.
Representation example:
Your Measure → Behavior → Your Measure → Published Measure in the Literature → Your Measure Underlying Construct/Variable
VALIDITY CONTINUUM
Strong relationship between scores obtained from two (or more) different methods of measuring the same construct: Convergent validity.
Showing little or no relationship between the measurements of two different constructs: Divergent validity.
VALIDITY: ARE YOU MEASURING THE VARIABLE OF INTEREST?
Poor overlap: the operational definition does not align well with the construct.
Better overlap: good alignment between operational definition and construct.
High overlap: strong alignment; high construct validity.
Diagram idea: Your operational definition ↔ Construct (Variable) ↔ Your operational definition (repeated for clarity).
SCALE DEVELOPMENT AND VALIDATION
Scales are a manifestation of latent constructs; they measure behaviors, attitudes, and hypothetical scenarios we expect to exist as a result of our theoretical understanding, but cannot assess directly.
Scales typically use multiple items to measure an underlying latent construct, which helps account for item-specific measurement error and yields more accurate findings.
Thousands of scales exist to measure a range of social, psychological, and health behaviors.
As science advances and new research questions arise, new scales become necessary.
Reference: Boateng, G. O., Neilands, T. B., Frongillo, E. A., Melgar-Quiñonez, H. R., & Young, S. L. (2018). Best Practices for Developing and Validating Scales for Health, Social, and Behavioral Research: A Primer. Frontiers in Public Health, 6, 149. https://doi.org/10.3389/fpubh.2018.00149
SCHOOL KINDNESS SCALE ASSIGNMENT
1) Describe constructs
2) Types of validity that were tested (and how they were tested)
3) How was reliability tested?
RELIABILITY OF MEASUREMENT
Sources of errors:
Observer error: the individual who makes the measurements can introduce simple human error.
Environmental changes: it is difficult to attain the ideal identical circumstances.
Participant changes: the participant can change between measurements (e.g., degree of focus).
TYPES AND MEASURES OF RELIABILITY
Successive measurements:
Test-retest reliability compares scores of two successive measurements of the same individuals and correlates the scores.
Parallel forms reliability:
Simultaneous measurements.
Inter-rater reliability:
Agreement between two observers who simultaneously record measurements of the behaviors.
THE RELATIONSHIP BETWEEN RELIABILITY AND VALIDITY
Reliability is a prerequisite for validity.
A measurement procedure cannot be valid unless it is reliable.
A measurement can be valid without being reliable.
Consistency of measurement is no guarantee of validity.
DEALING WITH EQUVOCAL MEASUREMENTS
Interval or ratio scale scores are:
Compatible with basic arithmetic
Suitable for statistical analysis (e.g., mean and variance) → allow hypothesis testing.
Scores from an ordinal scale are not:
Compatible with basic arithmetic or appropriate for statistical analysis.
Distinction of a measure as interval and ratio is not always clear cut.
OTHER ASPECTS OF MEASUREMENT
Multiple measures provide more confidence in the validity of the measurements.
Can introduce problems:
Complex statistical analysis and interpretation.
Lack of agreement between two measures.
Methods for limiting problems:
Combine the measures into a single score or treat each measure individually.
SENSITIVITY AND RANGE EFFECTS
Range effect: a measurement that is not sensitive enough to detect a difference.
Ceiling effect: clustering of scores at the high end of a measurement scale; allows little or no possibility of increases in value.
Floor effect: clustering of scores at the low end of a measurement scale; allows little or no possibility of decreases in value.
ARTIFACTS: EXPERIMENTER BIAS AND PARTICIPANT REACTIVITY
Artifact: a non-natural feature accidentally introduced into something being observed (flaw in the experiment or measure).
Experimenter bias: measurements are influenced by the experimenter’s expectations regarding the outcome of the study.
LIMITING EXPERIMENTER BIAS
Standardize or automate the experiment.
Single-blind study: The research study is conducted by an experimenter who does not know the expected results.
Double-blind study: Neither the experimenter nor the participants know the expected results of the study.
DEMAND CHARACTERISTICS AND PARTICIPANT REACTIVITY
Demand characteristics refer to any potential cues or features of a study that:
Suggest to the participants the purpose and hypothesis of the study, and
Influence the participants to respond or behave in a certain way.
Reactivity: Participants modify their natural behavior in response to knowing they are in a study.
DEMAND CHARACTERISTICS AND PARTICIPANT REACTIVITY (CONT.)
Good subject role: Supports the experimenter’s hypothesis.
Negativistic subject role: Acts contrary to the hypothesis.
Apprehensive subject role: Presents himself in a “good light.”
Faithful subject role: Follows instructions to the letter (ideal participant).