Measure Psych Research

Measures

  • Terms used interchangeably in the textbook: Measure; Measurement Procedure; Scale/Questionnaire/Inventory; Test; Instrument; Behavioral observation. All are ways of measuring a variable. They could be single-item or multiple-item.

Selecting a Measurement Procedure

  • Start by reviewing past literature.
  • If more than one procedure exists, consider which will best answer your question.
  • Especially consider how much sensitivity in measurement you will need.
  • Use the conventional way of measuring a variable so that your results can be compared to previous studies.

Constructs and Operational Definitions

  • Third step in the research process: Determining a method for defining and measuring the variables being studied.
  • Variables: characteristics or conditions that change or have different values for different individuals.

Constructs and Operational Definitions (continued)

  • Types of variables involved in research:
    • Well-defined, easily observed, and easily measured (e.g., height and weight).
    • Intangible, abstract attributes (e.g., motivation or self-esteem).
    • Measurement is more complicated for abstract attributes.

Theories and Constructs

  • Theory: a set of statements about mechanisms underlying a particular behavior.
  • Constructs (hypothetical constructs): hypothetical entities created from theory and speculation; cannot be seen, but are assumed to exist; help explain and predict behavior in a theory.

Operational Definitions

  • Methods of defining and measuring constructs; Variables that cannot be observed or measured directly.
  • Examples:
    • Intelligence can be operationally defined as a score on an IQ test.
    • “Hunger” can be operationally defined as the number of hours of food deprivation.
  • Reason we need to establish reliability and validity.

RELIABILITY VS. VALIDITY

  • Reliability: extent to which the measure is stable or consistent in producing similar scores.
    • Are you measuring it consistently?
  • Validity: extent to which the measure is assessing what it claims to measure.
    • Are you measuring what it’s supposed to measure?

Reliability and Validity (conceptual map)

  • Reliability: High / Validity: High → desirable combination
  • Reliability: Low / Validity: High → possible if a consistent but biased measure
  • Reliability: High / Validity: Low → consistently wrong
  • Reliability: Low / Validity: Low → poor measurement

TYPES OF RELIABILITY

  1. Test-retest reliability: If administered twice, are scores from the measure correlated?
  2. Split-half reliability: If items in the measure are divided in half, are scores from the 2 halves correlated?
  3. Inter-rater reliability: If 2 observers are rating the same behavior, are their scores correlated?
    • Example: marshmallow experiment
  • Diagrammatic example (conceptual): Time 1: Your Measure; Time 2: Your Measure; Observer 1 Ratings; Observer 2 Ratings; Your Measure: Even # Items; Your Measure: Odd # Items

CONSISTENCY OF A RELATIONSHIP

  • Demonstrating consistency: Two scores (one from the new measure and one from a well-established measure) are represented for each person in a scatter plot.
  • Positive relationship: the two measurements change together in the same direction.
  • Negative relationship: the two measures change in opposite directions.
  • Determine the consistency of a relationship by computing a correlation between the two measures.
    • Consistent positive relationship: a correlation near +1.00+1.00
    • Consistent negative relationship: a correlation near −1.00-1.00
    • Inconsistent relationship: a correlation near zero (≈ 0.000.00)

(Illustrative figures showing positive, negative, and no consistent relationship)

VALIDITY OF MEASUREMENT

  • To establish validity: Measurement procedure must accurately capture the variable that it is supposed to measure.
  • Example: Does an IQ test measure intelligence accurately?

TYPES OF VALIDITY

  1. Face validity: Does the measure “superficially” appear to tap the underlying construct/variable? Some consider it unscientific.
  2. Concurrent validity: Do scores from the measure correlate with different established measures of the same construct/variable?
  3. Predictive validity: Do scores from the measure predict a behavior of interest?
  4. Construct validity: Do scores from the measure assess what it is supposed to measure? Established if many studies using the same measure give similar results.
  • Representation example:
    • Your Measure → Behavior → Your Measure → Published Measure in the Literature → Your Measure Underlying Construct/Variable

VALIDITY CONTINUUM

  • Strong relationship between scores obtained from two (or more) different methods of measuring the same construct: Convergent validity.
  • Showing little or no relationship between the measurements of two different constructs: Divergent validity.

VALIDITY: ARE YOU MEASURING THE VARIABLE OF INTEREST?

  • Poor overlap: the operational definition does not align well with the construct.
  • Better overlap: good alignment between operational definition and construct.
  • High overlap: strong alignment; high construct validity.
  • Diagram idea: Your operational definition ↔ Construct (Variable) ↔ Your operational definition (repeated for clarity).

SCALE DEVELOPMENT AND VALIDATION

  • Scales are a manifestation of latent constructs; they measure behaviors, attitudes, and hypothetical scenarios we expect to exist as a result of our theoretical understanding, but cannot assess directly.
  • Scales typically use multiple items to measure an underlying latent construct, which helps account for item-specific measurement error and yields more accurate findings.
  • Thousands of scales exist to measure a range of social, psychological, and health behaviors.
  • As science advances and new research questions arise, new scales become necessary.
  • Reference: Boateng, G. O., Neilands, T. B., Frongillo, E. A., Melgar-Quiñonez, H. R., & Young, S. L. (2018). Best Practices for Developing and Validating Scales for Health, Social, and Behavioral Research: A Primer. Frontiers in Public Health, 6, 149. https://doi.org/10.3389/fpubh.2018.00149

SCHOOL KINDNESS SCALE ASSIGNMENT

  • 1) Describe constructs
  • 2) Types of validity that were tested (and how they were tested)
  • 3) How was reliability tested?

RELIABILITY OF MEASUREMENT

  • Sources of errors:
    • Observer error: the individual who makes the measurements can introduce simple human error.
    • Environmental changes: it is difficult to attain the ideal identical circumstances.
    • Participant changes: the participant can change between measurements (e.g., degree of focus).

TYPES AND MEASURES OF RELIABILITY

  • Successive measurements:
    • Test-retest reliability compares scores of two successive measurements of the same individuals and correlates the scores.
  • Parallel forms reliability:
    • Simultaneous measurements.
  • Inter-rater reliability:
    • Agreement between two observers who simultaneously record measurements of the behaviors.

THE RELATIONSHIP BETWEEN RELIABILITY AND VALIDITY

  • Reliability is a prerequisite for validity.
  • A measurement procedure cannot be valid unless it is reliable.
  • A measurement can be valid without being reliable.
  • Consistency of measurement is no guarantee of validity.

DEALING WITH EQUVOCAL MEASUREMENTS

  • Interval or ratio scale scores are:
    • Compatible with basic arithmetic
    • Suitable for statistical analysis (e.g., mean and variance) → allow hypothesis testing.
  • Scores from an ordinal scale are not:
    • Compatible with basic arithmetic or appropriate for statistical analysis.
  • Distinction of a measure as interval and ratio is not always clear cut.

OTHER ASPECTS OF MEASUREMENT

  • Multiple measures provide more confidence in the validity of the measurements.
  • Can introduce problems:
    • Complex statistical analysis and interpretation.
    • Lack of agreement between two measures.
  • Methods for limiting problems:
    • Combine the measures into a single score or treat each measure individually.

SENSITIVITY AND RANGE EFFECTS

  • Range effect: a measurement that is not sensitive enough to detect a difference.
  • Ceiling effect: clustering of scores at the high end of a measurement scale; allows little or no possibility of increases in value.
  • Floor effect: clustering of scores at the low end of a measurement scale; allows little or no possibility of decreases in value.

ARTIFACTS: EXPERIMENTER BIAS AND PARTICIPANT REACTIVITY

  • Artifact: a non-natural feature accidentally introduced into something being observed (flaw in the experiment or measure).
  • Experimenter bias: measurements are influenced by the experimenter’s expectations regarding the outcome of the study.

LIMITING EXPERIMENTER BIAS

  • Standardize or automate the experiment.
  • Single-blind study: The research study is conducted by an experimenter who does not know the expected results.
  • Double-blind study: Neither the experimenter nor the participants know the expected results of the study.

DEMAND CHARACTERISTICS AND PARTICIPANT REACTIVITY

  • Demand characteristics refer to any potential cues or features of a study that:
    • Suggest to the participants the purpose and hypothesis of the study, and
    • Influence the participants to respond or behave in a certain way.
  • Reactivity: Participants modify their natural behavior in response to knowing they are in a study.

DEMAND CHARACTERISTICS AND PARTICIPANT REACTIVITY (CONT.)

  • Good subject role: Supports the experimenter’s hypothesis.
  • Negativistic subject role: Acts contrary to the hypothesis.
  • Apprehensive subject role: Presents himself in a “good light.”
  • Faithful subject role: Follows instructions to the letter (ideal participant).