Week 5 PowerPoint Notes! Exhaustive Notes on Psychometrics, Operationalization, Reliability, and Validity

Fundamentals of Psychometrics and Measurement

  • Psychometrics Defined: The psychological process of assigning numerical values or scores to represent the amounts, degrees, and kinds of specified human behaviors, mental attributes, or traits.

    • Common Psychometric Tools and Methods:
    • Questionnaire development
    • Structured and unstructured interviewing
    • Behavioral and psychological assessment
    • Standardized testing
  • Foundational Topics in Measurement:

    • Operationalizing Constructs: Translating abstract concepts into measurable variables.
    • Measurement Tools: Instruments selected to capture psychological data.
    • Reliability: Evaluating the consistency and stability of measurement instruments.

Psychological Constructs and Operationalization

  • Construct: A theoretical or hypothetical object whose existence depends entirely upon a subject's mind.

    • Requirement for Measurement: A construct must be thoroughly and well defined theoretically before it can be measured accurately.
    • Transition to Operationalization: A theoretical definition transforms into an operational definition the moment a researcher selects a specific measurement tool to quantify the construct.
  • Common Categories of Constructs Measured by Psychologists:

    • Achievement:
    • Evaluates acquired skills or knowledge gained through training or experience.
    • Examples: Scholastic skills, job-related competencies.
    • Attitude / Opinion:
    • Evaluates subjective evaluations, preferences, or beliefs.
    • Examples: Public opinion polls, consumer product evaluations.
    • Personality:
    • Evaluates enduring patterns of thinking, feeling, and behaving or psychological states.
    • Examples: Anxiety, psychoses, assertiveness.
  • Ambiguous or Hard-to-Categorize Constructs:

    • Social Skills: Can be conceptualized either as an achievement (learned social behaviors) or as a personality trait.
    • Aptitude: Can overlap with achievement or personality; measures inherent capacity for learning.
    • Intelligence Quotient (IQ): Questioned as to whether it reflects pure achievement (accumulated learned facts) or aptitude (potential capacity for future learning).

Reliability vs. Validity

  • Reliability: Refers strictly to the consistency, stability, and repeatability of a measurement tool.

  • Validity: Refers to measuring what the test actually claims to measure (truthfulness and accuracy).

  • Conceptual Distinctions:

    • Reliability focuses entirely on consistency across measurements.
    • Validity focuses entirely on truthfulness relative to the target construct.
  • Demonstrative Scenario (Andrew's Memory Test):

    • Experimental Protocol: Andrew intends to measure long-term memory. He constructs a word list, allows participants to read it for 30 seconds30\,\text{seconds}, removes the list, waits 30 seconds30\,\text{seconds}, and asks participants to recall as many words as possible.
    • Reliability Evaluation: High. If repeated under identical conditions, participants will likely obtain similar recall scores across multiple attempts.
    • Validity Evaluation: Low. A retention interval of 30 seconds30\,\text{seconds} measures short-term or working memory, not long-term memory. Thus, the test fails to measure its intended construct.
  • Importance of Reliability and Margin of Error:

    • Demonstrative Scenario (Lauren's College Entrance Exam):
    • Context: Lauren applies to a college requiring a placement test cutoff score of 80%80\%.
    • Result: Lauren scores 78%78\% and is denied admission.
    • Margin of Error: The exam possesses a known margin of error of 10%10\%
    • Implications: A margin of error of 10%10\% means Lauren's true score lies within a range surrounding her observed score (e.g., 68%68\% to 88%88\%). Because this interval spans above the 80%80\% threshold, denying admission based on a score within the error margin causes significant practical frustration.
    • Retest Expectations: If Lauren takes the exam a 2nd or 3rd time, her observed score could fluctuate anywhere within that 10%10\% margin of error due to measurement variance.

Forms and Determination of Reliability

  • Inter-Rater (Inter-Observer) Reliability:

    • Definition: Evaluates whether multiple independent observers, coders, or raters assign identical or highly consistent scores to the same observed behavior or item.
    • Example Application: Coding emotional expressions from standardized facial stimuli (e.g., categorizing emotions present in numbered items 11, 22, 33, 44, 55, 66 across different raters to evaluate scoring agreement).
  • Split-Half Reliability:

    • Definition: Evaluates internal consistency by calculating the correlation coefficient between an individual's score on the first half of a test and their score on the second half.
    • Example Scenario (James's Research Methods Exam):
    • Exam length: 5050 total questions.
    • James's overall score: 78%78\%
    • James's score on the first 2525 items: 80%80\%
    • James's score on the second 2525 items: 76%76\%
    • Conclusion: The minimal discrepancy (80%80\% vs. 76%76\%) indicates good split-half reliability.
    • Example Scenario (Hank's Exam Score):
    • Hank achieves an overall score of 80%80\% on an exam.
    • If the exam items are reliable, Hank will score approximately 80%80\% on one half of the test and approximately 80%80\% on the other half.
    • Principle: Consistent performance across test halves indicates that all items consistently measure the same underlying construct.
    • Calculation: Computed across a large sample size (NN) to establish the overall test split-half reliability coefficient.
  • Internal Consistency Reliability:

    • Core Questions:
    • Do all individual items on a scale measure a single unified construct?
    • Do the individual items add up to a meaningful total composite score?
    • Measurement Methods:
    • Cronbach's α\alpha
    • Split-half reliability
    • Alternate forms reliability
    • Cronbach's α\alpha (Alpha):
    • Definition: Mathematically represents the average of all possible split-half reliability correlations for a given measure.
    • Interpretation: A high Cronbach's α\alpha demonstrates that all test items consistently measure one single construct.
    • Calculation Example (The Ross Eating Disorders Questionnaire):
      • Item 1: "In the past week, how many times did you avoid eating or eat less than you wanted when you were hungry?"
      • Item 2: "In the past week, how many times did you eat much more than you originally intended?"
      • Item 3: "In the past week, how many times did you engaging in 'purging' behavior such a vomiting or taking laxatives?"
      • Procedure: Inter-item correlations are calculated across all potential split combinations (e.g., correlation between First 2525 items and Second 2525 items yielding r=.86,p=.02r = .86, p = .02; correlation between Odd items and Even items yielding r=.86,p=.02r = .86, p = .02). The average across all combinations forms Cronbach's α\alpha
  • Alternate Forms Reliability:

    • Definition: Evaluates consistency by calculating the correlation between an individual's scores on two different equivalent versions/forms of the same assessment.
    • Good Alternate Forms Reliability Example (James):
    • James scores 78%78\% on Form A of a Research Methods exam.
    • James scores 80%80\% on Form B of the same exam.
    • Result: Consistent scores across forms demonstrate good alternate forms reliability.
    • Bad Alternate Forms Reliability Example (Susan):
    • Susan scores 90%90\% on Form A of the exam.
    • Susan scores 70%70\% on Form B of the exam.
    • Result: Discrepant scores demonstrate bad alternate forms reliability.
    • Calculation: Obtained by calculating the correlation coefficient (rr) between scores from Form A and Form B across a sample of test takers.
  • Test-Retest Reliability:

    • Definition: Evaluates whether an assessment yields stable, identical scores over time when administered to the same individual twice, assuming the underlying construct has not naturally changed.
    • Stable Construct Example (James):
    • James takes an IQ test on Friday and scores 105105
    • If he takes the exact same test on Monday, his score should remain approximately 105105, because intelligence is a stable construct.
    • Changed Trait / External Interference Example (Luis):
    • Luis completes a generalized anxiety questionnaire on Friday.
    • Over the weekend, Luis is involved in a severe car accident.
    • On Monday, Luis retakes the anxiety questionnaire.
    • Result: His score should not necessarily remain identical, because the traumatic event altered his true baseline level of state anxiety.

Reliability and Validity Target Models

  • Visualizing Reliability vs. Construct Validity:

    • Reliability: Represented by how tightly clustered individual data items or test scores (∗*) are relative to one another.
    • Construct Validity: Represented by how centered the cluster of items (∗*) is over the true target construct circle.
  • Four Structural Target Configurations:

    • Not Reliable and Not Valid: Items (∗*) are scattered completely at random both inside and outside the target construct circle (no cluster consistency, off-target).
    • Reliable but Not Valid: Items (∗*) are tightly clustered together (∗∗∗∗****), but the cluster is located outside or off-center from the construct target circle.
    • Reliable and Valid: Items (∗*) are tightly clustered together (∗∗∗∗****) and directly centered within the target construct circle.
    • Valid but Not Reliable: Items (∗*) are scattered around the construct target circle on average, but lack a tight cluster formation.

Summary of Measurement Principles

  • Consistency as a Prerequisite: A measurement tool must produce consistent results before it can be accurate. Reliability is an absolute prerequisite for validity.
  • Methods of Reliability Determination:
    • Inter-rater reliability
    • Split-half reliability
    • Cronbach's α\alpha
    • Alternate forms reliability
    • Test-retest reliability

Review Questions and Applied Practice Scenarios

  • Review Question:

    • Scenario: A research variable of interest is processing speed/reaction time measured using a computer program. Participants complete 1010 trial items where they see a visual stimulus and must push the designated response button as quickly as possible.
    • Question: Which form of reliability should the researcher be LEAST worried about?
    • A. Test-retest reliability
    • B. Inter-rater reliability
    • C. Internal consistency reliability
    • D. Split-half reliability
    • Correct Answer: B. Inter-rater reliability.
    • Explanation: Because a computer automatically measures and records reaction times in milliseconds, human observers or raters are completely absent, eliminating inter-rater scoring variability.
  • Applied Practice Scenarios for Assessing Reliability and Validity:

    • New Diagnostic Criteria for an Eating Disorder:
    • Assessing Reliability: Administer criteria via two independent clinicians to the same patients (Inter-rater reliability) or re-diagnose stable patients over a short time interval (Test-retest reliability).
    • Assessing Validity: Correlate diagnostic results with physiological biomarkers, clinical outcomes, or established eating disorder inventories (Construct/Criterion validity).
    • Graduate School Entrance Interview:
    • Assessing Reliability: Have multiple interviewers independently rate the same applicant candidate (Inter-rater reliability).
    • Assessing Validity: Correlate interview ratings with future graduate school performance metrics such as cumulative GPA or degree completion (Predictive validity).
    • Grading Rubric for Final Research Papers:
    • Assessing Reliability: Have two teaching assistants independently grade the exact same research papers using the rubric (Inter-rater reliability).
    • Assessing Validity: Compare rubric criteria against established academic writing guidelines and course learning objectives to ensure comprehensive coverage of research quality (Content validity).
    • Movie Rating System (G, PG, PG-13, R):
    • Assessing Reliability: Have distinct rating board committees evaluate the same set of films independently (Inter-rater reliability).
    • Assessing Validity: Measure whether films assigned an R rating objectively contain higher frequencies of adult content, profanity, or violence compared to PG films (Construct validity).
    • Music Critic Reviews:
    • Assessing Reliability: Compare scoring consistency across different professional critics reviewing the same album (Inter-rater reliability).
    • Assessing Validity: Correlate critic numerical ratings with objective musicological measures or standardized audio metrics (Criterion validity).
    • Diagnostic Test for Flu Virus Presence:
    • Assessing Reliability: Run duplicate biological samples from the same patient through the diagnostic machinery repeatedly (Test-retest / Alternate forms reliability).
    • Assessing Validity: Compare test detection rates against laboratory viral culture standards to verify true positive and true negative rates (Sensitivity and Specificity / Construct validity).