Week 5 PowerPoint Notes! Exhaustive Notes on Psychometrics, Operationalization, Reliability, and Validity
Fundamentals of Psychometrics and Measurement
Psychometrics Defined: The psychological process of assigning numerical values or scores to represent the amounts, degrees, and kinds of specified human behaviors, mental attributes, or traits.
- Common Psychometric Tools and Methods:
- Questionnaire development
- Structured and unstructured interviewing
- Behavioral and psychological assessment
- Standardized testing
Foundational Topics in Measurement:
- Operationalizing Constructs: Translating abstract concepts into measurable variables.
- Measurement Tools: Instruments selected to capture psychological data.
- Reliability: Evaluating the consistency and stability of measurement instruments.
Psychological Constructs and Operationalization
Construct: A theoretical or hypothetical object whose existence depends entirely upon a subject's mind.
- Requirement for Measurement: A construct must be thoroughly and well defined theoretically before it can be measured accurately.
- Transition to Operationalization: A theoretical definition transforms into an operational definition the moment a researcher selects a specific measurement tool to quantify the construct.
Common Categories of Constructs Measured by Psychologists:
- Achievement:
- Evaluates acquired skills or knowledge gained through training or experience.
- Examples: Scholastic skills, job-related competencies.
- Attitude / Opinion:
- Evaluates subjective evaluations, preferences, or beliefs.
- Examples: Public opinion polls, consumer product evaluations.
- Personality:
- Evaluates enduring patterns of thinking, feeling, and behaving or psychological states.
- Examples: Anxiety, psychoses, assertiveness.
Ambiguous or Hard-to-Categorize Constructs:
- Social Skills: Can be conceptualized either as an achievement (learned social behaviors) or as a personality trait.
- Aptitude: Can overlap with achievement or personality; measures inherent capacity for learning.
- Intelligence Quotient (IQ): Questioned as to whether it reflects pure achievement (accumulated learned facts) or aptitude (potential capacity for future learning).
Reliability vs. Validity
Reliability: Refers strictly to the consistency, stability, and repeatability of a measurement tool.
Validity: Refers to measuring what the test actually claims to measure (truthfulness and accuracy).
Conceptual Distinctions:
- Reliability focuses entirely on consistency across measurements.
- Validity focuses entirely on truthfulness relative to the target construct.
Demonstrative Scenario (Andrew's Memory Test):
- Experimental Protocol: Andrew intends to measure long-term memory. He constructs a word list, allows participants to read it for , removes the list, waits , and asks participants to recall as many words as possible.
- Reliability Evaluation: High. If repeated under identical conditions, participants will likely obtain similar recall scores across multiple attempts.
- Validity Evaluation: Low. A retention interval of measures short-term or working memory, not long-term memory. Thus, the test fails to measure its intended construct.
Importance of Reliability and Margin of Error:
- Demonstrative Scenario (Lauren's College Entrance Exam):
- Context: Lauren applies to a college requiring a placement test cutoff score of .
- Result: Lauren scores and is denied admission.
- Margin of Error: The exam possesses a known margin of error of
- Implications: A margin of error of means Lauren's true score lies within a range surrounding her observed score (e.g., to ). Because this interval spans above the threshold, denying admission based on a score within the error margin causes significant practical frustration.
- Retest Expectations: If Lauren takes the exam a 2nd or 3rd time, her observed score could fluctuate anywhere within that margin of error due to measurement variance.
Forms and Determination of Reliability
Inter-Rater (Inter-Observer) Reliability:
- Definition: Evaluates whether multiple independent observers, coders, or raters assign identical or highly consistent scores to the same observed behavior or item.
- Example Application: Coding emotional expressions from standardized facial stimuli (e.g., categorizing emotions present in numbered items , , , , , across different raters to evaluate scoring agreement).
Split-Half Reliability:
- Definition: Evaluates internal consistency by calculating the correlation coefficient between an individual's score on the first half of a test and their score on the second half.
- Example Scenario (James's Research Methods Exam):
- Exam length: total questions.
- James's overall score:
- James's score on the first items:
- James's score on the second items:
- Conclusion: The minimal discrepancy ( vs. ) indicates good split-half reliability.
- Example Scenario (Hank's Exam Score):
- Hank achieves an overall score of on an exam.
- If the exam items are reliable, Hank will score approximately on one half of the test and approximately on the other half.
- Principle: Consistent performance across test halves indicates that all items consistently measure the same underlying construct.
- Calculation: Computed across a large sample size () to establish the overall test split-half reliability coefficient.
Internal Consistency Reliability:
- Core Questions:
- Do all individual items on a scale measure a single unified construct?
- Do the individual items add up to a meaningful total composite score?
- Measurement Methods:
- Cronbach's
- Split-half reliability
- Alternate forms reliability
- Cronbach's (Alpha):
- Definition: Mathematically represents the average of all possible split-half reliability correlations for a given measure.
- Interpretation: A high Cronbach's demonstrates that all test items consistently measure one single construct.
- Calculation Example (The Ross Eating Disorders Questionnaire):
- Item 1: "In the past week, how many times did you avoid eating or eat less than you wanted when you were hungry?"
- Item 2: "In the past week, how many times did you eat much more than you originally intended?"
- Item 3: "In the past week, how many times did you engaging in 'purging' behavior such a vomiting or taking laxatives?"
- Procedure: Inter-item correlations are calculated across all potential split combinations (e.g., correlation between First items and Second items yielding ; correlation between Odd items and Even items yielding ). The average across all combinations forms Cronbach's
Alternate Forms Reliability:
- Definition: Evaluates consistency by calculating the correlation between an individual's scores on two different equivalent versions/forms of the same assessment.
- Good Alternate Forms Reliability Example (James):
- James scores on Form A of a Research Methods exam.
- James scores on Form B of the same exam.
- Result: Consistent scores across forms demonstrate good alternate forms reliability.
- Bad Alternate Forms Reliability Example (Susan):
- Susan scores on Form A of the exam.
- Susan scores on Form B of the exam.
- Result: Discrepant scores demonstrate bad alternate forms reliability.
- Calculation: Obtained by calculating the correlation coefficient () between scores from Form A and Form B across a sample of test takers.
Test-Retest Reliability:
- Definition: Evaluates whether an assessment yields stable, identical scores over time when administered to the same individual twice, assuming the underlying construct has not naturally changed.
- Stable Construct Example (James):
- James takes an IQ test on Friday and scores
- If he takes the exact same test on Monday, his score should remain approximately , because intelligence is a stable construct.
- Changed Trait / External Interference Example (Luis):
- Luis completes a generalized anxiety questionnaire on Friday.
- Over the weekend, Luis is involved in a severe car accident.
- On Monday, Luis retakes the anxiety questionnaire.
- Result: His score should not necessarily remain identical, because the traumatic event altered his true baseline level of state anxiety.
Reliability and Validity Target Models
Visualizing Reliability vs. Construct Validity:
- Reliability: Represented by how tightly clustered individual data items or test scores () are relative to one another.
- Construct Validity: Represented by how centered the cluster of items () is over the true target construct circle.
Four Structural Target Configurations:
- Not Reliable and Not Valid: Items () are scattered completely at random both inside and outside the target construct circle (no cluster consistency, off-target).
- Reliable but Not Valid: Items () are tightly clustered together (), but the cluster is located outside or off-center from the construct target circle.
- Reliable and Valid: Items () are tightly clustered together () and directly centered within the target construct circle.
- Valid but Not Reliable: Items () are scattered around the construct target circle on average, but lack a tight cluster formation.
Summary of Measurement Principles
- Consistency as a Prerequisite: A measurement tool must produce consistent results before it can be accurate. Reliability is an absolute prerequisite for validity.
- Methods of Reliability Determination:
- Inter-rater reliability
- Split-half reliability
- Cronbach's
- Alternate forms reliability
- Test-retest reliability
Review Questions and Applied Practice Scenarios
Review Question:
- Scenario: A research variable of interest is processing speed/reaction time measured using a computer program. Participants complete trial items where they see a visual stimulus and must push the designated response button as quickly as possible.
- Question: Which form of reliability should the researcher be LEAST worried about?
- A. Test-retest reliability
- B. Inter-rater reliability
- C. Internal consistency reliability
- D. Split-half reliability
- Correct Answer: B. Inter-rater reliability.
- Explanation: Because a computer automatically measures and records reaction times in milliseconds, human observers or raters are completely absent, eliminating inter-rater scoring variability.
Applied Practice Scenarios for Assessing Reliability and Validity:
- New Diagnostic Criteria for an Eating Disorder:
- Assessing Reliability: Administer criteria via two independent clinicians to the same patients (Inter-rater reliability) or re-diagnose stable patients over a short time interval (Test-retest reliability).
- Assessing Validity: Correlate diagnostic results with physiological biomarkers, clinical outcomes, or established eating disorder inventories (Construct/Criterion validity).
- Graduate School Entrance Interview:
- Assessing Reliability: Have multiple interviewers independently rate the same applicant candidate (Inter-rater reliability).
- Assessing Validity: Correlate interview ratings with future graduate school performance metrics such as cumulative GPA or degree completion (Predictive validity).
- Grading Rubric for Final Research Papers:
- Assessing Reliability: Have two teaching assistants independently grade the exact same research papers using the rubric (Inter-rater reliability).
- Assessing Validity: Compare rubric criteria against established academic writing guidelines and course learning objectives to ensure comprehensive coverage of research quality (Content validity).
- Movie Rating System (G, PG, PG-13, R):
- Assessing Reliability: Have distinct rating board committees evaluate the same set of films independently (Inter-rater reliability).
- Assessing Validity: Measure whether films assigned an R rating objectively contain higher frequencies of adult content, profanity, or violence compared to PG films (Construct validity).
- Music Critic Reviews:
- Assessing Reliability: Compare scoring consistency across different professional critics reviewing the same album (Inter-rater reliability).
- Assessing Validity: Correlate critic numerical ratings with objective musicological measures or standardized audio metrics (Criterion validity).
- Diagnostic Test for Flu Virus Presence:
- Assessing Reliability: Run duplicate biological samples from the same patient through the diagnostic machinery repeatedly (Test-retest / Alternate forms reliability).
- Assessing Validity: Compare test detection rates against laboratory viral culture standards to verify true positive and true negative rates (Sensitivity and Specificity / Construct validity).