Reliability and Validity

Introduction

• Personal update: Instructor mentions having been sick, leading to class schedule adjustments.
• Acknowledgment of being behind in the course material due to this illness.

Curriculum Adjustments

• Discussion of curriculum changes:
  • Removal of one week's topics: Probability and Z-scores.
  • Reasoning: Z-scores not needed for writing assignments, but basic understanding of probability is essential.
  • Key concept in probability for psychology is the p-value:
  • Definition: The p-value indicates the statistical significance of results, where any p-value < 0.05 suggests that results are statistically supported and unlikely to be due to chance.

Reliability and Validity Overview

• Today's focus:
  • Concepts of reliability and validity.
  • Preparation for the next writing assignment related to reliability.

Psychological Constructs

• Definition of a construct:
  • Summarized as what is being measured in psychology (e.g., anxiety defined as excessive worry in absence of stressors).
• Importance of conceptualization and operationalization:
  • Conceptualization: The definition of the construct.
  • Operationalization: The method used to measure the construct.
  • Potential discrepancies between theoretical definitions and practical measurements can occur.
• Illustration of reliability and validity:
  • Ensures accurate measurement of constructs.

Psychological Models and Constructs

• Template for a psychological model:
  • Items that contribute to a construct (e.g., personality traits measured in assessments).
  • Larger measures (e.g., BFI or IPIP) contain multiple items to capture one variable (e.g., extroversion).
• Example of constructs and their predictive capability:
  • Conscientiousness as a predictor of GPA, GRE scores, and job performance.
  • Hypothesized relationships:
  • Positive association between conscientiousness and GPA, GRE scores.
  • Possible negative association with actual job performance.
    • Implication: High academic performance does not necessarily correlate with job efficacy.

Key Definitions

• Reliability:
  • Defined as the consistency of test scores across different testing occasions, versions of the test, or by different raters.
  • Types of reliability:
  • Test-retest reliability: Similar scores when the same test is given at different times.
    • Example: Pretest and posttest in class.
  • Parallel forms reliability: Different versions of the same test yield similar scores.
  • Internal consistency: Consistency of scores across items within a test that measures the same construct.
  • Inter-rater reliability: Consistency between different raters evaluating the same participant's behavior.
• Validity:
  • The degree to which a test measures what it is intended to measure.
  • Types of validity:
  • Content-related validity: Coverage of domain of the construct.
  • Criterion-related validity: Correlation with outcome variables.
  • Construct-related validity: Relationship with other similar and dissimilar constructs.

Examples of Reliability and Validity

• Reliability Example:
  • Measurement could be consistently incorrect (e.g., scales that read 5 pounds heavy).
• Validity Example:
  • High reliability with low validity if a measure predicts unrelated outcomes, such as the weight measure predicting anxiety levels.

Diagrams and Visuals for Understanding Reliability and Validity

• Bull's-eye model of validity and reliability:
  • Illustrate hitting the target of measurement with various outcomes:
  • Accurate and reliable measurement (hits the center).
  • Reliable but inaccurate (hits consistently but not at the target).
  • Valid but inconsistent (hits the target on average but lacks consistency).

Philosophical Implications of Reliability and Validity

• Importance of reliability and validity before conducting any analyses:
  • If constructs aren't reliably measured, inferred conclusions based on these measures may be invalid or misleading.
  • Example: Reliably measuring weight as a proxy for anxiety leads to inaccurate conclusions regarding anxiety levels.

Cronbach's Alpha • Reliability Assessment

• Internal consistency is assessed using Cronbach's alpha:
  • Symbol: α (alpha symbol).
  • Desired range for reliability:
  • Acceptable: α > 0.70.
  • Strong: 0.80 - 0.89.
  • Very strong: α ≥ 0.90.
• Potential issues in calculating Cronbach's alpha include:
  • Incorrect data leading to scores exceeding 1;
  • Short measures yielding surprisingly high alphas.

Example Data Analysis in Jamovi

• Overview of statistical software used for data analysis (Jamovi).
• Steps to compute reliability:
  • Inputting items and reverse coding when necessary.
  • Procedures to check outputs for reliability and internal consistency measures.

Writing Assignments Related to Reliability and Validity

• Structure of upcoming assignments and their relation to reliability and validity concepts discussed in class:
  • Preparation of papers includes assessment of correlation, mean scores, and considering reliability coefficients in interpretations.
  • Need for proper exemplification of reliable but invalid measures.
• Instructions for incorporating outputs and statistics in assignments (e.g., graphs, results).
• APA formatting considerations for reporting results (e.g., spacing with symbols, etc.).

Testing Hypotheses

• Interpretation of results based on reliability and validity assessments:
  • Discussion on behaviors that were observed and measured.
  • Evaluating correlation coefficients between constructs and their implications for construct validity.

Conclusion

• Summary of core topics and procedures.
• Instructions for next week: peer review and writing workshop.
• Closing remarks due to backtracking on previously covered material.

Appendix

• Example reverse-coded items and related details:
  • Reverse coding must be consistent with directions of scoring.
• Recommendations for future explorations of construct validity and correlations among constructs.