A

PY2501 How good is your scale 2024 - Tagged

Course Overview

  • Course Title: PY2501 Research Methods and Data Analysis

  • Title Slide: How Good is Your Scale and What Does It Mean? by Andrew Schofield

  • Presentation by: Prof Adrian Burgess

Lecture Schedule

  • Week 3: Psychometrics 1: Theory of Measurement

  • Week 8: Psychometrics 2: From Items to Scales

  • Week 9: Psychometrics 3: How Good is Your Scale and What Does It Mean?

    • Lecture B: Item Analysis – Refining Scales to Improve Reliability

Key Topics in Psychometrics 3

  1. Reliability

  2. Validity

  3. Biases and Systematic Errors

  4. Interpretation of Scores

Reliability and Validity

Definitions

  • Reliability: The reproducibility of a measurement (precision of measurement).

  • Validity: The accuracy of measurement (measures what it claims to measure).

Accuracy vs. Precision

  • Accurate: Valid measurement.

  • Inaccurate: Systematic error (not valid).

  • Precise: Reliable measurement.

  • Imprecise: Reproducibility error (unreliable).

Understanding Reliability

  • Defined as the consistency of scores obtained by the same persons under different conditions.

  • Reliability Formula:

    • Reliability = True Variance / (True Variance + Error Variance)

True Variance vs. Error Variance

  • True Variance: Variability due to real individual differences.

  • Error Variance: Variability due to measurement errors or inconsistent items.

Types of Reliability

  1. Internal Consistency Reliability: Items should measure the same latent variable.

  2. Test-Retest Reliability: Consistency across measurements over time.

  3. Inter-Rater Reliability: Consistency between different raters' assessments.

Measuring Internal Consistency

  • Item-Total Correlation: Correlate scores of individual items with total scale score.

  • Split-Half Reliability: Divide items, score each half, and calculate their correlation.

  • Cronbach’s Alpha: A measure of internal consistency with acceptable thresholds:

    • α ≥ 0.9: Excellent

    • 0.9 > α ≥ 0.8: Good

    • 0.8 > α ≥ 0.7: Acceptable

Assessing Reliability

  • Reliability is sample-dependent; different samples may show different reliability estimates.

  • Published scales often quote reliability estimates.

Examining Validity

  • Validity Definition: Concerns what the test measures and its effectiveness.

  • Types of Validity:

    1. Content Validity: Does the scale encompass relevant items?

    2. Criterion Validity: Correlation with other established measures (concurrent & predictive).

    3. Construct Validity: Does it measure the theoretical construct?

The Hospital Anxiety & Depression Scale (HADS)

  • A clinical tool designed to detect anxiety and depression.

  • Reliability: Anxiety r=0.93; Depression r=0.90.

  • Construct validity established through factor analysis relating to clinical diagnoses.

Error Types in Measurement

  • Random Errors: Assumed to cancel out, lead to variability.

  • Systematic Errors: Accumulate and introduce bias into measurements.

Common Systematic Biases

  • Random/inattentive responding, Yea saying.

  • Social desirability biases: Individuals presenting themselves more favorably.

Detecting and Avoiding Bias

  • Strategies include using reverse items, lie scales, and normative scoring to reduce biases.

Understanding Scores

  • Scores are summative of individual item responses but need careful consideration for reverse-coded items.

  • Psychological scales are mostly interval-level, hence comparing against normative data is critical.

Criterion-Based Measures

  • Comparing scores against established 'Gold Standards' is vital for identifying clinical cases.

Evaluating Criterion-Based Measures

  • Sensitivity: Proportion of true cases correctly identified.

  • Specificity: Proportion of true non-cases correctly identified.

Final Insights

  • Psychological scales need diligent design and validation to ensure they effectively measure latent variables while remaining free of systematic biases.