PY2501 How good is your scale 2024 - Tagged

Course Overview

• Course Title: PY2501 Research Methods and Data Analysis

• Title Slide: How Good is Your Scale and What Does It Mean? by Andrew Schofield

• Presentation by: Prof Adrian Burgess

Lecture Schedule

• Week 3: Psychometrics 1: Theory of Measurement

• Week 8: Psychometrics 2: From Items to Scales

• Week 9: Psychometrics 3: How Good is Your Scale and What Does It Mean?

  • Lecture B: Item Analysis – Refining Scales to Improve Reliability

Key Topics in Psychometrics 3

1. Reliability

2. Validity

3. Biases and Systematic Errors

4. Interpretation of Scores

Reliability and Validity

Definitions

• Reliability: The reproducibility of a measurement (precision of measurement).

• Validity: The accuracy of measurement (measures what it claims to measure).

Accuracy vs. Precision

• Accurate: Valid measurement.

• Inaccurate: Systematic error (not valid).

• Precise: Reliable measurement.

• Imprecise: Reproducibility error (unreliable).

Understanding Reliability

• Defined as the consistency of scores obtained by the same persons under different conditions.

• Reliability Formula:

  • Reliability = True Variance / (True Variance + Error Variance)

True Variance vs. Error Variance

• True Variance: Variability due to real individual differences.

• Error Variance: Variability due to measurement errors or inconsistent items.

Types of Reliability

1. Internal Consistency Reliability: Items should measure the same latent variable.

2. Test-Retest Reliability: Consistency across measurements over time.

3. Inter-Rater Reliability: Consistency between different raters' assessments.

Measuring Internal Consistency

• Item-Total Correlation: Correlate scores of individual items with total scale score.

• Split-Half Reliability: Divide items, score each half, and calculate their correlation.

• Cronbach’s Alpha: A measure of internal consistency with acceptable thresholds:

  • α ≥ 0.9: Excellent

  • 0.9 > α ≥ 0.8: Good

  • 0.8 > α ≥ 0.7: Acceptable

Assessing Reliability

• Reliability is sample-dependent; different samples may show different reliability estimates.

• Published scales often quote reliability estimates.

Examining Validity

• Validity Definition: Concerns what the test measures and its effectiveness.

• Types of Validity:

  1. Content Validity: Does the scale encompass relevant items?

  2. Criterion Validity: Correlation with other established measures (concurrent & predictive).

  3. Construct Validity: Does it measure the theoretical construct?

The Hospital Anxiety & Depression Scale (HADS)

• A clinical tool designed to detect anxiety and depression.

• Reliability: Anxiety r=0.93; Depression r=0.90.

• Construct validity established through factor analysis relating to clinical diagnoses.

Error Types in Measurement

• Random Errors: Assumed to cancel out, lead to variability.

• Systematic Errors: Accumulate and introduce bias into measurements.

Common Systematic Biases

• Random/inattentive responding, Yea saying.

• Social desirability biases: Individuals presenting themselves more favorably.

Detecting and Avoiding Bias

• Strategies include using reverse items, lie scales, and normative scoring to reduce biases.

Understanding Scores

• Scores are summative of individual item responses but need careful consideration for reverse-coded items.

• Psychological scales are mostly interval-level, hence comparing against normative data is critical.

Criterion-Based Measures

• Comparing scores against established 'Gold Standards' is vital for identifying clinical cases.

Evaluating Criterion-Based Measures

• Sensitivity: Proportion of true cases correctly identified.

• Specificity: Proportion of true non-cases correctly identified.

Final Insights

• Psychological scales need diligent design and validation to ensure they effectively measure latent variables while remaining free of systematic biases.