Notes on Classical Test Theory and Assessment Integrity

Classical Test Theory (CTT)

  • Definition: CTT posits that psychometric measurements include a true estimate of ability or behavioral preference along with measurement error.
  • Objective: Enhance the true score and minimize error in assessments.

Sources of Error in Assessment

A) Test-related Error

  1. Norm Sampling

    • Issue: Poor representation in norm sample; too small or un-stratified.
    • Estimation: Standard Error of the Mean (SEmean) and Confidence Interval.
    • Solution: Use tests with a representative sample.

  2. Test Reliability

    • Issue: Low reliability in tests; reflected through Test Re-test, Internal Consistency, and Inter-Item correlation.
    • Estimation: Standard Error of Measurement (SEm).
    • Solution: Select reliable tests with coefficients above +0.70.
B) Candidate-related Error

  1. Candidate Compliance

    • Issue: Non-compliance with test administration.
    • Solution: Motivate and monitor candidates.

  2. Candidate Wellness

    • Issue: Impact of physical or psychological state during assessment.
    • Solution: Brief candidates on the importance of being well and focused for assessments.

  3. Conscious Distortion

    • Issue: Intentional distortion of responses (positive or negative).
    • Solution: Encourage accurate and honest responses through motivation.

  4. Unconscious Distortion

    • Issue: Inadvertently providing untrue representation of behaviors.
    • Solution: Validate scores with examples and pre-screen for self-insight.
C) Administrator Error in Test Administration, Scoring, Feedback, and Application

  1. Test Completion Requirements

    • Issue: Unsuitable tests due to factors like language or cognitive abilities.
    • Solution: Pre-screen candidates with sample tests for informed consent.

  2. Test Difficulty

    • Issue: Tests that are either too difficult or too easy for candidates.
    • Solution: Match test level to competency requirements; conduct a test run.

  3. Test Conditions

    • Issue: Inadequate environment (lighting, quietness, etc.).
    • Solution: Ensure test conditions are suitable and comfortable.

  4. Test Administration

    • Issue: Poor briefing leading to lack of understanding and motivation.
    • Solution: Deliver a comprehensive briefing pre-testing and on the day.

  5. Test Scoring

    • Issue: Errors in hand scoring.
    • Solution: Use correct scoring methods and validate norm tables.

  6. Feedback and De-briefing

    • Issues: Resistance, Accquiescence, and Impression Management.
    • Solutions:
      • Resistance (R): Verify test integrity; hold candidates accountable for results.
      • Acquiescence (A) & Impression Management (IM): Debrief scores and gather specific behavior examples.

  7. Application of Results

    • Selection Issue: Misinterpretation of profile scores without linking to competencies.
    • Solution: Thoroughly debrief roles and competencies during interviews.
    • Coaching Issue: Failure to address challenging behaviors; misalignment in coaching context.
    • Solution: Explore behavior patterns in conjunction with coaching goals to foster improvement.