CHAPTER IV RELIABILITY

CHAPTER IV: RELIABILITY

CONTENT

• Outline

  • Defining Reliability

  • Models of Reliability

    • Test-Retest

    • Alternate / Parallel Forms

    • Internal Consistency

    • Interscorer / Interrater

      Reliability: This refers to the degree to which different assessors provide consistent results when measuring the same phenomenon.

Defining Reliability

• Everyday Definition:

  • Synonym for dependability or consistency.

  • Example: A reliable train schedule.

  • Important in personal relationships.

• Psychometric Definition:

  • Refers to consistency in measurement.

  • Reliability doesn’t imply quality, only consistency.

• Emphasis: "Consistency is Everything."

Importance of Reliability

• Essential for users of tests to understand reliability to make informed decisions.

• Reliability is contextual; a test may be reliable in one situation and not in another.

• Different Types and Degrees of Reliability are recognized.

Reliability Coefficient

• Defined as an index of reliability, indicating the ratio between the true score variance and total variance.

• Federal Guidelines: Tests must be reliable before being used for employment or educational decisions.

• Formula:

  • r = σ²(T) / σ²(0)

  • Where:

    • r = theoretical reliability of the test

    • σ²(T) = variance of true scores

    • σ²(0) = variance of observed scores

Implications of Reliability Coefficient

• A test with a reliability coefficient of .40 means 40% of score variation can be explained by actual differences, while 60% is random error.

Sources of Error

• Observed Score vs. True Score:

  • Variations may be caused by situational factors such as noise, temperature, or test items not representing the desired domain.

Test Construction Errors

• Errors in item sampling can lead to different test experiences, impacting results.

• Random factors may influence test performance, e.g., hope for certain questions to appear.

Test Administration Errors

• Environmental factors may distract or demotivate test-takers, e.g., room conditions or noise.

• Test-taking conditions, such as discomfort or emotional factors can alter performance.

Examiner-Related Errors

• Variability based on examiners’ conduct or their interpretations can introduce error variance.

• The professionalism of examiners plays a crucial role in reliability.

Reliability Estimation Methods

• Test-Retest Method: Evaluates consistency over time.

• Parallel Forms Method: Assesses performance across different but equivalent test forms.

• Internal Consistency Method: Examines consistency within subsets of items on a test.

Test-Retest Method

• Most applicable for stable traits, e.g., intelligence.

• Concerns include carryover effects (memory of answers) affecting reliability if tests are spaced too closely.

Parallel Forms Method

• Compares different forms of a test measuring the same attribute using the Pearson correlation coefficient.

• Less frequently used due to development complexities.

Internal Consistency Methods

• Split-Half Method: Divides a test in halves to assess reliability.

• Spearman-Brown formula corrects split-half estimates due to half-test length.

  • Formula:

• Coefficient Alpha: General reliability coefficient, especially when items are not dichotomous.

Factors Affecting Reliability

• Reliability relies on the coherence of items measuring a similar trait; working towards unidimensionality is ideal.

Interrater Reliability

• Assessing reliability between different observers evaluating the same behavior.

• Percent agreement is commonly calculated to gauge consistency.

How Reliable is Reliable?

• Suggested reliability ranging from .70 to .80 is generally sufficient for research.

• Extreme reliability (above .90) may indicate duplicative item content.

• High reliability is critical in clinical settings for safeguarding patient outcomes.

Addressing Low Reliability

• Increase Item Count: More items generally lead to higher reliability.

• Prophecy Formula: Estimates how many items are necessary for desired reliability levels.

Correction for Attenuation

• Measurement errors can diminish the perceived correlation between tests. The formula allows estimation of true correlation if no errors existed.

  • Formula:

Conclusion

• Utilize concepts of reliability and sources of error to improve psychological measurement accuracy.