PSYC314 Lecture Notes - Validity, Test Bias and Fairness in Test Development

Overview of Validity, Test Bias, and Test Fairness

• Continued examination of validity focusing on test bias and test fairness.
• Importance of understanding test construction and test development stages.

Culture and Behavior

• Cultural background significantly influences:
  • Beliefs and attitudes toward health and illness.
  • Interpretation and recognition of symptoms.
  • Help-seeking behaviors and acceptance of treatment.
  • Compliance with treatment.
• Study by Shahin, Kennedy, & Stupans identified cultural factors that affect medication adherence in patients with chronic illnesses.
  • Factors include perception of illness, health literacy, cultural beliefs, and self-efficacy.

Impact of Culture on Psychological Testing

• Cultural effects cannot be entirely removed from psychological test performance, affecting test development:
  • Operational definitions and theories guide test structure.
  • Standardization and norming depend on the participant sample used.
  • Attempts to create culture-fair tests still utilize culturally specific frameworks.

Test Bias and Test Fairness Controversy

• Intelligence tests often show ethnic discrepancies, with significant average score differences.
  • Environmental vs. biological explanations for score disparities.
• Common criticisms:
  • Intelligence tests misnamed; should be called cultural background tests.
  • Test scores reflect test characteristics rather than test-taker ability.

Definitions

• Test Bias: A statistical concept assessing the differential validity of test scores among subgroups.
• Test Fairness: A social values concept focused on ethical implications and selection philosophies.

Technical Meaning of Test Bias

• Cole and Moss (1989) define bias in terms of differences in score implications for different subgroups.
• Differential Validity: Assessment of how different groups may interpret scores differently; illustrated by receptive vocabulary tests across language backgrounds.

Establishing Test Bias

• Contexts in which bias can be established:
  • Content validity: Items not equally accessible to minority groups.
  • Criterion-related validity: Tests predicting future performance differently across subgroups.
  • Construct validity: Demonstrating that a test measures constructs with varying accuracy across groups.

Approaches to Test Fairness

• Involves ethical philosophies:
  • Unqualified Individualism: Selection based solely on test abilities.
  • Qualified Individualism: Taking into account demographic factors for selection.
  • Quotas: Different selection processes for various community subgroups.

Ensuring Absence of Test Bias

• Strategies to eliminate bias:
  • Involvement of minority representatives in test development.
  • Multiple evaluations by expert panels.
  • Routine analysis of items to prevent group differences.
  • Appropriate accommodations for individuals with disabilities.

Test Development Process

Major Steps in Test Development

1. Test Conceptualisation: Define the construct to be measured.
2. Test Construction: Create the actual test items.
3. Test Tryout: Administer the items to a sample.
4. Item Analysis: Evaluate item effectiveness and reliability.
5. Test Revision: Modify the test based on analysis results.

Test Conceptualisation

• Initial steps include:
  • Clear statement of purpose identifying the traits or constructs to assess.
  • Literature review to check for existing relevant tests or justification for a new test.

Item Construction Details

• Anatomy of a test item includes:
  • Stimulus: The prompt or question posed to test-takers.
  • Response Format: Type of response expected from test-takers (e.g., multiple choice, constructed response).
  • Conditions Governing Response: Guidelines for test-takers on how to respond.
  • Scoring Procedures: Defined methods for how responses will be scored.

Types of Test Items

• Selected-response items: Require selecting from options (e.g., multiple-choice).
• Constructed-response items: Require creating an answer (e.g., essays).
• Importance of considering scoring challenges with constructed-response items due to variability.

Qualitative Evaluation of Test Items

• Review for:
  • Conformity to rules for writing items.
  • Content relevance and appropriateness.
  • Sensitivity to potential biases (gender, racial).

Test Tryout and Analysis

• Trial of test items in representative conditions to refine item usability.
• Gathering participant feedback for further refinements.

Specific Example: Pregnancy-related Anxiety Scale (PrAS)

• Development of an anxiety scale through structured item pool refinement, psychometric assessments, and expert evaluations.
• Empirical data supporting validity and reliability addressed.

References

• Important studies and works referenced throughout the lecture for further reading and validation of the discussed topics.
• Ensure to consult these studies for deeper insight into test biases, cultural impacts, and practical methodologies in test development.