source 8.2

Overview of the Study

• Title: Theory-Driven Game-Based Assessment of General Cognitive Ability: Design Theory, Measurement, Prediction of Performance, and Test Fairness

• Authors: Richard N. Landers, Michael B. Armstrong, Andrew Burnett Collmus, Salih Mujcic, Jason Blaik

• Publication: Journal of Applied Psychology, October 2021

• DOI: 10.1037/apl0000954

• Citations: 53

• Reads: 7,433

Purpose of the Research

• Investigate game-based assessments (GBA) for evaluating cognitive abilities in high-stakes testing contexts.

• Develop design theories and psychometric properties associated with GBA.

• Explore issues related to measurement fairness and prediction of performance.

Key Concepts

Game-Based Assessment (GBA)

• Defined as incorporating game elements to measure specific constructs, allowing for engagement and interactive assessment.

• Rely on theory-driven design versus generic game elements to ensure construct validity.

• Focused on measuring general cognitive ability (g).

Cognitive Ability (g)

• General mental ability that is a strong predictor of job performance and educational success.

• Operationally defined by CHC (Cattell-Horn-Carroll) theory, which organizes cognitive abilities hierarchically.

Methodology

Sample Description

• Academic Sample: 633 university students.

• Organizational Sample: 49 employees rated on job performance.

Measure Development

• Created an early version of Cognify, featuring seven mini-games targeting different aspects of g.

• Employed rigorous psychometric testing to ensure the game's effectiveness in measuring cognitive ability.

Measurement and Validation

Empirical Approach

• Convergence testing between GBA performance and traditional g testing results.

• Included both confirmatory and exploratory factor analyses to validate measurement constructs.

Results

• Confirmed high correlation between latent game performance and traditional measures of g (β = .97).

• Both GPA and supervisor ratings correlated with GBA performance (GPA: r = .16; Job Performance: r = .29).

Test Fairness and Adverse Impact

Definition of Adverse Impact

• Occurs when test scores significantly differ across different demographic groups.

• Addressed both race and gender differences in performance outcomes within the GBA and traditional assessments.

Findings

• Evidence of adverse impact between racial groups; the same patterns were observed in both GBA and traditional tests.

• Gender differences were significant, yet fair assessment predicted outcomes equitably across genders.

Reactions to GBA

Key Outcomes

• Test-taker perceptions of GBA were more favorable compared to traditional cognitive tests, emphasizing the importance of applicant experience and engagement.

• This relationship suggests that more enjoyable assessments may lead to better organizational outcomes.

Practical Implications

• Recommendations for integrating game design principles into assessment methodologies.

• Highlight the need for organizations to consider the costs versus benefits of adopting GBAs amidst their traditional testing counterparts.

Conclusions and Future Research Directions

• GBA's potential for measuring cognitive abilities reliably and engagingly offers a new approach in personnel selection and assessment.

• Future studies should deepen understanding of GBA design effectiveness for various constructs and explore broader settings beyond cognitive assessments.