Applied Psychology - 2017 - Kung - Are Attention Check Questions a Threat to Scale Validity

Attention Checks: Popular in survey research to filter out careless respondents.
Research Need: Limited empirical testing on how attention checks affect scale validity.
Hypothesis: Attention checks may induce a more deliberative mindset, changing how respondents answer and posing a threat to scale validity.
Studies: Two studies conducted with 816 participants to test the effect of common attention checks on a management scale.
Findings: No evidence that attention checks affect scale validity, in both reported scale means and measurement invariance tests.

Self-Report Measurement: Vital for survey research influencing organizational decisions.
Reality vs. Ideal: While ideally respondents pay attention to scale items, evidence suggests 5-60% respond carelessly.
Consequences: Careless responses can threaten measurement validity, leading to misleading conclusions.
Attention Checks as a Solution: Recommendations include embedding attention checks to identify and exclude careless respondents.

Instructed-Response Items: Items with obvious correct responses embedded in the scale.
- Examples: “please select four for this item”.
- Effectiveness: Successfully screen out inattentive participants but may not account for various interpretations of wrong answers.
Instructional Manipulation Checks (IMC): More elaborate checks that require careful reading to identify the correct response.
- Example: A task that requires understanding long instructions with a cue in the last sentence.
- Utility: More effort required may increase effectiveness in identifying careless respondents, but may also seem trickier to participants.

Deliberative Mindset: Attention checks can trigger a deliberate response style which may alter subsequent judgments and decisions.
Response Variance: Introducing variance unrelated to the target construct threatens validity and may lead to biased findings in decision-making.
Evidence from Research: Studies have indicated attention checks cause respondents to deliberate more, leading to differences in responses.

Study 1: Investigated the influence of instructed-response items on responses to the Organizational Citizenship Behavior (OCB) scale.
Participants: Recruited from Amazon MTurk, ensuring demographic relevance.
Results: No significant differences in OCB scores between participants with and without attention checks. Measurement invariance tests showed similar understanding of scale items across conditions.
Study 2: Examined the effect of IMC items on the same OCB scale.
Findings: Consistent results with Study 1, where the IMC showed no significant alteration to respondents' answers or understanding of the scale.

Null Effect of Attention Checks: Contrary to expectations, attention checks did not significantly influence scale responses or validity.
Implications for Future Research: The results encourage the use of attention checks for quality data without compromising validity.
Considerations for Use: Attention checks may act as warnings to respondents, potentially increasing the quality of responses by deterring careless answering.

Scope of Research: Limited to one scale; results may not generalize to other scales. Future research should explore varying scale characteristics and samples.
Participant Experience: Most samples from MTurk have survey-taking experience; implications of this familiarity need further examination.

Attention checks are vital for maintaining data quality but do not inherently threaten scale validity according to these studies.
Future research should continue examining the relationship between survey characteristics and response dynamics to enhance survey methodology.