Validity in Research Designs Part 2: Threats to Validity
Research Validity in Study Designs
This section delves into research validity, building upon an understanding of different validity types and their subtypes. It focuses on how various factors, both controllable and uncontrollable, can compromise internal and/or external validity. These compromising factors are termed "threats." We will also explore methods to mitigate or counteract these threats through modifications in research design. This section introduces many research-specific terms.
Classification of Threats to Research Validity
Threats to research validity are categorized based on their primary source, as outlined in your textbook. They fall into three broad categories:
Related to Participants or Subjects: These threats stem from participant characteristics, their distribution within study groups, or their attitudes and behaviors.
Related to Investigators: Threats originating from the research team's actions or inactions.
Related to Study Logistics: Threats arising from external factors or the operational aspects of the study.
Threats Related to Participants or Subjects
Selection / Poor Sampling
Selection is a critical first step in research. If the participants chosen for a study do not adequately represent the target population, the study's results cannot be reliably applied to that population, including potential patients. This issue primarily compromises external validity (generalizability).
To understand this fully, it is crucial to clarify several terms:
Population: This refers to a large group of individuals to whom the study results are intended to be applicable. For example, if a study evaluates a stroke treatment in young adults aged , then all young adults with stroke in that specific age range constitute the population.
Sampling Frame: This is a subset of the population from which participants are effectively available for enrollment in your study. It is created based on defined inclusion criteria (qualifications) and exclusion criteria (disqualifications).
Sample: These are the individuals ultimately selected from the sampling frame to participate in the study.
Random Selection (Probabilistic or Random Selection): This is a crucial method for selecting participants from the sampling frame. It ensures that every individual within the sampling frame has an equal opportunity or probability of being included in the sample. This process minimizes selection bias, increases the likelihood that the sample accurately mirrors the characteristics of the intended population, and reduces the risk of extraneous differences influencing the study's representativeness.
Poor sampling significantly compromises external validity because the study's findings may not be generalizable to the broader population. To avoid this threat:
Conduct a multi-site study: This helps ensure a larger and more diverse sampling frame, leading to better representation.
Utilize random selection: This method ensures the sample's characteristics are similar to those of the population, reducing bias and improving generalizability.
Baseline Equivalence / Differential Group Assignment
When a study involves multiple groups (e.g., treatment vs. control), after ensuring the sample is representative of the population, it's essential to distribute participant characteristics uniformly among these groups. This process is called randomization or random assignment.
Randomization (Random Assignment): This ensures that participants have an equal probability of being assigned to any given group. Crucially, it also aims to control for certain baseline characteristics that could impact study results, ensuring an even distribution of these characteristics across groups. For example, in a study assessing strength gains across different age groups, it is vital to balance the distribution of men and women between the groups.
If characteristics are unevenly distributed at baseline, the groups are not homogeneous. This constitutes a threat to internal validity. If groups start at different levels, it becomes difficult to definitively conclude that the treatment alone caused any observed effects. For instance, if a control group initially exhibits lower symmetry, it's unclear whether subsequent improvements in the treatment group are solely due to the intervention or if the control group's lower starting point inherently limits its potential for improvement.
Attrition (Mortality / Dropout)
Attrition refers to the loss of study participants over the course of the study, a particular concern in longitudinal designs. This can introduce bias if participants are lost non-randomly.
Solutions:
Intention-to-treat (ITT) analysis: This statistical approach analyzes data based on the initial treatment assignment, regardless of whether participants completed the study or switched treatments. It provides a more conservative estimate of treatment effect.
Statistical comparison with those lost: Analyze baseline characteristics of participants who dropped out versus those who completed the study to assess potential bias.
Replacing lost participants: While possible, this can be cost and time-prohibitive and may not fully address the bias if the reasons for dropout are systematic.
Maturation
Maturation refers to natural changes that occur in participants as time progresses. The common phrase