Validity and Threats to Validity
Validity
Threats to Validity
Variability
Variability is a fundamental issue in research, concerning the amount of difference observable among people, situations, or things. It is more than standard deviation.
Variability is almost always present; the key considerations are:
The source of the variability.
How the variability is accounted for.
Unsystematic Variability: Essentially random variance, which is generally considered undesirable ("BAD").
Sources include individual differences within the sample/population and measurement error.
Systematic Variability: Variance that is accounted for among groups.
Caused by the independent variable (IV) or treatment manipulation.
Extraneous Variance: Undesirable variance resulting from differences among groups for reasons outside of the controlled intervention; this is also known as a confounding variable.
Threats to Validity
Internal Validity
Internal validity refers to whether observed changes among people or differences in the population are due to the designed intervention or variables of interest.
Strong internal validity implies that the observed changes are indeed due to the intervention.
A threat to internal validity exists if changes observed in the study (or differences observed among groups) are due to factors outside the planned investigation.
These threats arise when an irrelevant variable "messes up" the study findings, indicating poor control.
External Validity
External validity concerns whether the findings of the study are generalizable to the population of interest.
Population Validity: Indicates whether the findings apply to people outside of the study.
Ecological Validity: Indicates whether the findings can be applied to conditions outside of the study.
Threats to Internal Validity: DIFFERENCES BETWEEN GROUPS
Maturation: Changes occur due to the typical development of the sample.
Selection: Bias or error in the selection of subjects accounts for the findings.
Selection X Maturation: The effect of maturation differs based on selection.
Statistical Regression (Regression to the Mean): When selecting extreme groups, anticipate that they will naturally regress towards the mean.
Mortality: Withdrawal or loss of participants skews the results.
Instrumentation: Measurement issues underlie the outcomes.
Testing: Involvement with dependent variable (DV) measures influences outcomes (e.g., the testing effect).
History: Something happened during the study that accounts for the findings.
Resentful Demoralization of Control Group: The control group becomes demoralized and performs worse due to feeling left out.
Threats to Internal Validity: SIMILARITIES AMONG GROUPS
Diffusion of Treatment: "Seepage effect," where the treatment affects the control group.
Compensatory Rivalry: "John Henry effect," where the control group works harder to compete with the treatment group.
Compensatory Equalization of Treatment: "Clinician's dilemma," where efforts are made to equalize treatment between groups, compromising the integrity of the intervention.
Threats to External Validity: POPULATION VALIDITY
Generalization Across Subjects (Participants): There are limits to the ability to expect that the study's outcome will be applicable to the general population (sampling frame).
Interaction of "Personological" Variables and Treatment Effects: An unaccounted-for within-sample factor in the experimental group causes a differential outcome, affecting the ability to generalize to the population at large.
Threats to External Validity
Verification of IV: Also known as Treatment Fidelity.
Multiple Treatment Interference: The effects of multiple treatments overlap (e.g., common in reading research).
Hawthorne Effect: Participants alter their behavior because they know they are being observed.
Novelty & Disruption Effects: Similar to the Hawthorne effect, where newness or disruption influences outcomes.
Experimenter Effects: Common in treatment interventions delivered by an expert (the creator of the intervention).
Pretest Sensitization: Critical when only the experimental group has the pretest, influencing their reactions to the treatment.
Posttest Sensitization: Only an issue when the posttest itself accounts for the outcome of the findings.
History X Treatment: Something else happened during the study that affected the treatment outcome.
Measurement of DV: "Straw Man," where the measurement of the dependent variable does not accurately reflect the intended construct.
Interaction of Time of Measurement & Treatment Effect: The intervention is only effective within a limited parameter of the assessment period in the study.
Other Validity Concerns
Statistical Validity
Studies using statistics meet the expectation of statistical validity when predetermined levels of "confidence" are met.
Attention to the appropriate use of statistics is crucial (avoid "fishing" for results).
Social Validity
The study provides use to the society involved.
Goals of study are socially relevant.
Study outputs were worth the “cost”.
Findings demonstrate a meaningful outcome for society (statistical significance does not mean practical significance).
Validity
Threats to Validity
Variability
Variability is a fundamental issue in research, concerning the amount of difference observable among people, situations, or things. It is more than standard deviation.
Variability is almost always present; the key considerations are:
The source of the variability.
How the variability is accounted for.
Unsystematic Variability: Essentially random variance, which is generally considered undesirable ("BAD").
Sources include individual differences within the sample/population and measurement error.
Systematic Variability: Variance that is accounted for among groups.
Caused by the independent variable (IV) or treatment manipulation.
Extraneous Variance: Undesirable variance resulting from differences among groups for reasons outside of the controlled intervention; this is also known as a confounding variable.
Threats to Validity
Internal Validity
Internal validity refers to whether observed changes among people or differences in the population are due to the designed intervention or variables of interest.
Strong internal validity implies that the observed changes are indeed due to the intervention.
A threat to internal validity exists if changes observed in the study (or differences observed among groups) are due to factors outside the planned investigation.
These threats arise when an irrelevant variable "messes up" the study findings, indicating poor control.
External Validity
External validity concerns whether the findings of the study are generalizable to the population of interest.
Population Validity: Indicates whether the findings apply to people outside of the study.
Ecological Validity: Indicates whether the findings can be applied to conditions outside of the study.
Threats to Internal Validity: DIFFERENCES BETWEEN GROUPS
Maturation: Changes occur due to the typical development of the sample.
Selection: Bias or error in the selection of subjects accounts for the findings.
Selection X Maturation: The effect of maturation differs based on selection.
Statistical Regression (Regression to the Mean): When selecting extreme groups, anticipate that they will naturally regress towards the mean.
Mortality: Withdrawal or loss of participants skews the results.
Instrumentation: Measurement issues underlie the outcomes.
Testing: Involvement with dependent variable (DV) measures influences outcomes (e.g., the testing effect).
History: Something happened during the study that accounts for the findings.
Resentful Demoralization of Control Group: The control group becomes demoralized and performs worse due to feeling left out.
Threats to Internal Validity: SIMILARITIES AMONG GROUPS
Diffusion of Treatment: "Seepage effect," where the treatment affects the control group.
Compensatory Rivalry: "John Henry effect," where the control group works harder to compete with the treatment group.
Compensatory Equalization of Treatment: "Clinician's dilemma," where efforts are made to equalize treatment between groups, compromising the integrity of the intervention.
Threats to External Validity: POPULATION VALIDITY
Generalization Across Subjects (Participants): There are limits to the ability to expect that the study's outcome will be applicable to the general population (sampling frame).
Interaction of "Personological" Variables and Treatment Effects: An unaccounted-for within-sample factor in the experimental group causes a differential outcome, affecting the ability to generalize to the population at large.
Threats to External Validity
Verification of IV: Also known as Treatment Fidelity.
Multiple Treatment Interference: The effects of multiple treatments overlap (e.g., common in reading research).
Hawthorne Effect: Participants alter their behavior because they know they are being observed.
Novelty & Disruption Effects: Similar to the Hawthorne effect, where newness or disruption influences outcomes.
Experimenter Effects: Common in treatment interventions delivered by an expert (the creator of the intervention).
Pretest Sensitization: Critical when only the experimental group has the pretest, influencing their reactions to the treatment.
Posttest Sensitization: Only an issue when the posttest itself accounts for the outcome of the findings.
History X Treatment: Something else happened during the study that affected the treatment outcome.
Measurement of DV: "Straw Man," where the measurement of the dependent variable does not accurately reflect the intended construct.
Interaction of Time of Measurement & Treatment Effect: The intervention is only effective within a limited parameter of the assessment period in the study.
Other Validity Concerns
Statistical Validity
Studies using statistics meet the expectation of statistical validity when predetermined levels of "confidence" are met.
Attention to the appropriate use of statistics is crucial (avoid "fishing" for results).
Social Validity
The study provides use to the society involved.
Goals of study are socially relevant.
Study outputs were worth the “cost”.
Findings demonstrate a meaningful outcome for society (statistical significance does not mean practical significance).
Validity
Threats to Validity
Variability
Variability is a fundamental issue in research, concerning the amount of difference observable among people, situations, or things. It is more than standard deviation.
Variability is almost always present; the key considerations are:
The source of the variability.
How the variability is accounted for.
Unsystematic Variability: Essentially random variance, which is generally considered undesirable ("BAD").
Sources include individual differences within the sample/population and measurement error.
Systematic Variability: Variance that is accounted for among groups.
Caused by the independent variable (IV) or treatment manipulation.
Extraneous Variance: Undesirable variance resulting from differences among groups for reasons outside of the controlled intervention; this is also known as a confounding variable.
Threats to Validity
Internal Validity
Internal validity refers to whether observed changes among people or differences in the population are due to the designed intervention or variables of interest.
Strong internal validity implies that the observed changes are indeed due to the intervention.
A threat to internal validity exists if changes observed in the study (or differences observed among groups) are due to factors outside the planned investigation.
These threats arise when an irrelevant variable "messes up" the study findings, indicating poor control.
External Validity
External validity concerns whether the findings of the study are generalizable to the population of interest.
Population Validity: Indicates whether the findings apply to people outside of the study.
Ecological Validity: Indicates whether the findings can be applied to conditions outside of the study.
Threats to Internal Validity: DIFFERENCES BETWEEN GROUPS
Maturation: Changes occur due to the typical development of the sample.
Selection: Bias or error in the selection of subjects accounts for the findings.
Selection X Maturation: The effect of maturation differs based on selection.
Statistical Regression (Regression to the Mean): When selecting extreme groups, anticipate that they will naturally regress towards the mean.
Mortality: Withdrawal or loss of participants skews the results.
Instrumentation: Measurement issues underlie the outcomes.
Testing: Involvement with dependent variable (DV) measures influences outcomes (e.g., the testing effect).
History: Something happened during the study that accounts for the findings.
Resentful Demoralization of Control Group: The control group becomes demoralized and performs worse due to feeling left out.
Threats to Internal Validity: SIMILARITIES AMONG GROUPS
Diffusion of Treatment: "Seepage effect," where the treatment affects the control group.
Compensatory Rivalry: "John Henry effect," where the control group works harder to compete with the treatment group.
Compensatory Equalization of Treatment: "Clinician's dilemma," where efforts are made to equalize treatment between groups, compromising the integrity of the intervention.
Threats to External Validity: POPULATION VALIDITY
Generalization Across Subjects (Participants): There are limits to the ability to expect that the study's outcome will be applicable to the general population (sampling frame).
Interaction of "Personological" Variables and Treatment Effects: An unaccounted-for within-sample factor in the experimental group causes a differential outcome, affecting the ability to generalize to the population at large.
Threats to External Validity
Verification of IV: Also known as Treatment Fidelity.
Multiple Treatment Interference: The effects of multiple treatments overlap (e.g., common in reading research).
Hawthorne Effect: Participants alter their behavior because they know they are being observed.
Novelty & Disruption Effects: Similar to the Hawthorne effect, where newness or disruption influences outcomes.
Experimenter Effects: Common in treatment interventions delivered by an expert (the creator of the intervention).
Pretest Sensitization: Critical when only the experimental group has the pretest, influencing their reactions to the treatment.
Posttest Sensitization: Only an issue when the posttest itself accounts for the outcome of the findings.
History X Treatment: Something else happened during the study that affected the treatment outcome.
Measurement of DV: "Straw Man," where the measurement of the dependent variable does not accurately reflect the intended construct.
Interaction of Time of Measurement & Treatment Effect: The intervention is only effective within a limited parameter of the assessment period in the study.
Other Validity Concerns
Statistical Validity
Studies using statistics meet the expectation of statistical validity when predetermined levels of "confidence" are met.
Attention to the appropriate use of statistics is crucial (avoid "fishing" for results).
Social Validity
The study provides use to the society involved.
Goals of study are socially relevant.
Study outputs were worth the “cost”.
Findings demonstrate a meaningful outcome for society (statistical significance does not mean practical significance).
Validity
Threats to Validity
Variability
Variability is a fundamental issue in research, concerning the amount of difference observable among people, situations, or things. It is more than standard deviation.
Variability is almost always present; the key considerations are:
The source of the variability.
How the variability is accounted for.
Unsystematic Variability: Essentially random variance, which is generally considered undesirable ("BAD").
Sources include individual differences within the sample/population and measurement error.
Systematic Variability: Variance that is accounted for among groups.
Caused by the independent variable (IV) or treatment manipulation.
Extraneous Variance: Undesirable variance resulting from differences among groups for reasons outside of the controlled intervention; this is also known as a confounding variable.
Threats to Validity
Internal Validity
Internal validity refers to whether observed changes among people or differences in the population are due to the designed intervention or variables of interest.
Strong internal validity implies that the observed changes are indeed due to the intervention.
A threat to internal validity exists if changes observed in the study (or differences observed among groups) are due to factors outside the planned investigation.
These threats arise when an irrelevant variable "messes up" the study findings, indicating poor control.
External Validity
External validity concerns whether the findings of the study are generalizable to the population of interest.
Population Validity: Indicates whether the findings apply to people outside of the study.
Ecological Validity: Indicates whether the findings can be applied to conditions outside of the study.
Threats to Internal Validity: DIFFERENCES BETWEEN GROUPS
Maturation: Changes occur due to the typical development of the sample.
Selection: Bias or error in the selection of subjects accounts for the findings.
Selection X Maturation: The effect of maturation differs based on selection.
Statistical Regression (Regression to the Mean): When selecting extreme groups, anticipate that they will naturally regress towards the mean.
Mortality: Withdrawal or loss of participants skews the results.
Instrumentation: Measurement issues underlie the outcomes.
Testing: Involvement with dependent variable (DV) measures influences outcomes (e.g., the testing effect).
History: Something happened during the study that accounts for the findings.
Resentful Demoralization of Control Group: The control group becomes demoralized and performs worse due to feeling left out.
Threats to Internal Validity: SIMILARITIES AMONG GROUPS
Diffusion of Treatment: "Seepage effect," where the treatment affects the control group.
Compensatory Rivalry: "John Henry effect," where the control group works harder to compete with the treatment group.
Compensatory Equalization of Treatment: "Clinician's dilemma," where efforts are made to equalize treatment between groups, compromising the integrity of the intervention.
Threats to External Validity: POPULATION VALIDITY
Generalization Across Subjects (Participants): There are limits to the ability to expect that the study's outcome will be applicable to the general population (sampling frame).
Interaction of "Personological" Variables and Treatment Effects: An unaccounted-for within-sample factor in the experimental group causes a differential outcome, affecting the ability to generalize to the population at large.
Threats to External Validity
Verification of IV: Also known as Treatment Fidelity.
Multiple Treatment Interference: The effects of multiple treatments overlap (e.g., common in reading research).
Hawthorne Effect: Participants alter their behavior because they know they are being observed.
Novelty & Disruption Effects: Similar to the Hawthorne effect, where newness or disruption influences outcomes.
Experimenter Effects: Common in treatment interventions delivered by an expert (the creator of the intervention).
Pretest Sensitization: Critical when only the experimental group has the pretest, influencing their reactions to the treatment.
Posttest Sensitization: Only an issue when the posttest itself accounts for the outcome of the findings.
History X Treatment: Something else happened during the study that affected the treatment outcome.
Measurement of DV: "Straw Man," where the measurement of the dependent variable does not accurately reflect the intended construct.
Interaction of Time of Measurement & Treatment Effect: The intervention is only effective within a limited parameter of the assessment period in the study.
Other Validity Concerns
Statistical Validity
Studies using statistics meet the expectation of statistical validity when predetermined levels of "confidence" are met.
Attention to the appropriate use of statistics is crucial (avoid "fishing" for results).
Social Validity
The study provides use to the society involved.
Goals of study are socially relevant.
Study outputs were worth the “cost”.
Findings demonstrate a meaningful outcome for society (statistical significance does not mean practical significance).
Validity
Threats to Validity
Variability
Variability is a fundamental issue in research, concerning the amount of difference observable among people, situations, or things. It is more than standard deviation.
Variability is almost always present; the key considerations are:
The source of the variability.
How the variability is accounted for.
Unsystematic Variability: Essentially random variance, which is generally considered undesirable ("BAD").
Sources include individual differences within the sample/population and measurement error.
Systematic Variability: Variance that is accounted for among groups.
Caused by the independent variable (IV) or treatment manipulation.
Extraneous Variance: Undesirable variance resulting from differences among groups for reasons outside of the controlled intervention; this is also known as a confounding variable.
Threats to Validity
Internal Validity
Internal validity refers to whether observed changes among people or differences in the population are due to the designed intervention or variables of interest.
Strong internal validity implies that the observed changes are indeed due to the intervention.
A threat to internal validity exists if changes observed in the study (or differences observed among groups) are due to factors outside the planned investigation.
These threats arise when an irrelevant variable "messes up" the study findings, indicating poor control.
External Validity
External validity concerns whether the findings of the study are generalizable to the population of interest.
Population Validity: Indicates whether the findings apply to people outside of the study.
Ecological Validity: Indicates whether the findings can be applied to conditions outside of the study.
Threats to Internal Validity: DIFFERENCES BETWEEN GROUPS
Maturation: Changes occur due to the typical development of the sample.
Selection: Bias or error in the selection of subjects accounts for the findings.
Selection X Maturation: The effect of maturation differs based on selection.
Statistical Regression (Regression to the Mean): When selecting extreme groups, anticipate that they will naturally regress towards the mean.
Mortality: Withdrawal or loss of participants skews the results.
Instrumentation: Measurement issues underlie the outcomes.
Testing: Involvement with dependent variable (DV) measures influences outcomes (e.g., the testing effect).
History: Something happened during the study that accounts for the findings.
Resentful Demoralization of Control Group: The control group becomes demoralized and performs worse due to feeling left out.
Threats to Internal Validity: SIMILARITIES AMONG GROUPS
Diffusion of Treatment: "Seepage effect," where the treatment affects the control group.
Compensatory Rivalry: "John Henry effect," where the control group works harder to compete with the treatment group.
Compensatory Equalization of Treatment: "Clinician's dilemma," where efforts are made to equalize treatment between groups, compromising the integrity of the intervention.
Threats to External Validity: POPULATION VALIDITY
Generalization Across Subjects (Participants): There are limits to the ability to expect that the study's outcome will be applicable to the general population (sampling frame).
Interaction of "Personological" Variables and Treatment Effects: An unaccounted-for within-sample factor in the experimental group causes a differential outcome, affecting the ability to generalize to the population at large.
Threats to External Validity
Verification of IV: Also known as Treatment Fidelity.
Multiple Treatment Interference: The effects of multiple treatments overlap (e.g., common in reading research).
Hawthorne Effect: Participants alter their behavior because they know they are being observed.
Novelty & Disruption Effects: Similar to the Hawthorne effect, where newness or disruption influences outcomes.
Experimenter Effects: Common in treatment interventions delivered by an expert (the creator of the intervention).
Pretest Sensitization: Critical when only the experimental group has the pretest, influencing their reactions to the treatment.
Posttest Sensitization: Only an issue when the posttest itself accounts for the outcome of the findings.
History X Treatment: Something else happened during the study that affected the treatment outcome.
Measurement of DV: "Straw Man," where the measurement of the dependent variable does not accurately reflect the intended construct.
Interaction of Time of Measurement & Treatment Effect: The intervention is only effective within a limited parameter of the assessment period in the study.
Other Validity Concerns
Statistical Validity
Studies using statistics meet the expectation of statistical validity when predetermined levels of "confidence" are met.
Attention to the appropriate use of statistics is crucial (avoid "fishing" for results).
Social Validity
The study provides use to the society involved.
Goals of study are socially relevant.
Study outputs were worth the “cost”.
Findings demonstrate a meaningful outcome for society (statistical significance does not mean practical significance).
Validity
Threats to Validity
Variability
Variability is a fundamental issue in research, concerning the amount of difference observable among people, situations, or things. It is more than standard deviation.
Variability is almost always present; the key considerations are:
The source of the variability.
How the variability is accounted for.
Unsystematic Variability: Essentially random variance, which is generally considered undesirable ("BAD").
Sources include individual differences within the sample/population and measurement error.
Systematic Variability: Variance that is accounted for among groups.
Caused by the independent variable (IV) or treatment manipulation.
Extraneous Variance: Undesirable variance resulting from differences among groups for reasons outside of the controlled intervention; this is also known as a confounding variable.
Threats to Validity
Internal Validity
Internal validity refers to whether observed changes among people or differences in the population are due to the designed intervention or variables of interest.
Strong internal validity implies that the observed changes are indeed due to the intervention.
A threat to internal validity exists if changes observed in the study (or differences observed among groups) are due to factors outside the planned investigation.
These threats arise when an irrelevant variable "messes up" the study findings, indicating poor control.
External Validity
External validity concerns whether the findings of the study are generalizable to the population of interest.
Population Validity: Indicates whether the findings apply to people outside of the study.
Ecological Validity: Indicates whether the findings can be applied to conditions outside of the study.
Threats to Internal Validity: DIFFERENCES BETWEEN GROUPS
Maturation: Changes occur due to the typical development of the sample.
Selection: Bias or error in the selection of subjects accounts for the findings.
Selection X Maturation: The effect of maturation differs based on selection.
Statistical Regression (Regression to the Mean): When selecting extreme groups, anticipate that they will naturally regress towards the mean.
Mortality: Withdrawal or loss of participants skews the results.
Instrumentation: Measurement issues underlie the outcomes.
Testing: Involvement with dependent variable (DV) measures influences outcomes (e.g., the testing effect).
History: Something happened during the study that accounts for the findings.
Resentful Demoralization of Control Group: The control group becomes demoralized and performs worse due to feeling left out.
Threats to Internal Validity: SIMILARITIES AMONG GROUPS
Diffusion of Treatment: "Seepage effect," where the treatment affects the control group.
Compensatory Rivalry: "John Henry effect," where the control group works harder to compete with the treatment group.
Compensatory Equalization of Treatment: "Clinician's dilemma," where efforts are made to equalize treatment between groups, compromising the integrity of the intervention.
Threats to External Validity: POPULATION VALIDITY
Generalization Across Subjects (Participants): There are limits to the ability to expect that the study's outcome will be applicable to the general population (sampling frame).
Interaction of "Personological" Variables and Treatment Effects: An unaccounted-for within-sample factor in the experimental group causes a differential outcome, affecting the ability to generalize to the population at large.
Threats to External Validity
Verification of IV: Also known as Treatment Fidelity.
Multiple Treatment Interference: The effects of multiple treatments overlap (e.g., common in reading research).
Hawthorne Effect: Participants alter their behavior because they know they are being observed.
Novelty & Disruption Effects: Similar to the Hawthorne effect, where newness or disruption influences outcomes.
Experimenter Effects: Common in treatment interventions delivered by an expert (the creator of the intervention).
Pretest Sensitization: Critical when only the experimental group has the pretest, influencing their reactions to the treatment.
Posttest Sensitization: Only an issue when the posttest itself accounts for the outcome of the findings.
History X Treatment: Something else happened during the study that affected the treatment outcome.
Measurement of DV: "Straw Man," where the measurement of the dependent variable does not accurately reflect the intended construct.
Interaction of Time of Measurement & Treatment Effect: The intervention is only effective within a limited parameter of the assessment period in the study.
Other Validity Concerns
Statistical Validity
Studies using statistics meet the expectation of statistical validity when predetermined levels of "confidence" are met.
Attention to the appropriate use of statistics is crucial (avoid "fishing" for results).
Social Validity
The study provides use to the society involved.
Goals of study are socially relevant.
Study outputs were worth the “cost”.
Findings demonstrate a meaningful outcome for society (statistical significance does not mean practical significance).