Design Validity (4.2)

Research Designs do what?

Create the structure for the study

• Control components of the overall study

• Assume the conclusions are accurate and true

• The results can be generalized to others

List the 3 key types of design validity

• Statistical conclusion validity

• Internal Validity

• External Validity

Statistical Conclusion validity

Appropriate use of statistical procedures to accurately infer relationships between variables. It ensures that the conclusions drawn from the data are reliable and reflect true effects.

Potential threats of Statistical Conclusion validity:

• Statistical power

• Violated assumption of tests

• Failure to use intention to treat analysis

• Fishing

Internal Validity

Evidence that the independent variable “causes” the change in dependent variable

• Assumes cause and effect

  • Considers temporal precedence (evidence comes before outcome)

  • Covariation

  • No other plausible explanations

Were changes in the dependent variable due to the normal development or the simple passage of time?

Maturation

Maturation Threat

Changes occur overtime in participants as a result of development or healing

How to protect against maturation?

• Use of control groups

• Ensure baseline equivalence through random assignment

Assignment/Selection threat:

Groups are not equal on some important characteristics

How to protect against Assignment/Selection threat:

• Random assignment

• Participant matching

• Statistical procedures such as covariance

History threat:

Events occur between the pretest and posttest

How to protect against History threat:

• Use control groups

• Ensure short time between pre and post test

• Ensure protection against exposure to alternative therapies

Regression to the mean threat:

Extreme scores change and move toward the mean with repeated testing

How to protect against Regression to the mean threat:

• Use control groups

• Exclude outliers

• Take the average of multiple measurements

Testing/Practice/Order effects:

Performance on measures changes due to exposure or some other feature of the testing experience

How to protect against Testing/Practice/Order effect threat:

• Control group

• Use measures with good test/retest reliability

• use alternate forms of measures

• Take breaks if fatigue is anticipated

Instrumentation threat:

Invalid or unreliable measures, tester error, or poor condition of the instrument result in inaccurate outcomes

How to protect against Instrumentation threat:

• Blind the tester

• Measures with good reliability and validity

• Train testers

• Maintain the instruments

List the Participant and Experimenter bias threats:

• Rosenthal/Pygmalion effect

• Compensatory equalization of treatment

• Compensatory demoralization

• Hawthorne effect

• Attrition/Mortality

Rosenthal/Pygmalion effect threat:

Intervention leaders expect participants to improve, or participants expect to improve

• Control group expects to perform worse than the other group

How to protect against Rosenthal/Pygmalion threat

• Blind intervention leaders and/or participants

• Limit contact between the control leaders and participants

• Establish and follow protocols

Compensatory equalization of treatment Effect:

Intervention leader’s behavior encourages participants in control group to improve equal to intervention group

• Or control group is motivated to “compete” with other group

How to protect against Compensatory equalization of treatment effect:

• Blind leaders/participants

• Limit contact between intervention and control leaders and participants

Compensatory demoralization effect:

Intervention leader’s behavior discourages control group

• Or control group gets discouraged b/c of their placement

How to protect against Compensatory demoralization effect:

• Blind leaders/participants

• Limit contact between intervention and control leaders and participants

Hawthorne effect:

Participants improve because of attention received from being in study

How to protect against Hawthorne effect:

• Blind intervention leaders and/or participants

• Ensure equal attention for intervention and control groups

Attrition/Mortality:

Participants who drop out affect the equalization of the group or other characteristics

How to protect against Attrition/Mortality:

• Employ strategies to encourage attendance/participation

• Use statistical estimation for missing data

• Perform intent to treat analysis

Is there bias in the way subjects have been assigned to experimental groups?

Assignment/Selection

Did unanticipated events occur during the study that could affect the dependent variable?

History

Was there a tendency for extreme scores on the pretest to regress toward the mean on the posttest?

Regression to the mean

Did the pretesting or repeated testing affect the dependent vairable

Testing/practice/order effects

Were measures invalid or unreliable orin poor condition?

Instrumentation

Did participants or researchers expectations introduce any bias

Rosenthal/Pygmalion effect

Are researchers adhering to the protocol for all groups or are participants adhering to their intervention protocol?

Compensatory equalization/rivalry

Did participants receiving less desirable interventions respond at lower levels of performance because they are resentful?

Compensatory demoralization

Are participants receiving the intervention better because they receive more attention?

Hawthorne effect

Is there a differential loss of participants across groups?

Attrition