validity issues in experiments

Validity Issues in Experiments

• Results are truthful (content-based) and trustworthy (process-based).

• Validity issues happen before, during, and after the experiment.

Truthfulness

• Do not fabricate data.

• What is important is that the data presented is what was actually gathered by all means.

• Present your data accurately and precisely.

Trustworthiness

• Work within your area of competence and know your expertise or specialty areas.

• Follow your procedures while remaining flexible in conducting your experiment.

• Demonstrate honesty and prioritize ethical experimental practices.

Threats to Validity

• Any factor that raises doubts about the experiment results or interpretation of results.

• Questions on the quality of the research process or the accuracy of the research results.

Construct Validity

• Quality of choices about the specific forms that the exogenous and endogenous variables will assume in the experiments (Myers, 2006).

  • Choice of treatment condition/s

  • Choice of outcome measure/s

  • Administration of the treatment

• What variables/instruments were used and if they were appropriately used.

Threat to Construct Validity For the Exogenous Variable

• Inadequate operationalization of the exogenous variable.

  • Non-standardized implementation of an exogenous variable to participants (especially in different groups).

  • Failure to represent the theoretical construct.

  • Exogenous variables that are non-impactful and not based on reconnaissance.

Treatment Artifacts

• Arise in the actual presentation of treatment conditions to participants.

• Result of participant and experimenter motivations and actions.

• Emanate within the experimental environments.

Examples of Treatment Artifacts

• Demand Characteristics in Experimental Setting

• Experimenter Expectancy Effects

Demand Characteristics in Experimental Setting

• Participants respond to unstated demands from the experimental environment, thereby making them good subjects.

  • E.g., class observations 

Experimenter Expectancy Effects

• Experimenters provide unconscious cues to participants about the expected results from them.

  • E.g., leading questions

Guinea Pig Effect

Participants alter their behavior, responses, or performance because they know they are part of an experiment.

Examples of the Guinea Pig Effect

• Hawthorne Effect

• Henry Effect

• Rosenthal/Pygmalion Effect

• Placebo Effect

Hawthorne Effect

• Workers showed increased productivity when they knew they were being observed.

  • E.g., Athletes during recruitment season

Henry Effect

• A worker outperformed a machine under an experimental setting because he was aware that he was being observed.

Rosenthal/Pygmalion Effect

• How teacher expectations influence children's performance.

  • ↑ expectations = ↑ performance

Placebo Effect

 Belief that something unusual is taking place due to an artificial stimulus.

Practice Effects

Pre-test sensitization to the treatment can make participants more familiar with the construct under study.

• Keyword: Pre-test; No exposure yet to the treatment.

Order Effects

Also called carryover effects, they occur in repeated-measures studies where participants may already recognize the construct due to repeated exposure.

• Keyword: Post-test; Multiple exposures

Threat to Construct Validity For the Endogenous Variable

• Inadequate operationalization of the endogenous variable.

  • Measure is invalid (e.g., lacking content validity) and unreliable (measurement variation is high, compromising its usefulness).

Measurement Artifacts

• Participants may develop response strategies in the measurement procedures (practice effects).

• Linguistic or cultural biases/barriers in the measurement tools employed.

  • Are the participants able to comprehend the material and express their responses accurately?

Criteria in Judging the Quality of Experiments

• Internal Validity

• External Validity

Internal Validity

• Experimental, investigative, or psychological realism.

  • The exogenous variables cause the experimental results.

External Validity

• Mundane realism.

  • Experimental results can be generalized to other groups.

Common Interdisciplinary Applications

• Efficacy Studies

• Effectiveness Studies

Efficacy Studies

• Address issues related to internal validity (e.g., clinical trials).

• Ex.: A vaccine that performed well in clinical trials may not work well in real-life situations.

Effectiveness Studies

• Address issues related to external validity (e.g., real-life settings).

• Ex.: People who undergo training interventions do not perform equally well in their actual jobs.

Campbell and Stanley (1963)

• Internal validity is indispensable and essential; external validity is never completely answerable.

  • Internal validity is more important because it proves the study’s point.

Cronbach (1982)

• A treatment is expected to be relevant to a broader context; the causal inference must go beyond the specific conditions.

• If the study lacks generalizability, the internally valid causal effect is useless to decision-makers, so the focus should be on external validity.

Derek Briggs (2008)

• Construct validity and external validity are still necessary for generalizing a causal conclusion to other settings.

  • Everything should be well-integrated.

Internal Validity

• Exists if the results or findings genuinely emanate from the study’s procedures.

• Extent to which the investigation can control different biases that affect the study.

• Whether the findings or results of the experiment are accurate.

Threats to Internal Validity

• History Effects

• Maturation or Fatigue Effects

• Assignment/Selection Bias

• Group Composition Effects

• Mortality

• Statistical Regression Effects

History Effects

• Environmental Factor Effects

  • External conditions that affect the study’s results (e.g., time, ventilation, width of physical space, etc.)

  • Observed when researchers fail to observe standardized procedures in conducting the study.

  • Differences in the respondents’ situations can confound the results.

Maturation or Fatigue Effects

• Prior participation in a condition or procedure tires participants, reducing performance on subsequent steps.

  • If the participant gets tired.

Assignment/Selection Bias

• Researchers did not follow proper sampling procedures and deliberately manipulated sample selection to achieve a particular set of results.

  • Convenience over appropriate sampling.

Group Composition Effects

• Uses different groups to compare the treatment’s effects.

  • Pre-existing differences among groups can obscure the exogenous variable’s impact on the endogenous variable.

Mortality

• Sample attrition; Respondents do not finish the research or withdraw without completing the entire research procedures.

  • More troublesome in experiments where considerations are highly essential.

  • A participant left before the experiment was finished.  If a participant withdraws, the experiment is no longer accurate.

Statistical Regression Effects

• Regression toward the mean.

  • Tendency of extreme (very high or very low) scores to fall closer to the mean on retesting.

  • When the results are too far off, it becomes a threat; they should be close to the mean.

External Validity

• Extent to which the investigation’s results can be generalized to the entire population or other groups.

• Generalized stability; can it be replicated or falsified?; is it understood by other groups?

Threats to External Validity

• Cross-species Generalization

• Non-representative Sampling

• Non-representative Research Context

• Multiple Activity Interference

• Sensitization

Cross-species Generalization

• Cultural factors vary and are not constant across all people.

  • Results of certain experiments may not apply or be acceptable to other contexts despite careful control and management of variables in the study.

  • Ex.: Sikolohiyang Pilipino would not be appropriate to understand psychology in India.

Non-representative Sampling

• Considers the representation of the group as supported by sample size, sampling design, and techniques used.

  • Determination of a good scope as evidenced by narrowing down the coverage.

  • Results from haphazard sampling.

  • Ex.: You look for a long-term employee, but they’ve only been there for 3 years.

Non-representative Research Context:

• Certain conditions may not be experienced by other groups; therefore, the study may not apply to them.

  • Experimenters conduct studies for the sake of doing while not considering reconnaissance.

  • E.g., Bahala Na implies proceeding with an action regardless of the outcome (‘whatever happens, happens’). Foreign audiences might misinterpret it as indifference or a lack of accountability. Failing to operationalize the term creates a threat to the study.

Multiple Activity Interference

• Difficulty generalizing due to participant-related factors.

  • Artificial results may be observed if participants experience fatigue or are sensitized by the treatment.

Sensitization

• Repeated administration of a procedure results in progressive amplification of the response.

  • If you do it repetitively, it won’t work anymore.

Statistical Validity

• Concerns the proper statistical treatment of data and the soundness of the experimenters’ statistical conclusions.

• Goes hand-in-hand with practical significance.

• Deals with the accuracy of statistical treatment.

Threats for Statistical Validity

• Inappropriate use of statistical techniques

  • Maling paggamit ng statistical technique.

• Wrong use of statistical treatment

  • Mali ‘yong mismong ginamit na statistical treatment.

• Committing errors in the interpretation of statistical results.