Prelude to Research Validity Flashcards

Introduction to Research Validity

  • Research Hypothesis (RH): A research hypothesis is defined as a guess about the relationships between behaviors.
  • Workflow for Testing a Research Hypothesis:
    • Select and decide on a research design.
    • Sample participants from the population of interest.
    • Collect empirical data from the sampled participants.
    • Statistically analyze the collected data.
    • Formulate a final conclusion regarding whether or not the empirical results support the research hypothesis.
  • Core Definition of Validity:
    • Validity encompasses numerous specific types, definitions, and methodological procedures, but it fundamentally refers to accuracy or correctness.
    • The ultimate goal of the research process is to ensure that the final conclusion drawn is valid.

The Four Major Types of Research Validity

  • External Validity:
    • Evaluates the extent to which research results can be accurately generalized to other participants, situations, and times.
  • Internal Validity:
    • Evaluates whether it is correct to give a causal interpretation to the relationship found between the variables or behaviors.
  • Measurement Validity:
    • Evaluates whether the operationalized variables and collected data accurately represent the specific behaviors intended to be studied.
  • Statistical Conclusion Validity:
    • Evaluates whether the statistical analysis produced the correct conclusion regarding whether or not there is a relationship between the variables or behaviors being studied.

External Validity

  • Key Methodological Decisions Influencing External Validity:
    • Participants: Selection of the specific individuals from whom data are collected.
    • Stimuli and Tasks: Selection of the materials, interventions, and activities with which participants interact.
    • Setting: Selection of the physical and contextual environment in which data collection occurs.
  • Applicability and Generalizability:
    • After a study is completed, analyzed, and reported (whether presented at an academic conference or published in a journal or textbook), researchers and practitioners must evaluate its applicability.
    • External validity assesses applicability by determining whether the target participants, stimuli/tasks, and settings of a prospective application are sufficiently close to those used in the original research design.

Internal Validity

  • Evaluation of Causal Claims:
    • Testing a causal research hypothesis requires ruling out confounds (alternative hypotheses regarding what causes the observed outcome).
    • Internal validity assesses whether the experimental design effectively eliminates these competing alternative hypotheses.
  • Concrete Example of Internal Validity and Confounding:
    • Experimental Setup: A researcher provides computer-based math practice to a sample of 4th graders and paper-based math practice to a sample of 2nd graders.
    • Observed Finding: The 4th graders receiving computer-based practice score higher on a math test than the 2nd graders receiving paper-based practice.
    • Causal Claim: The researcher asserts that computer-based math practice directly causes superior learning.
    • Alternative Hypothesis (Confound): 4th graders are inherently more advanced in math than 2nd graders and would predictably score higher on a math test regardless of the practice medium used.
    • Conclusion: The age/grade level confound invalidates the causal claim, demonstrating low internal validity.

Measurement Validity

  • Scientific Empiricism and Data Collection:
    • Scientific empiricism requires concrete data serving as measures of behavior.
    • Primary procedures for collecting behavioral data:
    • Observation: Systematically watching and recording participant behavior.
      • Example 1: Recording the number of aggressive behaviors exhibited by a child during school recess.
      • Example 2: Recording the number of items a participant successfully remembers from a list of 40 nouns.
    • Self-Report: Directly asking participants to report on their own behaviors, beliefs, or opinions.
      • Example 1: Asking participants, "Did you vote in the most recent ASUN election?"
      • Example 2: Asking participants, "How often do you try to make a new friend?"
  • Core Focus:
    • Measurement validity focuses directly on the accuracy and correctness of these measurement procedures and the data they generate.

Statistical Conclusion Validity

  • Data Analysis in Scientific Empiricism:
    • Scientific empiricism demands data-based evidence, requiring rigorous analysis of collected data.
    • Testing both associative and causal research hypotheses requires determining whether a statistical relationship exists between behaviors or variables.
  • Core Evaluative Questions:
    • Was the chosen data analysis procedure appropriate for the specific type of data collected and the research hypothesis being tested?
    • Was the decision regarding whether a relationship exists between the variables mathematically and statistically accurate?

Research Validity Practice Scenarios and Solutions

  • Scenario Set 1:

    • Scenario 1: "I don’t know that I should apply your results from college students to my teaching of 4th graders."
    • Validity Type: External Validity
    • Explanation: Evaluates whether findings from a specific sample (college students) generalize to a different population and setting (4th graders).
    • Scenario 2: "Are you sure that it is the cognitive-behavior treatment that decreased the participants' depression and not just that these folks had someone who would listen to their problems?"
    • Validity Type: Internal Validity
    • Explanation: Challenges a causal claim by raising an alternative hypothesis/confound (having an attentive listener).
    • Scenario 3: "Was the 'Beck Depression Inventory' a good choice for this study?"
    • Validity Type: Measurement Validity
    • Explanation: Evaluates whether a specific measurement tool (Beck Depression Inventory) accurately measures the intended construct (depression).
    • Scenario 4: "Are you sure there really is a relationship between practice and performance, or is this finding a 'fluke'?"
    • Validity Type: Statistical Conclusion Validity
    • Explanation: Questions the correctness of the statistical decision regarding whether a true relationship exists between variables.

  • Scenario Set 2:

    • Scenario 1: "I’m not sure it was practice that improved performance, since the group that had the practice also had more previous experience."
    • Validity Type: Internal Validity
    • Explanation: Identifies a confounding variable (previous experience) that provides an alternative explanation for the performance difference.
    • Scenario 2: "If cognitive-behavior treatment works, as your study found, why have the other 20 studies all found that it didn’t work?"
    • Validity Type: Statistical Conclusion Validity
    • Explanation: Questions the accuracy of the statistical outcome by comparing it against a large body of contradictory statistical findings.
    • Scenario 3: "I’m sure that your method works great in the laboratory, but what about here in the 'real world'?"
    • Validity Type: External Validity
    • Explanation: Evaluates whether findings obtained in a specific setting (a laboratory) generalize to practical real-world contexts.
    • Scenario 4: "Is how many problems they tried really a good index of success? I’d rather look at how many they got correct."
    • Validity Type: Measurement Validity
    • Explanation: Questions whether the operationalized measure (number of problems attempted) accurately reflects the target variable (success).