Research Methods: Hypothesis Testing, Validity, and Threats to Validity

Research Methods: A Process of Inquiry

Chapter 8: Hypothesis Testing, Validity, and Threats to Validity

Learning Objectives

  • Develop hypotheses from problem statements.

  • Describe the types of validity in quantitative research.

  • Differentiate among the major confounding variables that can occur in research.

  • Describe subject and experimenter effects that threaten the validity of a study.

  • Summarize the ethical issues associated with controlling confounding variables.

Capstone Project

  • Generate a problem statement regarding the relationship between two (or more) variables

    • Examples:

    • Social media and stress

    • Personality traits

  • Tips on how to generate a problem statement:

    • Consider aspects of human behavior that interest you.

    • Generate a list of variables that could be related.

    • Examples of pairs:

    • Time spent driving and level of impatience with other drivers

    • Nutrition and academic achievement

    • Surveys are quick and easy to administer!

    • Search through the literature for ideas, or consider how other researchers have explored similar ideas.

Testing Hypotheses

  • Researchers test specific hypotheses which are generated from the initial research idea through a series of steps.

    • A research idea can lead to numerous research hypotheses depending on:

    • How well the problem is translated into a question.

    • The variables are operationally defined.

The Initial Idea

  • The starting point of research is often vague or general.

  • Refinement of the initial idea occurs through:

    • A search of relevant research literature

    • Initial observations of the phenomenon.

  • Researchers formalize the initial idea into a clear statement of the problem.

Statement of the Problem

  • The problem is structured as a question indicating the expected relationship.

  • The nature of the question determines the required level of constraint in the study:

    • Causal questions necessitate experimental research.

    • Questions about relationships can be addressed with lower-constraint research.

Generating Research Hypotheses

  • Ideas lead to:

    • Observations

    • Library research

    • A well-formed statement of the problem

  • Problem statements become research hypotheses when constructs are operationalized.

Operational Definitions

  • Discussed in Chapter 3; they describe the procedures used to measure and/or manipulate variables.

  • Most variables can be operationally defined in multiple ways.

  • Examples of operational definitions:

    • Relaxation: An internal state observable through behavior.

    • Disruptive behavior: Measured through frequency, duration, and intensity.

    • Specific examples include:

    • Do male hormones increase aggressive behavior in rats?

    • Does the presence of a stranger in the room increase an infant’s crying?

    • Do stimulants help hyperactive children control their behavior?

    • Will mandatory arrest and jail time reduce spousal abuse?

Research Hypotheses

  • State the expected relationship between variables in a clear, declarative statement.

  • This statement serves as a tentative hypothesis for research testing.

  • Variables in research hypotheses should be stated in operational definition terms.

The Role of Theory

  • Theory informs all research planning and is often the primary source of research hypotheses.

  • Theory guides:

    • Selection of variables.

    • Operational definitions of variables.

  • Most research is based on multiple, overlapping, and interacting theories.

Testing Research Hypotheses

  • Testing involves three types of hypotheses:

    • Null Hypothesis: The initial hypothesis that there is no effect.

    • Confounding Variable Hypothesis: Examines potential confounding variables' influence.

    • Causal Hypothesis: Suggests a causal relationship between independent and dependent variables.

  • Accept the causal hypothesis only if:

    • The null hypothesis is rejected (statistical analysis).

    • All potential confounding variable hypotheses are ruled out via appropriate controls.

Evaluating Hypotheses

  • Decision-making process:

    • Check if the null hypothesis is true.

    • Determine if potential confounding variables are controlled.

    • A causal hypothesis is supported when:

    • The null hypothesis is not true.

    • Confounding variables are adequately controlled.

Types of Validity

  • Statistical Validity: Accuracy of statistical tests; threatened by unreliable measures and assumptions violations.

    • Strengthened by using well-validated measures and equal sample sizes.

  • Construct Validity: Evaluates if the theory is the best explanation for results; threatened by alternative explanations.

    • Strengthened through well-validated constructs and rigorous design elements.

    • Example: Males scored higher on math assessments than females—nature or nurture?

  • External Validity: Determines if results apply to broader populations; threatened by unrepresentative samples.

    • Strengthened by gathering representative samples and clearly describing the sample type.

    • Limits of generalizability predict behavior under specific conditions.

    • Ecological Validity: When studies accurately reproduce real-life situations.

  • Internal Validity: Tests if the independent variable causes observed changes in the dependent variable; threatened by confounding variables.

    • Strengthened through adding adequate controls.

Avoiding Confounding

  • Confounding can violate internal validity.

  • Several sources for confounding will be covered; proper controls can eliminate confusion.

  • Consider alternative theoretical explanations for phenomena to support construct validity.

Confounding Variables

  • Maturation: Changes due to growth or predictable changes.

  • History: Changes resulting from events occurring during a study.

  • Testing: Performance changes due to previous testing.

  • Instrumentation: Changes in the calibration of the instrument used in the study over time.

  • Regression to the Mean: Tendency for extreme scorers to become less extreme over time.

  • Selection: Group imbalance at the study's start; often due to lack of random sampling.

  • Attrition: Participants dropping out; individuals may differ in characteristics.

  • Diffusion of Treatment: Changes due to information shared between groups.

  • Sequence Effects: Impact of the order of conditions on performance.

Subject Effects

  • Responses by participants that don't reflect their normal behavior.

  • Participants try to interpret the study, potentially responding to cues (demand characteristics).

  • Placebo Effect: Treatment effect arising from expectations regarding treatment efficacy.

Experimenter Effects

  • Researcher behaviors that can influence participant behavior or dependent variable measurement.

  • Often linked to the researcher’s expectations which can be inadvertently transmitted to participants.

  • Not identical to scientific fraud, which is a deliberate misrepresentation.

Validity, Control, and Constraint

  • These concepts are closely tied:

    • Validity: The accuracy of the study or procedure.

    • Increased through appropriate controls: higher control levels lead to higher constraint in findings.

Ethical Principles

  • The risk of participation should be balanced with the reward.

  • A poorly designed study offers no useful information; thus, any risk may be deemed unacceptable.

  • Informed Consent: Although ensuring informed consent may introduce selection bias due to participant refusal, it is essential for maintaining ethical standards.

Summary

  • Start with a robust research hypothesis which is essentially three hypotheses (null, confounding-variable, causal).

  • Recognize various types of validity and confounding variables.

  • Acknowledge the effects of subject and experimenter manipulations on study outcomes.

Group Activity

  • Provide at least one example for each of the following confounds:

    • Maturation

    • History

    • Testing

    • Instrumentation

    • Regression to the mean

    • Selection

    • Attrition

    • Diffusion of treatment

    • Selection effects

Supplemental Slides

Website Resources

  • 08:01 Library Research

  • 08:02 Functional Statistical Flowchart

  • 08:03 Identifying Confounding Variables Exercises

  • 08:04 Study Guide/Lab Manual

  • 08:05 Related Internet Sites

Pretest-Posttest Design

  • A single-group, pretest-posttest design compares pre-treatment and post-treatment scores to identify improvements, but lacks control over most confounding sources.

Two-Group, Posttest-Only Design

  • A more robust method comparing different treatments on outcome measures.

    • Example:

    • Autistic children given relaxation drugs vs. those who are not; measures levels of disruptive behavior in each group.

Copyright Information

  • This work is protected by United States copyright laws and is provided solely for instructional use.

  • Dissemination or sale of this content is prohibited, and materials should not be made available to students except by instructors utilizing the corresponding text in classes.

  • All recipients must abide by these restrictions to honor the integrity of the work and support the pedagogical needs of other educators.