Understanding Research Validity in Experimental Design

What is research validity?

The truthfulness of inferences made from a research study.

What are the four major types of research validity?

Statistical conclusion validity, construct validity, internal validity, and external validity.

What does statistical conclusion validity refer to?

The validity with which we can infer that the independent and dependent variables covary. Inferential statistics allow us to establish this type of validity. It answers the question: “Do independent and dependent variables covary

What is a threat to statistical conclusion validity?

Small sample size.

What is construct validity?

The extent to which we can infer higher-order constructs from the operations we use to represent them.

Give examples of constructs used in research.

Depression, obsessive-compulsive disorder, intelligence, love.

What are some examples of independent and dependent variables in research?

Independent variables: frustration, anxiety, room temperature; Dependent variables: memory, intelligence, reaction time.

What is a primary threat to construct validity?

Participants' reactivity to the experimental situation: influenced by demand characteristics, primary motive is positive self presentation, implication for research

What are demand characteristics?

Cues available in an experiment that influence the responses of participants.

What is the experimenter effect?

Actions and characteristics of researchers that influence the responses of participants. (could be intentional or unintentional)

How can experimenter motives bias a study?

by supporting the hypothesis

What are the three categories of experimenter attributes that can bias a study?

Biosocial attributes, psychosocial attributes, and situational factors.

What is internal validity?

The correctness of inferences made by researchers about cause and effect.

What are the criteria for identifying a causal relation?

1) The cause (IV) must be related to the effect (DV); 2) Changes in IV must precede changes in DV; 3) No other plausible explanation must exist for the effect.

What is a primary threat to internal validity?

Confounding extraneous variables.

Define extraneous variable.

A variable that competes with the IV in explaining the DV.

What is a confounding extraneous variable?

An extraneous variable that co-occurs with the IV and affects the DV.

Provide an example of confounding extraneous variables in a study.

If tutoring (IV) is tested on grades (DV) without random assignment, and the experimental group is an honors class, the class type may confound the results.

What is external validity?

The extent to which research findings can be generalized to settings, populations, and times beyond the study.

What conditions threaten external validity?

Factors such as sample characteristics, settings, and the specific time of the study.

Why is statistical conclusion validity important in research?

It helps determine if the independent and dependent variables are related, influencing the reliability of the study's conclusions.

How can researchers eliminate threats to internal validity?

By using random assignment, controlling extraneous variables, and ensuring proper experimental design.

What is the implication of positive self-presentation in research?

Participants may alter their responses to appear more favorable, affecting the validity of the results.

How can the influence of extraneous variables be eliminated?

By holding their influence constant and using random assignment to balance their influence.

What is a threat to internal validity related to history?

Any event that can produce the outcome, other than the treatment condition, that occurs during the study before posttest measurement.

How can history threats to internal validity be controlled?

Typically by using a comparison control group.

What is differential history?

occurs in a multi-group design when event has differential impact on groups

Give an example of differential history affecting a study on bipolar disorder treatment.

If an actress announces her bipolar disorder and promotes treatment during the study, it may influence the patients' improvement.

What was the differential history threat in Shadish and Reid's evaluation of the WIC program?

Women who received WIC assistance also received food stamps, which could confound the results.

What is maturation in the context of internal validity?

Any physical or mental change that occurs over time and affects dependent variable scores.

What factors can contribute to maturation threats?

Age, learning, fatigue, boredom, and hunger.

How can maturation threats be controlled in a study?

Typically by using a comparison control group.

What is instrumentation as a threat to internal validity?

Changes from pretest to posttest in the assessment or measurement of the dependent variable

What can cause instrumentation bias?

Human observers changing their measurements due to boredom or fatigue.

How can the validity and reliability of observations be improved?

By using multiple observers and training them on observation techniques.

What is the testing threat to internal validity?

Changes in a person's score on a second test administration due to having previously taken the test.

How can testing threats be controlled?

Typically by using a comparison control group.

Why might researchers consider eliminating the pretest in a study?

To avoid the potential influence of prior test-taking on subsequent scores.

What is regression artifact?

Effects that appear to be due to the treatment but are actually due to regression to the mean.

What does regression to the mean refer to?

The tendency for extreme scores to be closer to average at posttest.

Why is regression to the mean a potential problem in research?

It can be problematic if participants with extreme scores at pretest are selected for study.

How can regression to the mean be controlled in research?

By using a comparison control group.

What is attrition in the context of research studies?

The loss of participants due to not showing up or dropping out of the study.

When is attrition particularly problematic?

When participants are tested multiple times.

What is differential attrition?

in a multi-group design, groups become different on an extraneous variable because of differences in the loss of participants across the groups

Provide an example of differential attrition.

In a study where the independent variable is a new exercise program, more participants may drop out of the experimental group due to lack of motivation.

What is selection in research design?

production of nonequivalent groups because a different selection procedure operates across the groups

Why is selection a threat in multi-group designs?

It can occur when no random assignment is used.

What are additive and interactive effects in research?

Effects produced by the combined effect of two or more threats, highlighting interactions between selection and other threats.

What can cause a failure to generalize study results?

Lack of random selection, chance variation, and failure to identify interactive effects of independent variables.

What is population validity?

The degree to which study results can be generalized to the target population.

What is ecological validity?

The degree to which study results can be generalized across different settings or environmental conditions.

What is temporal validity?

The degree to which results can be generalized across time.

What is treatment variation validity?

The degree to which results can be generalized across variations in the treatment.

What is outcome validity?

The degree to which results can be generalized across different but related dependent variables.

What is the relationship between internal and external validity?

They are often inversely related; factors that increase internal validity may decrease external validity.

How is external validity established?

Through replication of studies.

What is the emphasis on internal or external validity dependent on?

Whether or not a causal relationship has been established.

Which option may compromise the statistical conclusion validity of an experiment?

If the experiment has too few participants, thus reducing power.

What conclusion did Dr. Gomez reach regarding stress and carbohydrate consumption?

Increasing a person's stress level leads to an increase in carbohydrate consumption.

What type of validity is achieved if Dr. Gomez's inference is correct?

construct validity

Which type of validity is concerned with using operational definitions for research participants and variables?

Construct validity.

Whether experimental research is conducted in a laboratory, a doctor's office, or at the mall.

experimental setting

Which of the following is an example of research participants?

Individuals with ADHD.

What are demand characteristics in an experiment?

participants in a study are drawn from the large Introductory Psychology classes, and students who have already participated pass rumors to others about what the
experiment is trying to "prove” and try to determine what the experiment is about from the equipment in the laboratory

Which example illustrates demand characteristics?

Participants in a study are drawn from large classes and share rumors about the experiment's purpose.

What explanation for Nancy's survey findings falls under 'reactivity'?

Participants likely lied about cheating to present themselves positively.

What can influence participant performance in an experiment beyond the independent variable?

Participants' perceptions of what the experimenter wants them to do.

What can happen when researchers have a vested interest in confirming their hypotheses?

Researchers can subtly, unintentionally influence participants' performances.

Why might the results of Dr. Ferenz's and Dr. Martinez's studies on rape victims differ?

They probably asked very different kinds of questions due to experimenter expectancy.

What is the significance of operational definitions in research?

They clarify the meaning of variables and ensure consistency in measurement.

What is the role of the independent variable in an experiment?

It is the variable that is manipulated to observe its effect on the dependent variable.

What does the dependent variable represent in research?

It is the outcome or response that is measured in the experiment.

How can the experimental setting affect research outcomes?

Different settings may influence participant behavior and responses.

What is an example of a confounding variable?

Any variable that could influence the dependent variable and is not controlled for.

What is the purpose of using a control group in experiments?

To provide a baseline for comparison against the experimental group.

What is the difference between qualitative and quantitative research?

Qualitative research focuses on understanding concepts and experiences, while quantitative research focuses on numerical data and statistical analysis.

What is the importance of random assignment in experiments?

It helps ensure that participants are equally distributed across conditions, reducing bias.

What is the concept of 'reactivity' in research?

It refers to changes in behavior when participants are aware they are being studied.

What is the significance of replicating studies in psychology?

Replication helps verify findings and ensures they are reliable and generalizable.

What can lead to biased results in psychological research?

Experimenter expectancy and participant demand characteristics.

What is the role of ethics in psychological research?

To ensure the welfare and rights of participants are protected throughout the study.

What is an experimenter-attribute effect?

It refers to the influence of the experimenter's characteristics on the results of a study, such as participants being more comfortable disclosing information to a female experimenter.

What are demand characteristics in research?

They occur when participants figure out what the experimenter wants to know and provide answers that align with those expectations.

What is an example of an experimenter-bias effect?

When an experimenter 'mines' data for facts that weren't actually present, leading to biased conclusions.

What types of attributes may bias a research study?

Psychosocial attributes, situational factors, and biosocial attributes.

Which of the following is NOT a psychosocial experimenter attribute?

Hostility toward the participant.

What is an experimenter-expectancy effect?

It occurs when an experimenter's expectations unintentionally influence the participants' behavior, such as treating attractive participants differently.

What is an example of the experimenter-expectancy effect?

An experimenter expects females to perform worse on a math task and behaves in ways that undermine their performance.

What is an example of participant reactivity?

When a researcher unintentionally gives hints and extra time to one group of participants, affecting the results.

What does it mean for an experiment to have internal validity?

It means that any observed changes in the dependent variable are confidently attributed to manipulations of the independent variable.

Without internal validity, what cannot an experimenter do?

Make statements about cause and effect.

What is an extraneous variable?

A variable that is not controlled by the experimenter and can impact the results.

What is a confounding variable?

An extraneous variable that has a differential impact on the different groups in a study.

How can the confounding influence of extraneous variables be eliminated?

By holding the variable constant or using random assignment.

What is the significance of gender as an extraneous variable in an experiment?

If gender is not controlled, it can confound the results, making it unclear whether the observed differences are due to the treatment or gender.

What does it mean when an experiment is confounded by gender?

It indicates that gender differences may have influenced the results, rendering the conclusions inconclusive.

What is the role of operational definitions in internal validity?

They ensure that the dependent and independent variables accurately capture the concepts they represent.

What is the relationship between internal validity and cause-and-effect statements?

Internal validity allows for confident cause-and-effect statements regarding the relationship between variables.

What happens if an experimenter does not control extraneous variables?

The experiment may lack internal validity, making it difficult to draw reliable conclusions.