Chapter 8: Internal and External Validity

Internal Validity

Type of evaluation of your experiment; it asks whether your IV is the only possible explanation of the results shown for your DV; the most important property of any experiment (cause-and-effect)

History (Threat to Internal Validity)

Refers to events that occur between the DV measurements in a repeated-measures design

Maturation (Threat to Internal Validity)

Refers to changes in participants that occur over time during an experiment; could include actual physical maturation, or tiredness, boredom, hunger, and so on

Testing (Threat to Internal Validity)

A threat to internal validity that occurs because measuring the DV causes a change in the DV

Practice Effect

A beneficial effect on a DV measurement is caused by previous experience with the DV. (testing effect)

Reactive Measures

DV measurements that actually change the DV being measured, changes the behavior in question simply by measuring it (testing effect)

Nonreactive Measures

DV measurements that do not influence the DV being measured, no changes in behavior in question simply by measuring it (e.g., one-way mirrors, hidden cameras and microphones, naturalistic observation, deception, and so on.

Instrumentation / Instrument Decay (Threat to Internal Validity)

Occurs if equipment or human measuring the DV changes the measuring criterion over time (a careful experimenter checks out the equipment each day; or interrater reliability)

Statistical Regression (Threat to Internal Validity)

Occurs when low scorers improve or high scorers fall on a second administration of a test solely as a result of statistical reasons, regression to the mean (average)

Selection (Threat to Internal Validity)

Occurs if participants are chosen in such a way that the groups are not equal before the experiment; the researcher cannot then be certain that the IV caused any difference observed after the experiment

Morality (Threat to Internal Validity)

Occur if experimental participants from different groups drop out of the experiment at different rates; differential dropout rates could be a significant problem in research that spans a long time (weeks or months, for example).

Interaction With Selection (Threat to Internal Validity)

Can occur if there are systemic differences between or among selected treatment groups that interact with maturation, history, or instrumentation

Diffusion or Imitation of Treatment (Threat to Internal Validity)

Occurs if participants in one treatment group become familiar with the treatment (information / procedure) of another group and copy that treatment

External Validity

A type of evaluation of an experiment; do the experimental results apply to populations and situations that are different from those of the experiment? Unlike internal validity, this is not an absolute necessity; but still important for real-world applications

Generalization

Applying the results from an experiment to a different situation or population; we would like to take our results beyond the narrow confines of our specific experiment

Population Generalization

Applying the results from an experiment to a group of participants that is different and more encompassing than those used in the original experiment

Environmental Generalization

Applying the results from an experiment to a situation or environment that differs from that of the original experiment.

Temporal Generalization

Applying the results from an experiment to a time that is different from the time when the original experiment was conducted

What is the Interaction of Testing and Treatment in terms of Threat to External Validity?

It occurs when a pretest sensitizes participants to the treatment yet to come.

How can the Interaction of Testing and Treatment be addressed in research designs?

Nonpretesting designs like posttest-only control-group design and Solomon four-group design involve pretests for some groups and no pretesting for others to measure pretesting effects.

Interaction of Selection and Treatment (Threat to External Validity)

Occurs when a treatment effect is found only for a specific sample of participants; this becomes a greater risk when it becomes harder to get participants, the participants you find are likely will be unique and not representative of the general population.

Reactive Arrangements (Threat to External Validity)

Caused by an experimental situation that alters participants' behavior, regardless of the IV involved; refers to conditions of an experimental setting (other than the IV) that alter our participants behavior, we cannot be sure that the behaviors we observe in the experiment will generalize outside that setting because these artificial conditions do not exist in the real world

Demand Characteristics (Threat to External Validity)

Features of the experiment that inadvertently lead participants to respond in a particular manner; demand characteristics can convey the experimental hypothesis to the participants and give them clues about how to behave.

Multiple-Treatment Interference (Threat to External Validity)

Occurs when a set of findings result only when participants experience multiple treatments in the same experiment; findings may be specific to situations in which the experimental participants experienced these multiple treatments, if they received only one treatment, the results may differ.

Comparative Psychology

The study of behavior in different species, including humans (Infamous White Rat); concerns with external validity include: generalizing from rats to all other animals may be a stretch; if you are interested in generalizing from animal to human behavior, there are certainly closer approximations to humans than rats in the animal kingdom

Convenience Sampling

A researcher's sampling of participants based on ease of locating the participant; often it does not involve true random selection (College students enrolled in Introductory Psychology courses)

The "Opposite" or "Weaker" or "Inferior" or "Second" Sex

Refers to gender issues arising from limited women advancement and opportunities in the past and still now, also discusses lack of acknowledgement of women in some psychological theories and how the lack of men entering in psychology currently can make it more difficult to have adequate male representation in the future.

Even the Rats and Students Were White

Refers to racial issues arising from limited African American and other racial groups including Hispanics, Asians, and American Indians in research opportunities and advancement

Even the Rats, Students, Women, and Minorities Were American

Refers to the regional issues or cultural issues arising from limited advancement, research opportunities, and research participation outside of the United States and Western Europe

Replication

An additional scientific study that is conducted in exactly the same manner as the original research project

Replication with Extension

An experiment that seeks to confirm (replicate) a previous finding but does so in a different setting or with different participants or under different conditions.

Threats to Internal Validity

History, maturation, testing, instrumentation, statistical regression, selection, mortality, interactions with selection, and diffusion of treatment

Threats to External Valdity

Interaction of testing and treatment, interaction of selection and treatment, reactive arrangements, demand characteristics, multiple-treatment interference, and diverse experimental participants

Douglas Mook (1983) argued that not all experiments need to generalize to the real world. He suggested that the importance of external validity depends on the research goals. Mook outlined four alternative research goals that do not prioritize external validity they are?

Exploring Possibilities: Research may aim to discover whether something can happen, rather than if it usually happens.

Predicting Lab Behavior: Some studies predict behavior in a lab setting based on real-world observations.

Strengthening Validity: Demonstrating a phenomenon in a controlled lab setting can strengthen its validity.

Studying Unique Phenomena: Some research investigates phenomena that do not have direct real-world analogs.

Changing a behavior simply by measuring it refers to
a. A reactive measure
b. A measured IV
c. A manipulated IV
d. History

a. A reactive measure

Comparing intact groups is likely to cause a problem because of
a. Morality
b. Selection
c. Maturation
d. History

b. Selection

Why is it important to evaluate your experiment for internal validity?

It is important to evaluate your experiment for internal validity because you cannot place any confidence in your results if your experiment does not have internal validity. Cause-and-effect statements cannot be made without internal validity.

History
a. Your DV scorer gets sick and you recruit a new person to help you
b. You are conducting an experiment on racial prejudice and a race riot occurs between tests.
c. Your participants grow bored and disinterested during your experimental sessions.
d. You use a before-and-after DV measurement and the participants remember some of their answers.

b. You are conducting an experiment on racial prejudice and a race riot occurs between tests.

Maturation
a. Your DV scorer gets sick and you recruit a new person to help you
b. You are conducting an experiment on racial prejudice and a race riot occurs between tests.
c. Your participants grow bored and disinterested during your experimental sessions.
d. You use a before-and-after DV measurement and the participants remember some of their answers.

c. Your participants grow bored and disinterested during your experimental sessions.

Testing
a. Your DV scorer gets sick and you recruit a new person to help you
b. You are conducting an experiment on racial prejudice and a race riot occurs between tests.
c. Your participants grow bored and disinterested during your experimental sessions.
d. You use a before-and-after DV measurement and the participants remember some of their answers.

d. You use a before-and-after DV measurement and the participants remember some of their answers.

Instrumentation
a. Your DV scorer gets sick and you recruit a new person to help you
b. You are conducting an experiment on racial prejudice and a race riot occurs between tests.
c. Your participants grow bored and disinterested during your experimental sessions.
d. You use a before-and-after DV measurement and the participants remember some of their answers.

a. Your DV scorer gets sick and you recruit a new person to help you

Statistical Regression
a. Many participants find one treatment condition very boring and quit
b. You choose boys from lower-class homes and girls from upper-class environments
c. Students in your control group talk to students in the experimental group and imitate their treatment
d. You select the worst students in the class and try a new tutoring strategy.

d. You select the worst students in the class and try a new tutoring strategy.

Morality
a. Many participants find one treatment condition very boring and quit
b. You choose boys from lower-class homes and girls from upper-class environments
c. Students in your control group talk to students in the experimental group and imitate their treatment
d. You select the worst students in the class and try a new tutoring strategy.

a. Many participants find one treatment condition very boring and quit

Selection
a. Many participants find one treatment condition very boring and quit
b. You choose boys from lower-class homes and girls from upper-class environments
c. Students in your control group talk to students in the experimental group and imitate their treatment
d. You select the worst students in the class and try a new tutoring strategy.

b. You choose boys from lower-class homes and girls from upper-class environments

Diffusion of Treatments
a. Many participants find one treatment condition very boring and quit
b. You choose boys from lower-class homes and girls from upper-class environments
c. Students in your control group talk to students in the experimental group and imitate their treatment
d. You select the worst students in the class and try a new tutoring strategy.

c. Students in your control group talk to students in the experimental group and imitate their treatment

You want to compare the formal education of college students and senior citizens. You select a group of each type of participant and give each a written test of math, social studies, and grammatical information. What threat of internal validity appears likely in this situation? Why?

In this experiment the internal validity threat of selection is likely. Because the selected participants are senior citizens, it is possible that they will have less formal education than college students of today. Obtaining a college education was not as common for today's senior citizens when they were young as it is for today's youth.

Experimental dropouts are a problem linked to
a. Morality
b. Selection
c. Maturation
d. History

a. Morality

What is external validity? Why is it important to psychology?

External validity is the ability to take your experimental findings and apply them beyond the experimental participants to the larger population. It is important to psychology so that we can develop general findings that apply to large groups of organisms.

Generally speaking, as internal validity increases, external validity
a. Increases
b. Decreases
c. Remains the same
d. Fluctuates unpredictably

b. Decreases

Distinguish among population, environmental, and temporal generalization.

Population generalization involves applying results from participants to the larger group. Environmental generalization involves applying results to different settings from those in the original experiment. Temporal generalization involves applying experimental results to different times from those in the original experiment.

Why should we be concerned about trying to use different types of participants (such as minorities and people who are not college students) in psychology studies?

Using different types of participants helps increase the external validity of our findings.

What is cross-cultural psychology? Why is it of particular relevance to this section of the chapter?

Cross-cultural psychology involves testing psychological principles in different cultures to determine the generality of those principles. It is relevant to this chapter because it deals with external validity.

Testing-Treatment Interaction
a. An effect occurs only if participants experience all the experiment's treatment conditions
b. Women, but not men, demonstrate an experimental effect
c. Only participants who are pretested demonstrate an experimental effect
d. Demand characteristics provide cues to the participants about how they should respond

c. Only participants who are pretested demonstrate an experimental effect

Selection-Treatment Interaction
a. An effect occurs only if participants experience all the experiment's treatment conditions
b. Women, but not men, demonstrate an experimental effect
c. Only participants who are pretested demonstrate an experimental effect
d. Demand characteristics provide cues to the participants about how they should respond

b. Women, but not men, demonstrate an experimental effect

Reactive Arrangements
a. An effect occurs only if participants experience all the experiment's treatment conditions
b. Women, but not men, demonstrate an experimental effect
c. Only participants who are pretested demonstrate an experimental effect
d. Demand characteristics provide cues to the participants about how they should respond

d. Demand characteristics provide cues to the participants about how they should respond

Multiple-Treatment Interference
a. An effect occurs only if participants experience all the experiment's treatment conditions
b. Women, but not men, demonstrate an experimental effect
c. Only participants who are pretested demonstrate an experimental effect
d. Demand characteristics provide cues to the participants about how they should respond

An effect occurs only if participants experience all the experiment's treatment conditions

Why did Mook argue that external validity is not always necessary?

Mook thinks that external validity is necessary only when we are attempting to predict behavior in the real world. Much research in psychology is not aimed at such prediction, and Mook believes that external validity is not necessary in such situations.

Why is the notion of external validity for a single experiment virtually impossible to achieve?

It is almost impossible to achieve external validity in a single experiment because of the large number of threats to external validity. Usually an experiment can answer only one such threat at a time.