Chapter 14
External Validity
Example 1: Generalizing a particularly negative dating experience with a Kevin to form the conclusion that all Kevin’s are bad → overgeneralization where the name of a period is presumed to cause their behavior
Example 2: Generalizing that a communication style that previously worked will work in future relationships → undergeneralization, where you ignore the characteristic that you don’t like
Example 3: Generalizing what has worked for you in the past in a similar situation will also work for another → Over-generalization can be attributable to either 1) over-generalizing the specific behavior or 2)
Example 4: Generalizing the same behavior with the same person will work across different contexts → overgeneralization based on situation
Generalization
One of the most critical, yet under-discussed, aspects of research methods is generalization
Generalization is the extent to which the findings in your study apply to a larger population or different circumstances
Do 22-year-old students at OSU represent all 22-year-old students? All 22-year-olds?
Are the effects of financial stress and depression during COVID applicable to financial stress and depression now?
If you recruit people and they are highly heterogeneous in terms of their characteristics, what is the larger population you are attempting to generalize to?
External validity is related to the ability of the findings of our study to generalize to other circumstances
Person, Settings, Treatments, and Outcomes
Person: Do the findings of my study apply to another group of people who have different characteristics (ex., race, gender, income, severity of problems)
Example: Would the findings of my couple therapy intervention in heterosexual couples also apply to gay and lesbian couples?
Setting: Do the findings of my study apply to a nearly identical sample (ex., same proportion of gender), but in a different context
Example: Would the findings of my couple therapy intervention in heterosexual couples conducted in a research lab also be found in a community-based mental health clinic
Treatments: Would treatments that are very similar to each other have the same effect?
Example: Principles of cognitive behavioral therapy are seen in over half a dozen treatments for marital problems (ex., Behavioral Couple Therapy (BCT), Gottman Couple Therapy (GCT)). Does it matter if couples get BCT or GCT?
What if BCT has half the number of sessions, requires less training, and is cheaper to train therapists in?
Outcomes: Do the effects that were found in the study generalize to other, closely related examples?
Example: Couple therapy is effective for reducing conflict. To what extent can we also say that it is effective in increasing intimacy and couples coping together
It is plausible and even necessary to make generalizations to other populations, settings, treatments, and outcomes
It is impossible to study every known treatment, in every population, in every setting
We must gauge how big a generalization is appropriate
Generalizing that therapy reduces generalized anxiety symptoms to social anxiety symptoms isn’t too much of a stretch
Generalizing that therapy reduces generalized anxiety symptoms to psychosis is far too big a stretch
Increase generalizability through multiple conditions, samples, populations, settings, etc.
Test for differences within your sample (age, race, gender)
Use different measures assessing the same construct (ex., testing clinical levels of depression, depressive symptoms)
It is, however, impossible to identify all possible conditions, subpopulations, and settings
Knowledge is cumulative
This is why scientists have a program of research (ex., studying the same thing in different ways over the course of a career)
We must also be careful by adding in too many subpopulations
Expensive
Time consuming
Treat to statistical conclusions’ validity (if there are differences in the effects in various subpopulations, then you’ve messed up your stats!)
Example: If your correlation between two variables is .30 in one group and -.30 in another, when analyzing them together, the correlation is 0 → we’re missing two effects that are actually there
Characteristics of Your Study and Generalization
Gender
Historically, gender has been equated to biological sex, but this is no longer the case
Biological sex refers to an individual’s physiology, while gender is a social construct related to identity
When writing a grant, I must have a scientific rationale for the exclusion of a specific gender, and my study must include both to increase generalization
Race and Ethnicity
Much like gender, race, and ethnicity are inappropriately equated
Race is categorization based on skin color (most common) or other physical features (less common), while ethnicity refers to the cultural heritage
Race is general, while ethnicity is more specific - Asian would be considered a race, but Chinese or Japanese would be ethnicity
By using the term Asian (race), you are assuming all Asian cultures are the same in relation to what you are studying
Culture reflects the attitudes, norms, expectations, interpretations, knowledge, language, and expression of a social group
A majority of studies represent White middle-class adults from the industrialized world
Several major/large-scale longitudinal (ex., MIDUS) have predominantly White middle-class adults, but have made strides to include others (ex., recruitment of an exclusively African American sample from Milwaukee)
Gender, race/ethnicity, and culture will influence your study in numerous ways
Contribute to how they interpret the questions (ex., asking someone if they are feeling down or blue is commonly accepted in the US, but this doesn’t translate to other languages)
Scales that we use may indicate pathology, but in other cultures, it may be a sign of health (ex., individual decision making)
Individuals coming from less wealthy countries may not be literate in any language (common among Hispanic Immigrants)
The phenomenon of interest may not be acceptable to disclose (ex., mental health), and participants are unwilling
Threats to External Validity
Interaction with “Units” (ex., participants) → the effect/association differs across various subgroups of participants, such as race, education, income, or level of the variable
An intervention is effective at reducing the effects of depression, but only for those who are more severely depressed
An intervention is only effective for men and is not effective for women
Interaction with Treatment Variations → the effect/association differs across closely related treatments (ex., variations)
Couple therapy does not reduce the risk of divorce, but couple therapy and simultaneous individual therapy are effective for reducing the risk of divorce
Interaction with Outcomes → the effect/association differs across outcome variables
Couple therapy is effective for reducing conflict, but it does not increase intimacy
Interaction with setting → the effect/association differs across the context in which the study was tested
Group therapy is highly effective in general, but is ineffective in inpatient hospital settings
College students are a significant threat to the external validity of processes that are general and not specific to college students
2 out of 3 studies in psychology journals used college student samples
College Students and External Validity
Locations
College towns tend to be more liberal than the surrounding areas
Manhattan, Kansas Vs Manhattan, New York
Florida vs California vs New York vs Minnesota
Recruitment
College students are generally in introductory psychology courses (Freshmen and Sophomores)
Participation in research is often a requirement (not even extra credit)
Sample Characteristics
There is a selection effect where college students have higher cognitive abilities than those who were not accepted into college
College is a time of substantial biopsychological change - you may be catching students in a time period when the phenomenon being studied is or is not particularly salient
Example: Alcohol use in college students
There is actually an inverse relationship between high school SAT scores and college GPA
Berkson’s paradox
You find the inverse of an expected/hypothesized relationship due to sample selection
Overall, there is a positive relationship between SAT and college GPA, but if you only sample a specific slice of students (those accepted), then you’re censoring your data
You are not including those who did not get into college, and missing those who got into college but went to a better school
In-person vs online studies
Would you rather complete an online survey that is 2 hours long or go to a lab for one hour and be hooked up to physiological equipment?
Colleges are becoming more and more diverse and representative of the larger population
Still have a generalizability issue related to the pursuit of a college degree
Generalization Across Situation
In addition to generalizing to a larger population, which is an absolutely critical piece of research, so is generalization across situations.
Measurement Example:
If you are testing the relationship between marital conflict and divorce, you can have participants engage in a conflict discussion task, have a couple therapist interview them, and have them complete a survey - do you get the same results??
Experimental Conditions Example:
If you conduct a conflict discussion task in a laboratory, do couples fight the same way at home?
Replication
Replication vs Reproductivity
Replication: a researcher designing a study in the exact manner another researcher designed the study to determine if they get the same findings
Mirror recruitment, methods, measures, and statistical analysis
Reproductivity: taking another researcher’s data set and statistical analysis and determining if they can get the same findings
Statistics vary, but most studies cannot be replicated
Ranges from discipline to discipline, but can be less than 1% to 25% of studies that cannot be replicated