Chapter 4
Study Critique
Previous research suggests that regular family meals may strengthen family bonds and promote positive emotional outcomes for adolescents. This study examines whether adolescents who eat dinner with their families more frequently report higher emotional well-being than those who do n
Method
Participants
85 families (2 parents, one child) recruited from one suburban public middle school
Each family included at least one adolescent between the ages of 12 and 15
Participation was voluntary; each family member received a $20 gift card ($60 total)
Design
Families were divided into two groups based on self-reported family dinner frequency:
High Dinner Frequency Group: Families reporting 5 or more family dinners per week
Low Dinner Frequency Group: Families reporting 2 or fewer family dinners per week
Procedure
At the beginning of the semester, parents reported how often their family ate dinner together in a typical week.
Adolescents completed an online survey measuring emotional well-being.
Families in the high-dinner-frequency group were encouraged to continue their usual routines.
Families in the low-dinner-frequency group were encouraged (but not required) to increase the number of family dinners during the semester.
After eight weeks, adolescents completed the same emotional well-being survey again.
Measures
Independent Variable (IV): Family dinner frequency (high vs. low)
Dependent variable (DV): Adolescent emotional well-being, measured using a 6-item self-report questionnaire created by the researchers
Family Support
Anxiety
Depression
Peer Problems
Results
Adolescents in the high dinner frequency group showed an average increase of 1.2 points on the well-being scale.
Adolescents in the low dinner frequency group showed an average increase of 0.4 points.
A repeated-measures ANOVA was conducted.
The main effect of group was reported as statistically significant (p = .04).
No information about effect size or attrition was reported.
The Basics of Research
Variables: an event or process that is measured in a study
Measured: The assignment of numbers to observations based on a set of rules/principles
Can be manipulated or changed (treatment or control), or they can remain constant (ex., race)
Variables MUST have at least 2 levels or categories
A variable with one level is called a constant
Dichotomous variables: 2 levels
Trichotomous: 3 levels
Continuous: Infinite levels (theoretically)
Example Variables
Body mass index (Continuous)
Depressive Symptoms (Continuous)
Major Depressive Disorder (Binary: Clinical depression yes/no)
Response time (Continuous)
Number of correct responses (Count)
Attitudes towards deviance (Continuous)
Each variable has an operational definition: a set of methods and procedures to obtain the variable
The same construct is operationalized differently across studies
Pain will NOT be measured the same way → there are dozens of surveys assessing Pain
Pain may be measured using numbers (1-10) or using images
Pain can also be measured via observational data (ex., number of winces)
Can be coded by either the participant later on (ex., video review) or trained research assistants
Conceptualization and Operationalization
You cannot study a variable without conceptualizing and operationalizing all of your variables
Conceptualization: a theoretical definition of the construct that specifies what the construct is as well as what the construct is not
Operationalization: A specific set of criteria by which your variables will be measured or how they will be manipulated (in experimental designs)
Indicators serve as the specific items you will ask or the specific ways in which the variable will be manipulated
Conceptualization
Conceptualization should be a brief overview of the construct of interest and provide a concrete explanation of what the construct is
Conceptual definitions are general and the most abstract
Good conceptual definitions identify the key properties of the construct
When defining a construct, be able to answer the question “What do you mean by that?”
Example: What do you mean by childhood adversity?
Conceptualizations do not provide examples of the construct, but identify common themes among the examples
If struggling to conceptualize a variable, pick some examples and find a common theme
If some examples do not fit with the others, then they may not be within the conceptual definition
Conceptualization of ACEs
Childhood adversity is defined by non-normative events (ex., discrete) and processes (ex., ongoing, chronic) that have the potential to cause short and long term harm, exceeding the ability for the child's ability to cope. ACEs can be directly experienced by the child or exposure to these events and processes where someone else is the victim; these events and processes can be interpersonal or non-interpersonal in nature
Operationalization
For most concepts/constructs, it will be impossible to effectively measure every part of the construct
To address this issue, we need to operationalize our constructs
Explicitly defining what aspects of the constructs we are going to measure
Your constructs can be operationalized in numerous different ways
Up to you to determine which way is best based on your research questions
Operationalization of ACEs
The frequency of occurrence of emotional abuse, physical abuse, sexual abuse, neglect, exposure to domestic violence, parental divorce, parental mental illness, and substance use
You’ll notice that this list of adversities consistent with our conceptual definition does not consist of every adversity that would meet the criteria to be an ACE
You’ll also notice that we didn’t include the severity of ACEs (ex., being slapped is very different than being punched
Types of Associations
Types of Relationships Between Variables
Positive Association
As the levels of one variable increase, the other variable also increases
Education Example: Hours studied and grades
HDFS example: Emotional support received and relationship satisfaction
Negative Association
As one variable increases, the second variable decreases
Education Example: Hours watching TV per week and GPA
HDFS example: Marital happiness and depressive symptoms
No Association
There is no relationship between the variables
Education Example: the association between classroom temperature and the number of questions asked after class
HDFS example: Neighborhood quality and number of text messages exchanged between marital partners
Curvilinear Relationship
The relationship is NOT linear - the rate of change differs as the X variable increases or decreases
Education Example: Arousal and learning/performance
HDFS Example: Frequency of interaction and relationship satisfaction
Validity
The extent to which knowledge represents the “Truth.”
Occurs on a spectrum
Examples
Do you believe the testimony of the witness?
Do you believe the claims that a product works in the way it is said to?
If obtaining news from Twitter, how do the following influence how valid you evaluate the following:
No profile picture
What does having an avatar “profile”
Having a blue checkmark
Having a gold checkmark
Validity is the approximate truth of an inference
Often used in a dichotomous way (valid or invalid), but validity is actually a spectrum
Inherently entails human judgement → fallible
Is a property of INFLUENCEs, not methods or design (ex., the same design can have more or less influence on conclusions under different circumstances)
Example: Randomized control trials assessing psychological responses to discrimination comparing race (“not hidable”) to sexual orientation (“hidable”)
Attrition in trauma therapy (may be higher in intervention vs control group) - abreactions.
In the social sciences, we establish validity through consistent empirical findings that mirror the theory
Introduction to the Four Flavors of Validity
Construct Validity
The extent to which the measurement or manipulation of a variable accurately represents the theoretical construct being studied
In the social/behavioral sciences, many of the constructs we are interested in cannot be directly measured
Cannot physically hold depression
You cannot test “marital satisfaction” in an MRI
Instead, we use methods (ex., surveys, observations) to infer that a construct exists
Example: Depression has criteria in the DSM-5-TR
We infer the constructs’ existence from the data that we gather
We infer that people are depressed from their scores on a survey assessing symptoms of depression
Another critical issue in construct validity is ensuring that you have good indicators of the construct
In the late 1800s and early 1900s, researchers were increasingly interested in measuring IQ
The predominant view of the time was eugenics (ex., the biological inferiority of nonwhite people, particularly black people)
To compare the IQs of white and black individuals, two early methods were head circumference and volume of the skull
How valid are these methods at measuring IQ?
Internal Validity
The accuracy of conclusions drawn about a study’s findings and the found relationships is “Truth.”
Talked about most frequently in the context of a causal relationship
Focused on the results of the study → how accurate are they?
Researchers try to increase internal validity via a strong methodology in a highly controlled setting
Isolate the effects
Generally, the most important thing when first studying a phenomenon
Teaching Example
We want to conduct a teaching intervention in a highly controlled environment first, then apply it in the classroom
If teaching intervention is ineffective in highlight controlled setting → unlikely to be effective in the classroom
Researchers are conducting an experiment to evaluate memory performance among older adults (65 and older), and these are their protocols:
Bring adults into a lab on the first floor of a research center
Obtain informed consent and baseline surveys
Use three conditions based on difficulty (easy memory task, moderate memory task, and difficult memory task - each person will do all three
Randomize order
Rule out adults with Alzheimer’s / dementia or TBI
Research protocols are done between the hours of 9 am and 3 pm
Protocols were conducted in the most isolated wing of the center
Identify a specific and homogenous population when the phenomenon was occurring (memory in older adults)
Use experimental designs with control groups
Randomize the order in which adults do the memory tasks - reduce the influence of one condition (easy memory) on other conditions (difficult memory)
Randomization of participants to conditions
Helps significantly with the 3rd variable problem
Inclusion/exclusion criteria rule out people with existing memory conditions
Conduct memory tasks during times of day when memory and cognitive performance are highest, and have everyone complete protocols during that time period
Avoid distractions by experimenting in a secluded area
Collect data in multiple ways
Surveys about stress
Physiological data of stress
Ask other people (friend, partner, therapist) about the participant’s levels of stress
Variables that decrease internal validity
Using a sample of adults with a greater age range (45 and over) → phenomenon of interest is more salient in older adults
Everyone goes through the same protocols in the same order (protocol effects may induce bias)
No exclusion criteria - adults with memory related disorder will severely skew the results
The task is scheduled at the convenience of the participant
Some people could be available at 9 am, 12 pm, or 5 pm - highly available
Experimenting just off a busy hallway - distractions
External Validity
The extent to which the findings from your study generalize across people, settings, other variables, and time
A study consisting of HDFS college students doesn’t apply to adults in the general population
Not all adults go to college
Not all adults who go to college major in HDFS
Is a study on the effectiveness of a therapy that is conducted in a research lab the same as the same therapy done in the community
A community setting is much less controlled
Participants aren’t receiving compensation like they would in the lab setting
If a therapy is effective for reducing generalized anxiety, is it also effective for reducing social anxiety?
Statistical Conclusion Validity
Fancy way of saying, did you screw up your stats?
Statistical analyses are far more complicated than they may seem
Example: Randomized control trial tracks symptom reduction and side effects of a drug over 10 weeks. In their analysis, they remove participants who stopped taking the drug after only 2 weeks
Removing people based on an arbitrary criterion (not theoretically justified) could change the conclusions drawn about the effectiveness
Experimental vs Non-Experimental Methods
Experimental methods: relationships between the measured (observed) variables are a function of a manipulation of a variable (IV)
Randomized control trial
Non-Experimental Methods: relationships between observed variables are not a function of manipulation and instead are simply observed/measured
Most survey designs
Obtaining records
Observations
Experimental Design
Can begin to establish causality
No single study (even with manipulation) can establish cause and effect; it takes a cumulative science to establish causality
Causality isn’t the most pernicious problem (astronomers cannot manipulate the solar system, but can make predictions about planetary movement
Third variable problem: when there is a 3rd unmeasured variable that affects both your IV and DV and accounts for the association between them
Example: Ice cream and violence - both are higher in the summer
Experimental design involves the manipulation of your independent variable (ex., teaching intervention, model of psychotherapy)
Studies are designed to create temporal precedence (ex., manipulate IV THEN observe DV)
Creates temporal precedence among the variables
Begins to rule out / account for third variables via a highly controlled environment (high internal validity)
Do this methodologically (Example: use a homogenous sample of SES and look at the relationship between GPA and the value of the first car) or through randomization
Do this statistically (Example: ask questions about family socioeconomic status and use it as a control variable in your model)
Dreaded “above and beyond” language
There are downsides and problems with experimental design
Highly controlled setting → low external validity
Findings from a highly controlled setting to a setting with very little control will affect findings
Some variables cannot and should not be manipulated
Research ethics: cannot randomize people into child abuse vs no child abuse conditions
Instead, study the variables in their naturalistic setting and compare group differences
There is bias because one group may be systematically different than another group and likely in multiple ways
Some variables cannot be manipulated (ex., race, socioeconomic status)
Non-Experimental Design
Cannot establish causality - even longitudinal research
Exceptionally difficult, if not impossible, to establish directionally
Plausible reasons for IV predicting DV and DV predicting IV
Suppose that you were interested in understanding the relationship between communication patterns and depressive symptoms in married couples
Varying methodologies
Give each partner a paper and pencil survey
Have each partner complete an online survey
Invite the couple into the lab and engage in a behavioral observation
Have each couple go through a diagnostic interview with a psychiatrist
Non-experimental designs have lower internal validity, but have greater external validity
Recruiting people in their own communities and having the researchers go to them to collect data in their own environment
Provides additional data
Doesn’t require a visit to the lab
Sampling tends to be more general (ex., not college students)
College students are a convenient sample → who do they generalize to??
Methodologies
Implement multiple methodologies in your data collection
Monomethod bias: systematic bias that occurs when you collect data in only one way (ex., surveys)
Multitrait, Multimethod research collects data from multiple people (ex., student self-report, parent report, teacher report) and multiple methodologies (ex., observation, surveys, standardized testing)
An entire subset of statistical analyses designed to analyze dyads and groups