1/94
Looks like no tags are added yet.
Produce answers
If you do research (as a scientist) or part of your career
Consume answers
Sometimes very important answers that are very real consequences for decisions you make
Descriptive questions
You use empirical methods to describe and explain
Definition: claim about a state of the world
Theory
You use empirical methods to describe and explain
Definition: claim about how and why things are related
Basic (reason for asking)
asking for the sake of knowing, for the sake of generating knowledge
What are the basic patters of human behavior, thought and feeling (personality) over time?
What are the causes of psychopathology?
Applied
asking for the sake of solving a practical problem
How can people improve their well being?
How can we improve student learning?
How can we treat anxiety disorder?
Basic vs Applied research
Fuzzy boundary btw the two
basic often ends up informing application
Applied research on one particular problem can lead to questions about how things work more generally, which leads to basic questions and basic knowledge
Quantitative Methods
turn empirical observations into numbers
Qualitative Methods
Create rich descriptions
Empiricism
Evidence acquired from systematic, objective unbiased observation, independently verifiable by other observers or scientists (practice of gathering information through direct observation)
the belief that all knowledge, behavior, and mental processes come from sensory experience and environmental interaction rather than from innate ideas
Empiricist (scientist)= show me the data
Non-scientific evidence
Intuition or common sense
own personal experience
anecdotes or testimonials (experiences of others)
Authority or convention (base advice on intuition, convention)
4 problems with non-scientific answers
Bias (intentional or unintentional)
Lack of valid comparison
Problems with casual interference
Lack of generalizability
Bias (problem w/ non scientific answers)
Bias of others, trying to sell of persuade (intentional)
Bias in your own memory and interpretation of your own experience (unintentional)
Availability heuristic
relying on information that comes to mind quickly
Confirmation bias
Tendency to search for, interpret, and recall information that confirms preexisting beliefs
Safety of vaccinations
Evidence of human cased climate change
Lack of valid comparison (4 problems with non-scientific answers)
(non-scientific answer compared to what?)
No control group
Problems with causal inference (4 problems with non-scientific answers)
Other things can correlate with this happening
Ex: I learned more when I got coffee before class
Confounds: less stress, more sleep, may correlate with with time to coffee (less stressed if have time to get coffee?)
Placebo effect: expected that coffee will improve learning
Bad at figuring an actual outcome
Frequency
How often does X happen?
Focuses on one variable
How many people smoke?
How many people text while driving?
How common is voter fraud?
Association
Is X related to Y?
Is course performance related to attendance?
Is time in day care related to behavioral problems?
NOT CASUAL question/claim, just whether two or more variables related
Causality
Does X cause Y (Descriptive)
Does caffeine improve memory?
Why/how does X cause Y (explanatory)
Why does caffeine improve memory?
Variables
A factor or dimension that can take more than one value (on which people can vary)
any characteristic, quality, or condition that can change or take on different values across people, situations, or time
In the questions and claims we talked about, X and Y are placeholders for variables
How does X happen? (how many students will study)
Is X relative to Y? (gender and fear of horror of films)
Does X cause Y? (personality and aggression)
How does X cause Y? (how many time of day affect mood)
Conceptual (variables)
The abstract concept
It explains the meaning of a variable/term that you can’t measure
ex: Height, intelligence, anxiety
Operational definition
How the concepts gets cashed out in terms of something concrete that can be observed
turn an abstract, general idea into a clear, measurable, and concrete action or value so it can be tested in a research study
Easy sometimes (height=centimeters)
Sometimes tricky
anxiety (I cant see, hear, or feel from your anxiety… What can I see)
learning (I cant see your learning or mine.. what can I seee)
Measured variable
Naturally occurring
the data or outcome that a researcher records, observes, or tests in an experiment
Study time
sadness
Manipulated variable
Researcher sets up condition to produce it
Study Time
Sadness
Discrete value
countable values
Gender (many studies compare men to women)
Smoking (smoker, non-smoker)
Yes or no, this or that
Continuous
views human growth and change as a gradual, cumulative, and smooth process rather than a series of abrupt, distinct stages
Response time variable: 1.07 seconds, 2.53 secs, 5.25 secs
Smoking variable: 1,5, 10 cigarettes per day
range
Constant
A factor or dimension that could vary but doesn’t in a particular study
Participants are all the same age or gender
All participants are given the same amount of time for a task
Sample of participant all diagnosed with ADHD
Self-report
A measure in which participants provide responses to questions about themselves
Informant-report
Reporting beliefs about what someone else knows or does
Observational/behavioral
A measure in which the researcher records observable behaviors, including behavioral indicators of internal states
Observe what people do
Physiological measures
measure of a biological state
Converging evidence
When different measures provide outcomes pointing to the same conclusion
Reliability
The consistency of the measure
bathroom scale (can we trust it?)
Three types of reliability
test retest
interrater
internal
Test retest reliability
similar scores when the measure is repeated
height of an adult measured by tape measure
Use scatterplot to evaluate relaibility
Interrater reliability
different observers record similar result
short answer test responses scored by same two people
number of people on cell phones
Internal reliability
different questions/ parts of a measure produce similar results
People who are more likely to agree with any one item are more likely to agree with all items
Why isn’t reliability perfect?
Observed score (we can never know true score, just measured score)
Measurement error
Observed score
Observed score= signal+ measurement error
Measurement error
any factor that distorts the observed score for reasons unrelated to the construct
ex: construct=learning
ex: operationalization = exam score
How to improve test-retest reliability
Standardize the administration procedure
Control situational factors (lighting, temp)
minimize fluctuation in participants' transient states (mood, fatigue)
ex: administer at same time
How to improve internal relaibility
Carefully examine items
ex: depression (sad, feel sad, cry, irritable, worry)
Give clear instructions to participants
Make task length manageable
Minimize distraction
Improve interrater reliability
Develop good scoring key
Train the raters: teach evaluators how to assess performance, behavior, or data accurately while reducing personal bias and subjective errors
Check for coding errors
Construct validity
Does the assessment measure what it’s supposed to measure?
Does it predict what it is supposed to predict?
Face validity
Does the measure resemble what it’s supposed to measure?
judged based on own theory of construct
Note: this is subjective criteria
Ex: How adventurous are you?
Construct (abstract concept used to explain human emotions)= Extraversion (high energy, wanna interact with external world)
Sometimes, valid items do not have face validity
Ex: Do you prefer showers or baths? (low face validity, used to measure empathy, able to use measure cause people who seem to pick showers are more empathetic)
Concurrent Evidence
What should the measure relate to right now?
Predictive Evidence
What should the measure predict later?
Convergent Evidence
What similar measures agree with it?
Discriminant Evidence
Is it distinguishable from different constructs?
Concurrent validity
measure related to an outcome or behavior that it should be related to, that is measured at the same time
shows how well the results of a new test or measurement agree with an established, trusted benchmark (a "gold standard") when both are administered at the same time
ex: blood pressure measure by doctor, while blood pressure ring also measuring blood
ex: motivation measured NOW —> current GPA
Predictive Validity
Is a measure related to an outcome or behavior that it should be related to, that is measured in the future?
ex: motivation measure NOW —> GPA at end of Year
Similarities between concurrent validity vs. predictive validity
Both collect measures of X NOW
collect measure of outcome either NOW or in FUTURE
timing of outcome based on when you measure it, NOT when it happens
Both established via
Correlations (across participants)
Differences in means via known groups
differences in means via manipulated varaible
Convergent validity
Is the measure strongly related to other measures of the same (or similar) constructs?
similar measures
How is convergent validity different than “outcome” in concurrent/predictive validity?
Another measure vs. the outcome of a measure
Sometimes fuzzy boundaries between measures and outcomes
Discriminant validity
Is the measure weakly related (or not at all) with measures of a different construct?
ex: Extraversion -> Eye color (not compelling)
Internal validity
Can this study support a causal conclusion?
Does X and only X produce the observed change?
the degree of confidence that a study's results truly show a cause-and-effect relationship between the independent variable and the dependent variable, rather than being the result of other unintended factors
Experiments have strong internal validity
External validity
The degree to which the results generalize to people and contexts besides those in the study (generalizability)
Does the claim generalize across the dimensions of who, what, where, and
when?
E.g., Claim: “Caffeine improves memory”
• Measure: cued recall test (good construct validity for memory)
• Study: College students, word pairs, 20-minute delay
Would it generalize to...
• older adults?
• children?
• face recognition?
• eyewitness memory?
• exams?
• two weeks later?
How to create good self reported measure? (survey methods
Good = reliable and valid: influenced by how you create and how you administer
How do you ask the question (survey methods)?
Understandable: construct validity compromised if people don’t understand the question
Simple questions (why?)
misunderstanding lowers construct validity
technical wording increases error
literacy levels vary
Overall: Clear wording is not “dumbing down.” It is better measurement.
Use unbiased questions
Framing effect
The frame in which objective information is
communicated can change how people respond
Common kinds of frames
Gain/loss (promotion/prevention)
What is the likelihood you’ll remember? What is the likelihood you’ll forget?
E.g., health messages, flossing, etc.
Anchor points
People rely to heavily on the first info they are receiving
Out of 1,000 people, how many experience spiritual
struggles?
What is problem with order of questions (Framing)
Order of questions can bias subsequent responses
Earlier questions can change later answers
ex: Forced choice items followed by open-ended items
What was your most significant spiritual struggle: Moral or Ultimate Meaning?
Please describe your most significant spiritual struggle:
Order of response options can bias responses
Each year, do more people join religion or leave religion?
Solution for ordering and framing effect
Frame or order questions/options more than one way and see if the outcomes are the same
Pros and cons: open ended questions
Pros: more information, richer information, not constrained by researcher response options
Cons: require coding, length/detail desired may not be clear to participant, recall memory is worse than recognition memory
List your favorite books (requires recall) vs. choose your favorite books from this list (recognition)
Pros and con: close ended
Closed-ended:
Pros: Clearer response options, quicker to complete and score
Cons: May not include all options participants would report, may be influenced by framing effects
Observation focuses on
Systematic recording of observable behavior
what people do — not only what they say they do
performance in tasks or tests — often speed or accuracy
behavioral traces or outcomes — products of behavior
ex: Helping, texting, blushing, arguing, hand-washing, littering, free throws,
reaction time
How to Collect Observational Measures
Three design questions
What behavior can be observed?
Where will it be observed?
How might observation influence behavior or recording?
Bottom line: Observation is not automatically objective. It has to be designed
Naturalistic Observation (what is being observed)
Observe and record behavior in its natural environment without intrusion.
Examples
How do children form cliques on a playground?
Is cell-phone use more common on highways or city streets?
Who washes hands more after using the restroom?
Advantage: behavior observed in an environment it occurs in
Laboratory Observation (what is being observed)
Observe and record behavior in a controlled environment.
The researcher controls more of the situation:
who participates
what task they complete
what is recorded
when and how often behavior occurs
Key tradeoff
More control, but potentially less natural behavior.
Lab and naturalistic observation combined
Field experiments create a contrived situation in the real world.
Examples
Helping behavior after staged subway accidents
Hospital hand-washing reminders tested in real settings
Public behavior after signs, prompts, or environmental changes
Key idea:
Field studies combine ecological realism with some experimental control.
Pros and cons of Naturalistic observation
Behavior in the wild
Less artificial
Less control over who, what, and when
Pros and cons of Laboratory observation
More control over task, timing, and recording
More options for cameras, physiology, reaction time, and coding
May alter real behavior
How observation influence data
Observer bias
observer effects
Observer bias (how observation influence data)
Expectations influence what observers notice, record, or interpret.
Observer effects (how observation influence data)
Participants may behave differently because they are being watched.
Solutions to reducing Observer Bias
Possible solutions
Use blind observers or coders.
Develop clear coding rubrics.
Use more than one observer.
Check interrater reliability.
Video record behavior and code later.
Takeaway
Objectivity is designed into the method; it does not happen automatically.
Reactivity
Reactivity occurs when participants change behavior because they know they are
being observed.
Solutions to reducing reactivity
Unobtrusive observation
One-way mirrors or hidden recording when ethical
Habituation: wait until people adjust
Measure behavioral traces instead of observing directly
ex: Litter in public spaces, beer bottles in trash, parenting in lab settings
Bivariate correlation studies
Examine the relationship between two measured variables in the same participants.
measured not manipulated
Multivariate analysis
several variables
Methods for studying association questions
Bivariate correlation: two variables
Multivariate analysis: several variables
Basic design for bivariate coorelation
One group of participants
Measure X
Measure Y
Estimate their association
Two parts of correlation
Strength and direction (pos, neg and no association)
Positive association (Direction of an Association)
Higher values of one variable tend to occur with higher values of the other.
Effective study strategies ↔ learning
Social support ↔ life satisfaction
Important: direction and strength are independent
• -1 is as strong as 1 (sign = direction; absolute value = strength)
• 0 means no linear relationship (a curved association may remain)
Negative association (direction of an association)
Higher values of one variable tend to occur with lower values of the other.
Absenteeism ↔ grades
Stress ↔ sleep quality
Important: direction and strength are independent
• -1 is as strong as 1 (sign = direction; absolute value = strength)
• 0 means no linear relationship (a curved association may remain)
No association (direction of an association)
Knowing one variable does not help predict the other.
Important: direction and strength are independent
• -1 is as strong as 1 (sign = direction; absolute value = strength)
• 0 means no linear relationship (a curved association may remain)
Strength of a correlation
A statistic indicating the strength of the linear relationship between variables
• Ranges from -1.00 to 1.00
• Cohen’s (1992) guidelines for strength:
• .10/-.10 (weak), -.30/.30 (moderate), .50/-.50 (strong)
• Newer guidelines from Gignac & Szodorai (2016):
• .10/-.10 (weak), -.20/.20 (moderate), .30/-.30 (strong)
Important: direction and strength are independent
• -1 is as strong as 1 (sign = direction; absolute value = strength)
• 0 means no linear relationship (a curved association may remain)
Dichotomous (measured value categorical)
exactly 2 categories
Ordered Categorical (measured value categorical)
categories have a meaningful order
Multivariate studies include:
one outcome variable
two or more measured predictors
no manipulated variables
no random assignment
Allows for Moderation and Multiple Regression analyses
research designs that involve measuring and analyzing three or more variables at the same time to understand complex relationships
Moderating variable
a third variable that changes the strength, direction, or presence of the relationship between an independent (predictor) variable and a dependent (outcome) variable
A moderator changes the X–Y slope or pattern; it is not simply another
predictor of Y.
Regression variable
a data factor or element used in a statistical model to analyze or predict an outcome
what remains after measured adjustment?
Regression estimates the association between X and Y (moderator variable) conditional on included measured Z. It does not assign X at random or identify every plausible Z.
Common ways of reporting multiple regression
“when researchers controlled for…”
“even after accounting for…”
“above and beyond…”
Common ways of reporting moderation
“relationship was weaker for…”
“the association was the same regardless of…”