CHAPTER 8: DOING NONEXPERIMENTAL RESEARCH MAKING AND ASSESSING DIRECT OBSERVATIONS
Inter-rater reliability is the most important for observational research
Developing behavioral categories
Ex. predicting divorce: name-calling, bringing up things from the past, yelling, threats, etc.
Ex. 3rd grade: boys are more aggressive than girls. What are aggressive traits? Hitting, verbal attacks, taking things away, cursing at somebody, spitting at somebody, etc.
Must be very specific
Quantifying behavior in an observational study
Frequency Method
Counting how many times the behavior occurs
Duration Method
How long does the behavior last
Intervals Method
Divide into time intervals, and you mark yes or no whether the behavior occurs within the time interval
Recording Single Events or Behavior Sequences
Making Your Observations Live
Coding the Behaviors
SAMPLING STRATEGIES
Time Sampling
Sample a period of time (ex., recess) and observe during that time, and use another time to write down your observations
Individual Sampling
Assign people to observe (random)
Event Sampling
Often layered with time & individual sampling
No matter if you are assigned a person, if something really important or rare happens, you must record it
ESTABLISHING ACCURACY AND RELIABILITY OF OBSERVATIONS
Interrater reliability
Percent agreement
Frequencies - Valid percent 85.7 (must be 70 or higher)
Correlation coefficient
Correlation tells you if they were both in the same order, not whether they agree
Reliability correlation (will use the ones with a filled box, correlation=0.7)
Cohen’s Kappa
Intraclass correlation coefficient
BIAS IN OBSERVATIONAL RESEARCH
Observer bias
Observers need to keep their bias out (give clear definitions, train, and pilot testing)
Bias in the interpretation of what was observed
QUALITATIVE VS. QUANTITATIVE DATA
Quantitative: expressed numerically
Qualitative: cannot be expressed numerically (interviews, for example)
NONEXPERIMENTAL RESEARCH DESIGNS
Naturalistic Observations
Provides insight into behaviors in the real/natural world
Time-consuming & expensive
Ethnography
The researcher becomes a part of the group of participants being studied
Nonparticipant observation vs. participant observation (can become very biased, ethical?)
It may be difficult to gain access and entry to the group
Data is qualitative
Sociometry
Measuring interpersonal relationships within a group
Case History
Observe one or a few cases and report on these
Nonexperimental
Archival Research
Analyzing existing data/records
Nonexperimental
Content Analysis
Analyzing a written or spoken record
METANALYSIS
Statistical procedures that combine and compare results from multiple studies
3 Steps
Identify relevant variables
Locate relevant research to review
Conducting meta-analysis
2 observers: 20% watch the other people's
Data in 2 columns (dependent variable) = you average them or establish a main observer or make observers come to an agreement (video makes this easier)
Blind overver - don’t know the hypothesis, so they won't be biased
Key terms:
Behavioral categories
The general and specific classes of behavior to be observed in a study.
Ex: “On-task behavior” broken down into
“listening,” “taking notes,” and “raising hand.”
General and specific behavior
Interrater reliability
The degree to which multiple observers agree in their classification or quantification of behavior.
Ex: 2 observers watch a kid and measure how aggressive they are. If both observers got similar counts, interrater reliability is high
Cohen’s Kappa (K)
A popular statistic used to assess interrater reliability. It compares the observed proportion of agreement to the proportion of agreement that would be expected if agreement occurred purely by chance.
Intraclass correlation coefficient (rI)
A measure of agreement between observers that can be used when your observations are scaled on an interval or ratio scale of measurement.
Three therapists measure a patient’s knee flexibility in degrees.
The ICC measures consistency in numeric (interval/ratio) data from multiple raters. Here, the measurements are continuous numbers (degrees), so ICC is appropriate.
Quantitative data
Data collected that are represented by numbers that can be analyzed with widely available descriptive and inferential statistics.
Ex: Test scores, hours studied, or reaction times (using numbers)
Qualitative data
Data in which the values of a variable differ in kind (quality) rather than in amount.
Ex: descriptive, non-numerical data that differ in kind or quality, not amount (labels)
Naturalistic observation
Observational research technique in which subjects are observed in their natural environments without making any attempt to control or manipulate variables. The observers remain unobtrusive so that they do not interfere with the natural behaviors of the subjects being observed.
Ex: What would you do?
doesn’t manipulate variables and remains unobtrusive, just watching behavior in its natural setting,
Grounded theory
An approach for developing theory that is grounded in data systematically gathered and analyzed.
Ex: interviewing immingrants to create a new theory of cultural adaptation
The theory emerges from the data itself & not from prior assumptions
Ethnography
A nonquantitative technique used to study and describe the functioning of cultures through a study of social interactions and expressions between people and groups.
Ex: Going to a rural village to study traditions
Immersing oneself in a culture to understand its social interactions and customs
Participant observation
An observational research technique in which researchers insinuate themselves into a group to be studied and participates in the group’s activities.
Ex: researcher joins the environment that they are studying
The researcher actively participates in the group’s activities to gain an insider understanding
Nonparticipant observation
An observational research technique in which the observer attends group functions and records observations without participating in the group’s activities
Ex: watching students in a classroom
The researcher observes without direct involvement
Sociometry
A non-experimental research technique involving identifying and measuring interpersonal relationships within a group.
Ex: Students are asked to submit 3 classmates they would like to work with in a group
This measures interpersonal relationships and preferences within a group
Sociogram
A graphical representation of the pattern of interpersonal relationship choices.
Ex: word web with friends' names
A sociogram is a visual map of social connections, and arrows show patterns of friendship
Case history
A non-experimental descriptive research technique in which an individual case is studied intensively to uncover its history
Ex: A person in therapy
Intensive study of one individual’s experiences to understand causes and outcomes
Archival research
A non-experimental research strategy that involves studying existing records.
Ex: Adult therapist getting info about a patient from their child therapist
The researcher uses existing records or archives instead of collecting new data
Content analysis
A non-experimental research technique that is used to analyze a written or spoken record for the occurrence of specific categories of events, items, or behavior.
Ex: Analyzing how many times the word freedom comes up in a speech
The researcher systematically analyzes written or spoken text for specific words or themes
Information extraction
A specialized form of content analysis in which language processing software is used to analyze large databases of information.
Ex: Software scans thousands of news articles for mentions of “climate change.”
Meta-analysis
A statistics-based method of reviewing literature in a field that involves comparing or combining the results of related studies.
Ex: Combining 25 studies on mindfulness and stress reduction
Statistically combines results from multiple studies to find an overall effect