PSY3213 Research Methods Final Exam

Example of Quasi-Experimental Design

In a study of the effectiveness of a new management technique, one branch office is given a new method to implement, and another branch office retains the traditional management policy.

At the end of the year, the new management branch has higher quality of production than the one that retained the traditional management technique.

Most likely a quasi-experiment. Random assignment to two classes was probably not made, so groups may not be equivalent. Possibility of results being confounded.

Quasi-Experimental Design

describes non-randomly assigned participants and controls subjects to impact assessment

Contracted comparisons

-Most common QED design

-Matching participants and comparisons

-Statistical adjustments

Quasi-Experimental Design

"research procedure in which the scientist must select subjects for different conditions from preexisting groups"

True Experiment

-random assignment of subjects to condition

-Manipulate IV

-Measure DV

-Control allows ruling out of alternative hypotheses

Quasi-Experiment

-Selection of subjects for the conditions

-Predictor = IV

-Criterion = DV

-Don't know whether differences are caused by IV or differences in subjects

Independent variable =

predictor

Dependent variable =

criterion

Quasi-Experiment Usage

Random assignment not within evaluator's capability

Powerful stakeholders oppose randomization

Non-Experimental Research

-Shows cause and effect

-Studies variables as they exist

Non-Experimental Research

"any quantitative study without manipulation of treatments or random assignments"

Purpose

Descriptive

Predictive

Explanatory

Descriptive

-Primary focus for research is to describe some phenomenon or to document its characteristics

-Establish status-quo or norm

Predictive

Primary focus for research is to predict some variable of interest (criterion) using info from other variables (predictors)

Explanatory

Primary focus for research is to explain how some phenomenon works/why it operates

Cross-sectional

Data collected at one point in time, often in order to make comparisons across different types of respondents/participants

Prospective (longitudinal)

Data collected on multiple occasions starting with present and going into future for comparisons across time

Retrospective

Look back in time using existing or available data to explain an existing occurrence

Requirements for causality

1) correlation

2) temporal precedence

3) there is no other possible alternative explanation for observed relationship

*first/second requirement fairly easy to meet

*third requirement much more difficult

Correlation

variables must be related

Temporal precedence

changes in X must occur before change in Y

Baseline

first 10 days

Treatment

second 10 days

Reversal

remove treatment; last 5 days

Issues with QEDs

-QEDs sometimes come from botched randomized experiment

-Program staff subvert randomization process --> assigning only those random subjects that will yield good results

-Attrition from treatment

-Problems with data collection

"Validity of research"

refers to the degree to which conclusions are accurate and generalizable

Threats to validity

-Both experimental and quasi-experimental research are subject to threats to validity

-If threats aren't controlled for, they may introduce error into study, which leads to misleading conclusions

Internal validity

extent to which differences on the DV are a direct result of manipulation of IV

reduced due to presence of controlled/confounded variables

History

when factors other than treatment can exert influence over results; problematic over time

Maturation

when changes occur in DV that may be due to natural developmental changes; problematic over time

Testing

pretest may give clues to treatment or posttest and may result in improved posttest scores

Instrumentation

nature of outcome measure has changed

Regression

tendency of extreme scores to be nearer to the mean at retest

Differential selection of participants

participants are not selected/assigned randomly

Experimental treatment diffusion

control conditions receive experimental treatment

If experimental setting more closely replicates the setting of interest, ______ can be higher than a true experiment run in a controlled lab setting

external validity

Threats to external validity

selection bias

Selection bias

if sample is biased, you cannot generalize to population

Reactive effects

-experimental setting

testing

-multiple-treatment inference

Experimental setting

differs from natural setting

Testing

pretest influences how subjects respond to the treatment

Multiple-treatment inference

if subjects are exposed to more than one treatment, findings could only be generalized to individuals exposed to same treatments in same order of presentation

Types of QEDs

-Posttest-only design with nonequivalent groups

-Pretest-posttest design with nonequivalent groups

-Mixed factorial design with one non-manipulated variable

-Interrupted time series design

-Regression-discontinuity design

Posttest-only Design with Nonequivalent Groups

X1 O

X2 O

-Uses two groups from same population

-Questions must be addressed regarding equivalency of groups prior to introduction of treatment

Pretest-posttest Design with Nonequivalent Groups

O X1 O

O X2 O

-Stronger design --> pretest may be used to establish group equivalency

Mixed Factorial Design with One Non-Manipulated Variable

Example:

-Experiment on pain perception (effect of caffeine, expected differences between men/women)

Protocol:

-25 men/25 women

-Each takes part in two sessions, one week apart

-One session: drink coffee (decaf) and put hand in ice water until pain

-Other session: drink coffee (caffeinated) and put hand in ice water until pain

Between-subjects variable: male/female

Within-subjects variable: caffeine intake

Interrupted Time Series Design

-Variant of pre-test-post-test design

-Set of measurements taken at intervals over a period of time

Regression-Discontinuity Design

-Pre-test/post-test design variant

-Unique characteristic

-Method (cutoff scores) by which participants are assigned to conditions

-Participants at rick are targeted

QED Matching

-participants sought first, comparisons matched afterwards

-matching info often sought in published literature

-attend to variables that are potentially related to self-selection processes

-use only as many variables for matching as are necessary

What is matching based on?

Prior knowledge and theoretical understanding of social processes in question

Pertinent characteristics for matching will tend to be ___ and, therefore, somewhat ___

intercorrelated; redundant

Characteristics of individuals

age, sex, educational attainment, SES, ethnicity

Characteristics of families

life-cycle stage, number of members, number of children

Characteristics of organized units

size differentiation, levels of authority, growth rate, budget

Characteristics of communities

population size, territorial size, industry mix, governmental organization

Options for matching procedures either ___ or ___ ____

individual; aggregate matching

Individual matching

draw a "partner" for each participant from unexposed pool

-usually preferable

-more expensive/time-consuming

-difficult to execute for a large number of matched variables

Aggregate matching

overall distributions in participant and control groups on each matching variable are made to correspond

Statistical procedures

now generally used in both ex ante and ex post quasi-experimental evaluations as primary approach to dealing with selection bias

Multivariate statistical methods

commonly used to adjust for a number of contaminating variables simultaneously

Statistical controls

processing some superior qualities arising from retention of observations that might have to be discarded under matching procedures

Matched and ___ controls are equivalent ways of proceeding

statistical

Multivariate statistical models

-allows for creation of statistical model to account for initial measurement difference b/w intervention and comparison groups

-adjusts outcome difference b/w those groups to subtract portion attributable to those initial differences

-whatever difference on outcomes remains after subtraction, it's interpreted as net effect of intervention

Need also to control for variables dealing with ___ of individuals into ___ vs. ___ group

selection; intervention; control

Examples of control variables

proximity of individuals to program site, motivation to enroll in program, etc.

Variables related to ___ are only useful if they also relate to ___

selection; outcome

Generic controls examples

age, sex, income, occupation, race

Generic controls

distribution of certain characteristics and processes (e.g., birth rates, sex ratios, etc.) and derivatives of these measures

What are the best examples of generic controls from?

epidemiological studies

-used successfully because selection processes are either known or unimportant

Why are generic controls tempting?

low cost and limited time to collect data

Large N

-participants are grouped

-data represented as averages

-replicability determined based on statistical significance

Small N

-each participant represents their own experiment, often in repeated measures design

-individuals' data are presented

-replicability determined by repeating experiment in another subject

Experimental Analysis of Behavior Approach (B.F. Skinner)

-analysis of behavior can lead to intervention

-studying capabilities and limitations in very small populations

Gustav Fechner (1860s)

explored sensory processes on an in-depth basis with a series of individuals

Hermann Ebbinghaus

pioneered research in verbal learning/memory, using himself as subject in single-case design

When would small-n designs be used?

-when research is extremely time-consuming/expensive

-requires extensive training or has difficulties with control

-when a single negative instance would refute a theory or an assumed universal relation

Reinforcement always increases ___

responding

Corpus callosum

-connects hemispheres

-important b/c brain is lateralized in processing and function, so hemispheres need to talk to each other

Severe, intractable epilepsy

-seizures spread from one hemisphere to other across corpus callous

-patients have c.c. severed (split brain), so seizure activity can't spread to other hemisphere

-generally function normally, with some exceptions noticeable on careful inspection (e.g., presenting certain images to only one eye/nostril, etc.)

What hemisphere does the left visual world go to?

right hemisphere

What hemisphere does the right visual world go to?

left hemisphere

Split brain patients have to shift eyes back and forth quickly for both ___ to get both ___ fields

hemispheres; visual

Experiment on split brain patients

-present two different images to the two eyes, too quickly for eyes to turn

-picture presented in left visual field goes to right hemisphere; pic in right visual field goes to left hemisphere

-results: could draw what they saw with right hand

-could not report what they saw in left v.f.

-could report what they saw in right v.f.

Disadvantages of split brain experiment

-are split brain people representative of all people?

-question of external validity

-cant manipulate variable in humans

-can you draw conclusions? can you use patterns and triangulate data?

Henry Gustav Molaison (HM)

studies on single patient led to huge advancements on our understanding of the role of hippocampus in learning/memory

Small-N Designs for Behavior Change Studies

stable-baseline design

multiple-baseline design

reversal design

-stable baseline is important to ensure that, if a change occurs after intervention, the change is due to intervention

Stable-baseline design

many baseline measurements --> intervention --> more measurements

baseline measurements of memory in an Alzheimer's patient --> taught a new memory strategy --> more memory measurements

-turns out memory improves

-stable baseline suggests it was due to intervention

Multiple-baseline design

baseline measure --> intervention --> measurement

add new component to baseline measure --> intervention --> measurement

ex: baseline measure for touching face --> overcorrection --> measurement

Variations of Multiple-Baseline Design

1) same person and behavior in multiple situations (home, work, school)

2) different people in same location, but starting at different times

Reversal design

baseline measure --> intervention --> measure --> removal of treatment --> measure

Reversal Design (A-B-A)

-teaching a 7-year-old boy to increase social interaction

-sessions 1 and 7 represent baseline periods (no cuing from facilitator)

-sessions 2-6 used cuing to prompt interaction

-at 7, the cuing (treatment) is gone (the second "A" in A-B-A), and the behavior decreases some, but not to as low as it was

Reversal Design (A-B-A-B)

talking-out behavior in a mentally disabled student

-difference b/w baseline and treatment sessions is dramatic --> treatment IS EFFECTIVE

-increased productive behavior was CAUSED BY contingent attention

Ethical Considerations

is it ethical to remove a treatment that's working?

is it ethical to use a treatment that hasn't been empirically proven to work?

Piaget

observations of own three kids informed his theories of childhood cognitive development

Object permanence

shape can change, but volume does not

Ebbinghaus' Memory Studies

-memorized long lists of nonsense syllables

-varied how many and studied time

-forgetting curve

Forgetting curve

memory declined quickly over first hour, then more slowly

Ericsson et al Memory Studies

-on a single college student over 1.5 years

-learned to memorize up to 79 digits

-identified method he used to remember, disrupted that, and memorization went back to baseline

-implications for practice, motivation, memory, expertise, and drew cause/effect relations

External validity in small-n designs

-replicate on other, similar participants

-study problem experimentally in animal models

-use other techniques in a larger population

-no expectation that results will generalize to everyone

-sometimes no need for generalization at all

Construct validity in small-n designs

-is measure valid?

-is it reliable? --> be sure to check inter-rater reliability for behavioral observation