COGS 14a Final - UCSD

Experiment

- researcher manipulates IV to determine differences in DV among EQUIVALENT groups of participants

- yields causal information

- two types: between group and within group

Quasi Experiment

- groups of participants are NOT EQUIVALENT (i.e. participants were not randomly assigned; seeking parents in oregon versus california)

- MAY yield causal information when all confounds are eliminated

Correlational study

- effect of subject variable on dependent measure

- does NOT yield causal information, instead identifies relationship between SV and DV

example between experiment and correlational study

if you were to measure reaction time at a party between drunk and not drunk people, it would be a CS and not an experiment because they weren't randomly assigned, they assigned themselves and could be influenced by other variables.

If you took a group of people and randomly divided them into two groups, with one drinking juice with alcohol and one drinking juice that just tasted like alcohol, and a difference in reaction time was recorded, then that would yield causal data and be a experiment as they were randomly assigned.

placebo

an inert treatment with no effect on the DV

control group

- the more "natural condition"

- the IV is not present

requirements for an experiment to yield casual information:

1. must be equivalent groups before introduction of IV

2. IV must be introduced before DV is measured, also known as "temporal priority"

3. design must be free of any confounds

equivalent groups

-random assignment of participants

-avoids selection bias

- usually obtained via random assignment

between-groups research design

- manipulates a variable between subjects

- i.e. one group gets juice with alcohol, one group gets juice without alcohol

- example: a two group experiment

- also called between subjects research design

benefits of between group research design

- equivalent groups are only exposed to one level of IV so causal effects and comparison are straightforward

- participating in only one condition lessens chance of subject attrition

- may be only option when impractical to run multiple conditions on one participant (i.e. having one person take several trial drugs for a curable disease)

disadvantages of between group research design

- you need an entire group of participants for each condition

- demand characteristics may be different for each group

- separating groups may lead to inequal treatment

- participants in groups may be too different; subject variables like age or gender may vary too much between groups which leads to larger error variance

random assignment

- when participants are randomly assigned to one of two groups, each participant is equally likely to be assigned to either group, so anything that could effect the DV is equally distributed

-yields best results with larger samples

-allows for equivalent groups

what does it mean if an IV is a subject variable?

-example: age ---> one can't bring in a bunch of two year olds and randomly assign them to be two or six years old, it is something that cannot be manipulated

- this means we cannot randomly assign, so we cannot say an SV is yielding a casual relationship

- better suited for cross sectional design

selection bias

- increases risks of differences between groups

-explains why researchers pick who takes part in the experiment but NOT which group (experimental vs control) they are in.

confounds

-uncontrolled/extraneous variable that influence the DV and damages integrity of the experiment/interferes with results

- damages internal validity

- related to the IV

-experiments must determine that the IV and not a confound is influencing the DV

-increases error variance

extraneous variable

variables that can affect the DV----> makes it so we have a confound and can't conclude the IV impacts the DV

error variance

the statistical variability of scores from variables other than the independent variable

- random

- the greater the error variance, the harder to identify consistent differences between groups

- ratio of between group variance is to within group variance is smaller (more differences within groups than between them) is smaller when the error variance is greater

internal validity

the extent to which we can ensure the IV influenced the DV and not other variables

- an internally valid study has no extraneous variables that would confound the results

possible confounds in between group variables

1. Experimenter bias

2. demand characteristics

3. instrumentation effects

4. comparable treatment of groups

5. subject attrition

6. sensitivity of DV

experimenter bias

the experimenter, unconsciously or consciously acts in a way that biases the results

- ways to limit experimenter bias and demand characteristics are single blind procedure and double blind procedure.

demand characteristics

participants guess what the study is about and act in a way to meet assumed predictions

-ways to limit experimenter bias and demand characteristics are single blind procedure and double blind procedure

instrumentation effects

any changes in the DV caused by changes in the measuring instrument used (e.g. wear and tear from use)

Comparable treatment of groups

both of the two groups must go through comparable experiment procedure. if one group gets better treatment, it isn't comparable treatment and thus a confound

subject attrition

subject quitting partway through the experiment

- Non-systematic attrition: attrition is random and evenly distributed among groups (not due to experiment)---> does not threaten internal validity

- Systematic attrition: subjects who quit are unevenly distributed among the groups ----> does threaten internal validity

sensitivity of DV

the testing scale is too hard or too easy so there is a disproportionate amount of maximum/minimum scores and true ability cannot be determined.

- floor/ceiling effect

single-blind procedure

either the experimenter or the participants ignore which part of the study they are in ---> avoids bias

double-blind procedure

neither the experimenter nor the subject know what group they are in ---> avoids bias

within subjects design

each participant experiences each level of the IV at least once

- two types are pretest posttest design and the repeated measures design

pretest posttest design

- one group of participants is tested two times using the same measuring tool, once before and once after the IV is manipulated in some way

-example: when studying how exercise affects mood, the pretest would be mood before the workout, and the posttest would be mood after the workout

repeated measures design

-similar to pretest postest but measures the same participant multiple times-- not just before and after the implementation of the variable.

- common example of this design is the longitudinal design

longitudinal design

testing participants several times while looking for changes that occur over time. duration can be months, years, or even decades

- example: observing vocabulary scores during infant. toddler, childhood, and teenage years

within subjects design benefits

- requires fewer participants than between-group designs

- less time

- subject variables remain constant across experimental conditions (i.e. we compare a two year old at different levels instead of a two year old to a six year old)

- Low level of error variance, so the experiment has more power

within subjects design disadvantages

susceptible to confounds caused by:

- demand characteristics

- carryover effects (order effects)

- history effects

- maturation effects

- testing effects

demand characteristics

participants guess what the study is about and try to meet the expectations

order effect (carryover effect)

performance on earlier trials might affect performance on later trials

-types are practice effect and fatigue effect

(order effect) practice effect

as experiment progresses, performance gets better

(order effect) fatigue effect

as experiment progresses, performance gets worse

what is the solution to the order effect?

counterbalancing: presenting the experimental conditions to participants in different orders so the carryover effects can be controlled

- these are divided into Complete WS Design and Incomplete WS design

Complete with subject design

all subjects experience each experimental condition several times until they have experienced all possible combinations of the conditions

- if there are 2 conditions ---> ABBA balancing (good for linear practice effects, bad for nonlinear practice effects)

- if there are 3+ conditions ---> block randomization (ABC, CBA, BCA, BAC, CAB, ACB) (not good if time commitment for each block is too long)

Incomplete with subject design

each participant receives a unique order of conditions at least once but do not receive all possible sequences of the condition

- random order rotation (ABCD, DCBA, BACD, CADB, etc)

-balanced latin square ---> helps counteract fatigue effects

history effect

an event that occurs outside of the experiment at the same time the independent variable is being changed

- example: in a longitudinal study, someone catches an illness that effects their results on test scores over time

maturation effect

any change in behavior caused by the passage of time (subjects getting older, progression of disease, etc)

testing effect

when subjects are tested repeatedly, previous tests are likely to effect their performance on their next tests.

- example: when students take SATS the second time, even if they study the same they usually do better because they are more familiar with the tests

Pre-experimental (NON-EXPERIMENTAL) designs

- little to no control over confounds

- no random assignment

- many alternative explanations for observed effect

- NOT EXPERIMENTS

pre-experimental design examples:

Two types: pretest posttest design, non equivalent control group design

- before and after weight loss campaigns

- often seen in pre-test and post-test; before drugs, the person is happy ---> after drugs, the person is sad

Quasi-Experiment design

studies that manipulate an IV, but for ethical or practical reasons equivalent groups and random assignment is not possible (no equivalent groups/random assignment)

- not the most ideal but often the closest to a true experiment one can get

Types of Quasi-Experimental Designs

- non-equivalent control group design

- time-series design

- multiple time-series design

non-equivalent control group design

comparing the experimental group with a comparable -- but not equivalent/randomly assigned-- control group before and after treatment

- example: if studying infant mortality in CA, using parents from Oregon may serve as a comparable but not equivalent group

- on a graph, there are two lines if the two lines are near parallel or parallel, there is no effect. if the two lines diverge, there is a suggested effect.

time-series design

observes one group multiple times before and after treatment when there is no comparable group that would suffice as an equivalent

- example: make multiple observations before and after the program to decrease infant mortality in CA families

- on a graph, there is one line. after the treatment is indicated on the graph, if the line is continuing in the same direction, no change is indicated. if it gets steeper or shallower, the results suggest a change

Multiple time-series design

observes two non-equivalent groups multiple times before and after a treatment

- example: make multiple observations before and after the implementation of the infant mortality program in both families in CA and Oregon

- on a graph, there are two lines. After the indication of treatment, if any of the lines diverge or their is a change in direction, it suggests a change. If they stay the same, no change is indicated.

threats to internal validity of quasi-experiments

- history effects

- maturation effects

- testing effects

- instrumentation effects

- subject attrition

- selection bias

- interaction of selection with other threats

--- all effects but subject attrition can be partially accounted for by control groups

history effect

change other than the experimental treatment occurs during the course of an experiment ---> affects the dependent variable

- can be controlled with time series design, multiple time series design

maturation effect

naturally occurring changes occur within the subjects from passage of time ----> could be responsible for the observed results

- can be controlled with nonequiv. control group, multiple time series design

testing effect

better performance because of taking the test several times, influences effect on DV

- can be controlled by using nonequiv. control group or multiple time series design

subject attriton/mortality

subject quits participating

- has no option to try to control, researchers must watch out

selection bias

when there are differences between the experimental groups in a study

- can be controlled using time series design

interaction of selection with other threats

when extraneous variables effect one group and not the other

- example: measuring anxiety levels in schools, and a fire breaks out in one school but not the other

- can be controlled using time series design

factors

another word for independent variables

factorial design

design that assesses effect of two or more IV (factors) on a DV

- each variable has 2 or more levels

- tests several hypothesis (main effects, interaction effects) simultaneously

main effect

the effect each factor (IV) ALONE has on the DV

- analyzed using 2 way ANOVA

- example: feeback on the confidence rating/ test type on confidence rating

interaction effect

how one level of the IV effects another level of the IV

- analyzed using 2 way ANOVA

- example: negative feedback would harm one more on a math test than a reading test

factorial (2-way) ANOVA

When testing n-number of factors:

- Factorial within-subjects ANOVA (2-way factorial within-subjects ANOVA)

- Factorial between-subjects ANOVA

- Mixed-subjects ANOVA

- H0: the mean of the different conditions do not differ

CALCULATE W SQUARE

marginal mean

the average in the margin

- helps us calculate the main effect

how to tell if there is an interaction on a factorial design graph

if the lines are nearly parallel, there is no interaction.

if the lines are different, in direction or trajectory (crossover), then an interaction is suggested

other types of two factor design

- both between group factors

- both within group factors

- one between group, one within group (mixed design or split-plot design)

what are the differences in choosing other types of two factor design?

- error variance

- number of participants

- potential confounds

- these are all usually reflected using 2 way ANOVA, but sometimes other formulas better suit the difference in designs

other higher order designs

- three factor designs (eg ABC)

--example: three IVs, one DV (DV measures performance)

----a) type of test (logic, math, reading)

----b) time of test (morning, afternoon, evening)

----c) type of major (STEM, humanities, art)

-------> this example is a 2 x 2 x3 design

- statistically analyzed via 3 way ANOVA

- increasing factors increases complexity of interpretation

other higher order designs (cont)

7 effect design:

- 3 main effects (A, B and C)

- 3 two-way interaction effects (how AB interact, AC interact, etc)

- 1 three-way interaction effect (how ABC interact)

- statistically analyzed via 3 way ANOVA

- increasing factors increases complexity of interpretation

higher order designs (cont)

- appealing because of their flexibility, efficiency and elegance

- adding factors to a design should always be studied very carefully

correlational studies

- does NOT study how IV impacts DV, instead studies how SV influences DV

- does NOT yield causal information

- aims to identify: is there a relationship between X and Y

analyzing correlation

is there a relationship between X and Y? ---> correlation coefficent

If so, how does Y change with changes in X ----> regression line

bi-variate correlation

- USE PEARSON R

- if r>0, there is a positive correlation

- if r<0, there is a negative correlation

- if r~0, there is no correlation

multiple correlation

relationship between a variable and multiple other variables

- multiple correlation coefficient

- always between 1 and 0

- indicates strength of relationship

when do we use a regression line?

when bivariate distribution shows a LINEAR relationship

correlational studies with multiple variables

Multiple correlation:

- relationship between one score and larger number of scores

- multiple correlation coefficient (values between 0 and 1)

Multiple regression

- prediction of Y from multiple X points

- assessment of relative importance of various predictors

why do we use observational studies?

1. good starting point

2. generalization of lab findings

3. can be only method for studying phenomenon

different types of observational studies

1. Naturalistic observation: there is no interference, the researcher only watches

2. Researcher participation: researcher is an active participant in situation

naturalistic observation

overt observations have two methods:

1. desensitization: gradually moves closer so they can sit closer to participants (eg watching wild animals)

2. habituation: appears in environment until presence is no longer an issue

systematic naturalistic observation

focuses on specific behaviors/settings

participant observation

disguised and undisguised observation

- researcher must be aware of bias, so making relevant terms beforehand helps this

problems with observational studies

- don't allow for establishing causal relationships-- except for field experiments

- hawthorne effect: participants change their behavior because they know they are in a study

- reactivity measures: participants change behavior because they know they are being watched

- expectancy effects: the experimenter has a biased perception of actions due to their own unconscious preferences (can be combatted with naïve observers or recording devices)

what is a field experiment?

an experiment that takes place in a natural setting

what are advantages of a field experiement?

- great external validity (experiment results are operational to actual situation)

- strong ecological validity

what are disadvantages of a field experiment?

- internal validity is low as it is difficult to account for confounds

- ethical considerations as participants cannot always provide informed consent

when should i use each type of observational study?

1. ecological validity: naturalistic observation > field experiment > lab experiment

2. controlling for variables: lab study > field experiment > naturalistic observation

3. whatever is best given ethical/practical circumstances

what is data collection?

- deciding what phenomenon will be observed and how

- collecting data in a way that is both reliable and valid

reliability in data collection

- measurement procedure yields consistent results

- interobserver reliability; determined through formulas like % agreement/disagreement, ordered data, interval/ratio data, etc

techniques for data collection

- narrative records

- checklists

- tech-mediated advanced methods

narrative records

running records of behavior in a given situation

- can be complete or sketchy

- obtained with naked eye observations or audio/visual recordings

- provide much information, but can be hard to organize it. that is why we use data reduction (through transcription and coding), apply reliability criteria

checklists

lists used to guide observations on what type of behavior/phenomenon to look out for

- useful when there is a focus on a few specific behaviors

- must use operational definitions to avoid subjective interpretations

- divided into two types: static and active checklists

static checklist

records observations that will not change during the observation

- example: demographics; ie the location of somewhere, gender of participants, etc

action checklist

records status of specific features over time

example: children's behavior over series of weeks

narrative records versus checklists

narrative records:

pro- no restrictions of pre-defined behaviors

con- data reduction (eg by coding) is usually needed

checklists:

pro- no need of data reduction

con- focuses on relatively small subset of behavior

types of data collected

-frequency (how often)

- duration

- timing

- ratings

sampling techiques

natural settings cannot be observed at all times and at all occurences, so we need to sample in order to effectively measure our question

behavior sampling

measures a subset of behaviors

time sampling: measures every interval of time, good when behavior happens frequently/continuously

- example: traffic times during rush hour versus average rate of traffic

event sampling: measures on the occurence of a more rare event, good when the behavior is less frequent/rare

example: animals fleeing in a wildfire

situation sampling

observations of an operationally defined situation are made in different settings and circumstances

- example: children's behavior on playgrounds are measured in several different neighborhoods

what are examples of technology-mediated advanced methods?

- HD cameras/recording

- behavior elicitation eg gesture production

- multi layer naive coding

example: time concepts in the Yupno tribes

why the Yupno tribe?

- topographic relations central to cosmology and everyday language

- ADD MORE

questionaire

a written set of questions to learn about an individual

- not meant to be aggregated