Lecture 21: non-experimental and quasi-experimental designs

fill the blanks

why would you use a non-experimental or a quasi-experimental design instead of an experimental design?

• an experimental design involves manipulating the IV and measure its effect on the DV

• but sometimes, we cannot manipulate the IV and must work with pre-existing levels of the IV → quasi or non

how are groups defined in a quasi or non-experimental design? (2)

• by time

• by participant characteristics: age, degree, gender

why is internal validity a problem when you’re using quasi- or non-experimental designs?

because groups can differ on other factors

define “quasi-independent variable”

variable used to differentiate groups of participants being compared

what’s the difference between an independent variable and a quasi-independent variable?

• IV: variable being manipulated by the experimenter

• quasi-IV: variable used to differentiate the groups being compared (define the groups)

what are the similarities between quasi and non-experimental designs? (2)

• no direct manipulation of the IV

• no random assignment

what’s the difference between quasi and non-experimental designs?

• non-experimental: doesn’t try to control threats to internal validity (confound), no control conditions

• quasi-experimental: tries to control threats to internal validity (confound), control conditions

what are the general designs that fall under quasi- and non-experimental research? (3)

• between-groups

• within-groups

• developmental research design (mostly with kids)

*graph is just ordered differently

what are the between-groups designs? (3)

• differential design: compare pre-existing groups

• post-test only NEG design: compare two non-equivalent groups (treatment VS control)

  • group 1: treatment, measure

  • group 2: no treatment, measure

• pre-test/post-test NEG design: compare two non-equivalent groups by adding a pre-test (still treatment VS control)

  • group 1: measure, treatment, measure

  • group 2: measure, no treatment, measure

*measure = observe

define ”non-equivalent group design” (NEG)

the experimenter cannot control the individual differences in the groups, meaning that the groups are different (non-equivalent)

explain how we use non-equivalent group designs

• we use pre-existing groups (ex: different high schools, languages, etc)

• these groups are differentiated by one specific factor

• the purpose is to show that this factor is responsible for differences between group scores

what’s the problem with non-equivalent group design?

assignment bias: because your groups are non-equivalent (different), your groups will different on other variables than the IV, which means confounds

define “differential design”

compare pre-existing groups (individual differences) to see if they differ on the variable of interest

true or false: in a differential design, you manipulate and or control the assignment of participants to groups

false: you’re only comparing the participants based on pre-existing characteristics (individual differences)

define “post-test only NEG design”

• compare two non-equivalent groups of participant: one with the treatment and the other without (control)

• measure once only, after the treatment

true or false: post-test only NEG designs solve the assignment bias

false: we only predetermine who’s in which group depending on their personal characteristics

*assignment bias: groups will differ on variables other than the IV

define “pre-test/post-test NEG design”

• compare two non-equivalent groups (treatment VS control)

• measure once before the treatment and once after

why is pre-test/post-test NEG design quasi-experimental?

• it limits threats of internal validity (time threats only, which is why it’s quasi)

• you measure the two groups before one receives the treatment (pre-test)

• if they are similar, then the groups aren’t too different (and that if there is a change in the post-test, it might be caused by the treatment)

• doesn’t mean that the groups are equivalent, but it reduces assignment bias

*assignment bias: groups will differ on variables other than the IV

what’s the difference between the pre-test/post-test NEG design that is quasi-experimental and a true experimental design?

• pre/post NEG: by having a pre and post-test, you will control for one variable (ex: time)

• experimental: you can control for multiple variables

what are the threats to internal validity found in experimental, quasi-experimental designs? (5)

• history

• instrumentation

• testing effects

• maturation

• statistical regression

what are the within-groups designs? (2)

• one group pretest-posttest design: each individual in one group is measured once before the treatment and once after

• time series design: at least 3 measures are taken before and after the treatment

define “one-group pretest-posttest”

each individual in one group is measured once before the treatment and once after

why is the one-group pretest-posttest considered as a non-experimental design?

there is no control: you evaluate the treatment by comparing observations made before and after the treatment

true or false: time-related factors affect the one-group pretest-posttest design

true: anything before or after the treatment could cause the change

define “time series design”

at least 3 measures are taken before and after the treatment for each participants in one group

why is the time series design considered as quasi-experimental?

• there is an attempt to add control by making a series of observations for each participants

• this allows us to identify threats to internal validity since they should be measurable before the treatment (any pre-existing trend?)

what are the data features in a time series analysis? (2)

• level: absolute value, y-axis

• trend: relative value, slope

• → look at the slope before and after the treatment to see if there is a difference

in a time series design, when is an external event a threat to internal validity?

when the external event occurs at the same time as the treatment

• ex: measure anxiety before and after treatment, but won the lottery during the treatment…. are changes caused by the treatment or the lottery

define “developmental research designs”

designs used to examine changes in behaviour related to age

why are developmental research designs considered as non-experimental?

• they don’t directly manipulate the IV (developmental, how do you control age)

• instead, they compare natural differences between groups who differ in age

what are the developmental research designs? (3)

• cross-sectional: different groups of people of different ages (2+ groups)

• longitudinal: measuring one cohort over time (1 group, measured 2+ times)

• cross-sectional and longitudinal (2+ groups, measured 2+ times)

define “cross-sectional developmental research designs”

• between-group design that uses different groups for each age (or age bracket)

• the outcome variable is measured for each group at one point in time and compared with other groups

cross-sectional or longitudinal: when a study evaluates the differences related to age

cross-sectional

true or false: in a cross-sectional design, you should let some time elapse between your group comparison (measure your 20 year old in 2010 and your 30 years old 2020)

false: you want every groups at the same time point

what are the advantages (2) and disadvantages (2) of cross-sectional designs?

advantages:

• time efficient: between-group, don’t have to wait for your participants to grow older

• no long-term cooperation required

disadvantages:

• cohort or generational effects

• individual changes aren’t assessed

define “cohort/generational effects”

changes between cohorts caused by unique characteristics or experiences other than ages

define “longitudinal developmental research designs”

within-groups design that measures a variable in the same group of individuals (cohort) over time

what’s the treatment in a longitudinal design?

age

what are the advantages (2) and disadvantages (3) of longitudinal designs?

advantages:

• no cohort or generational effect

• individual changes are assessed

disadvantages:

• time-consuming and construction

• attrition can cause unbalanced measures

• risk of practice effects

define “cross-sectional longitudinal designs”

mixed developmental design: compare the results obtained from separate samples (cross-sectional) that were obtained at several times (longitudinal)

what’s the advantage of cross-sectional longitudinal design?

you can examine the development of behaviours due to mechanisms other than individual aging

what’s the difference between quasi- or non-experimental and experimental designs?

• quasi- or non-: no random assignment of participants to conditions

• experimental: random assignment required

associate the design and the procedure:

• differential design

• one group pretest-post test design

• post test only NEG design

• pretest/post test NEG design

• time series design

• O & O

• XO & o

• OXO & OO

• OXO

• OOOXOOO

quasi-experimental:

• pretest/post test NEG design: OXO & OO

• time series design: OOOXOOO

non-experimental:

• differential design: O & O

• one group pretest-post test design: OXO

• post-test only NEG design: XO & O

which design is between-subjects (3) and within-subjects (2)?

• differential design

• one group pretest-post test design

• post test only NEG design

• pretest/post test NEG design

• time series design

between-subjects:

• differential design: O & O

• post-test only NEG design: XO & O

• pretest/post test NEG design: OXO & OO

within-subjects:

• one group pretest-post test design: OXO

• time series design: OOOXOOO

which design is quasi-experimental (2) and non-experimental (3)?

• differential design

• one group pretest-post test design

• post test only NEG design

• pretest/post test NEG design

• time series design

quasi:

• pretest/post test NEG design: OXO & OO

• time series design: OOOXOOO

• → attempt of control by putting a measure before and after the treatment

non:

• differential design: O & O

• post-test only NEG design: XO & O

• one group pretest-post test design: OXO

  • *there is a measure before and after, but there is one group only, we can’t compare to anything else (we can only say that there is a change)