Class 4 - Factorial Designs & Single Subject Designs

Factorial designs have:

more than one independent variable (aka more than one factor or grouping variables)

A 2 factor design has how many IVs?

2 independent variables (regardless of number of levels of each factor)

A 2x3 factorial design means:

1 factor with 2 levels, 1 factor with 3 levels (ex: gender and grade level (3 grades))

A 3 factor design would have ____ IVs

3

A 2x2x2 design means that:

there are 3 factors, and each has 2 levels

What happens when you add complexity (i.e more factors and levels)?

when you add complexity, you need lots of participants to keep the same power

What is an example of a between subjects factorial design?

Do classical and rock singers show different rates of vocal polyps and does this vary with training or no voice training?
IV1: classical vs. rock
IV2: training vs. no training

What does a within subjects factorial design look like?

each participant is tested on three hearing aids immediately and after a month's trial
IV1: hearing aid type
IV2: time (two levels)

What is an example of mixed subjects factorial design?

Does the "Marvelous Metaphors" program improve children's comprehension of metaphors?
1 group gets program, 1 group does not (between factor)
test pretreatment, post-treatment, maintenance (within factor, 3 levels)
*this is a very standard treatment study

What are marginal means?

- means for each independent variable (averaged across cell)
- main effect (measuring main effect of each variable i.e the difference between levels of 1 independent variable)

How many marginal means in a 2x2 factorial design?

4

What is a cell mean?

- A cell mean is the mean of the scores in one cell (treatment) of a factorial design

- a cell is a unique combination of levels of factors (combinations of independent variables) AKA interaction effect

In a 2-way between subjects factorial design:

- there is different participants in each cell
- each measured once for DV
- 3 F-ratios (main effect of IV1, main effect of IV2 and interaction)

What is an example of a between subjects 2-way factorial design?

Does the program stream (SLP vs. AUD) and home province make a difference in satisfaction among SCSD students?
IV1: Program (2 levels) - main effect (SLP vs Audio - 2 means)
IV2: Home (Atlantic vs. ON/Q vs Western prov) - main effect (3 means - one for each province)
Interaction: SLP maritime, Audio maritime, SLP ON/Q, Audio ON/Q, SLP west, Audio west (6 means)
*Is program driving their satisfaction or being close to home?

What is an interaction effect?

- When effect of one IV depends on the effect of another IV (effect of one variable not constant across different levels of the second variable)
- interaction between IVs has a UNIQUE effect

How do we interpret results with an interaction effect?

- with an interaction effect, significant main effects should not be interpreted directly
- can still have significant main effects even if you have an interaction (but its not likely)

What is the layout of a 3-way factorial ANOVA?

3 independent variables

7 F ratios:

main effect IV1

main effect IV2

main effect IV3

interaction IV1 x IV2

interaction IV1 x IV3

interaction IV2 x IV3

interaction IV1 x IV2 x IV3

Mixed designs are the _____ for group design

gold standard

Mixed designs include:

both between and within subject IV

Mixed design is a very:

common treatment design
- between variable (treatment/no treatment)
- within variable (time - pre and post treatment)

Recall that covarying is:

statistically making the groups more equivalent on some variable/factor

What is ANCOVA?

analysis of covariance
- like ANOVA with another variable(s) added to the IV and DV

When is ANCOVA used?

- used to statistically compensate for group differences when randomization doesn't result in equal groups
- also used to "remove the effect" of the covariate

What does ANCOVA do for the interpretation of results?

makes the interpretation of the gain more valid

What is a factorial MANOVA?

- multiple IV and DV
- get multivariate Fs for main effects and interactions
- univariate Fs can be used to follow up significant effects

If you have covariates in factorial MANOVA, whats it called?

MANCOVA

How do we work down an example of Factorial MANOVA with short-term vs. long term recall of facts based on lecture duration and fact type?

Is there differences in students' short-term and long-term recall of facts based on lecture duration and fact type (i.e., the two dependent variables are "short-term memory recall" and "long-term memory recall", whilst the two independent variables are "lecture duration", which has four groups - "30 minutes", "60 minutes", "90 minutes" and "120 minutes" - and "fact type", which has two groups: "quantitative (numerical) facts" and "qualitative (textual/contextual) facts").
Factorial MANOVA --> 1-Factor MANOVA --> ANOVA --> t-test

Group designs allow for greater _____ to the larger population as a group, while single subject designs provide _____ analysis of individual performance

greater generalizability
provide detailed analysis

What are some benefits to single-subject designs?

- helps isolate characteristics that influence behavior
- group averages never mirror individual behavior exactly
- no individual matches the "theoretical average" client
- does allow "case-to-case" generalization
- client centered practice clinically calls for individualized care

Group designs test infrequently, what does this mean?

we cant see natural variability and growth in measure

Can group and single subject designs co occur in the same study?

yes, can do a group study and then do single study with select participants
*can test more frequently with these single subjects

What are some reasons to use single subject design?

- when withholding treatment is considered unethical (but on the other side, not ethical to provide treatment when no research to show its effective)
- when random assignment is not possible
- when you cant get enough participants --> useful for studying rare events
- when we want a lot of detail on participants, or intervention modifications, settings
- when you expect behaviors to change when conditions change
- when you dont have the resources to do group (ex: funding)
- useful for clinical innovation (want some evidence that its effective before investing in costly group study)

How are letters used in single subject designs?

different letters denote different stages of a study
A = baseline, no treatment
B = first treatment
C = second treatment, different than B
B' = first treatment with small variation

How are subscripts used with the letters?

subscripts are used to denote repetitions of a segment

How do we establish a baseline for single subjects design?

- repeated, continuous measurement of DV before beginning any manipulation (such as treatment)
- we are trying to capture variability in DV that occurs naturally (maturation, testing effect)
- assumed to represent how the DV would behave without intervention

What is a good baseline?

- multiple measures (minimum needed 3-4; ideally more)
- stable (limited variability, no clear trend up or down)
- no ceiling/floor effects (room for improvement/opportunity for contrasting results)

What are the 4 types of baselines?

stable
variable
stable accelerating/decelerating
variable accelerating/decelerating

How long is the baseline in single subjects design?

until we get stability
minimum 3-4; more is better

How long is the treatment(s)?

until we get stability, ideally
minimum 3-4; more is better
repeated, continuous measurement
*can be set by length (ex: 3 months)/number of sessions (20 sessions) OR until you achieve criterion (ex: 80% accurary)

Length of treatment in single subject design is influenced by:

target
type of client
type of intervention

What is a phase A descriptive design or case study?

- a baseline (not a hypothesis test)
- no experimental control or manipulation, only a DV
- careful and systematic descriptions of 1 or a few interesting or unusual case(s)
* this is used when you are curious how something changes naturally

What is a phase B descriptive design or case study?

- observe effects of treatment over time
- DV = behavior of interest
- IV = treatment

What does a phase B descriptive design or case study not allow for?

- unable to test causal relationships between the IV and the DV as no experimental control
- dont know what DV is like without treatment, what the normal variability is
- factors other than treatment may have influenced change in DV seen during phase B

What is a phase B descriptive design or case study a threat to?

internal validity

What is a pre-experimental A-B design?

A = baseline
B = treatment
- not experimental as no experimental control (don't know if something else happened at the time that may have influenced the results)
*allows you to see what DV look like without treatment (A), but you are unable to test causal relationships between the IV and DV because no experimental control

How do we establish control with experimental single subject designs?

replications
control goals

How do we establish control through replication with experimental single subject designs?

- Increased internal validity: the more frequently an effect can be replicated in a design, the stronger control against threats to internal validity and the more able to attribute change in DV to intervention
- Increased external validity (i.e. generalizability): systematically plan studies to vary across conditions
- interpretation: Rx is effective when:
--> behaviour changes when Rx is implemented and does not change for remaining baselines
--> behaviour changes only when the Rx is implemented and does so directly or closely after implementation

What is a withdrawal design?

Remove treatment during one or more phases to demonstrate the effect of treatment on behavior (e.g., A-B-A design)

In an A1 B A2 design, there is: _______ , which means that:

2 baselines, 1 treatment
if the DV reduces in second baseline (A2), we can more easily attribute change during B to treatment

How do we add more control into withdrawal designs?

by adding more phases
A1 B1 A2 B2

What is beneficial about an A1 B1 A2 B2 design?

- provides 2 opportunities to evaluate the effect of treamtent
- if DV declines or levels off in A2 and then improves again in B2, it demonstrates the consistency of response (gives strong support for the effect of treatment)

What are the problems to a 2nd baseline?

- improvement may be irreversible
- many treatments are designed to develop indepedent learning --> strategies individual can apply on their own (so maybe we cant take it away)
- withdrawing a treatment that is working is potentially unethical

What do we typically do instead of A1 B1 A2 B2?

ABC, where C is maintenance phase
- we assume the new skill remains when intervention is withdrawn
*this is the true goal of intervention, to have them maintain the results after treatment

What is concurrent monitoring? (in multiple baseline designs)

- once you've establish stability in the baseline, introduce treatment to all at the same time
*this can be problematic because what if something else started for them all at that time?

What is non-concurrent monitoring? (in multiple baseline designs)

- arbitrarily define different baseline lengths
- randomly assign length of baseline to subjects or conditions
- alternative, introduce treatment to second subject after response to treatment in first becomes stabilized

Which is preferable (concurrent or non-concurrent)? why?

non-concurrent
controls for maturation, history, other threats to internal validity

What is a multiple baseline across behaviors?

- two or more related yet functionally independent behaviors (DVs) within a participant
- vary length of baseline for each behavior
--> controls for external forces being responsible for change
--> strengthened when non-targeted behaviors remain stable until they are targeted

What is a multiple baseline across behaviors with control goal?

- choose something unrelated, that is not expected to change
- controls for external factors (ex: maturation)
- difficult to choose the one

What is a multiple baseline across behaviors with generalization goal?

- related, expectation that these will be affected by training on treatment goal (not main goal, but assume it may be impacted)

What is a multiple baseline across behaviors with treatment goal?

- directly treatment

What is multiple baseline across subjects?

- single subject design repeated with similar participants (6 or 7 individuals)
- looking to replicate findings
--> this increases internal validity
--> increases external validity/generalizability to some extent

What is the reality with multiple baseline across subjects?

its messy!

What is a multiple baseline across settings?

- single subject design repeated with same individual(s) in different settings

What is multiple baseline across settings typically used for?

used to show generalization - which is what you want in treatment

What does an ABCBCB design do?

compares the effects of rapidly alternating treatments B and C (where 1 of the treatments may be a placebo)

What are 2 problems with an ABCBCB design?

problem 1: treatment reactivity
- must only compare adjacent phases
- C cannot be assessed seperately from B
- ABAC: introduces a 2nd baseline; alleviates reactivity somewhat
problem 2: order effects
- order effects might be addressed using an ABC ACB design
- can be controlled through replication across subjects with alternate ordering

What is an interaction design?

purpose is to examine the impact of pieces of a complex intervention

What is a reduction design?

total package is evaluated and then compare to single component
A-BC-B-BC-B

What is an additive design?

determine effect of a simple protocol then add another component
A-B-BC-B-BC

What do additive and reduction designs allow for?

allow you to look at interactions among components of treatment
--> order/reactivity effects are not eliminated (can be controlled through replication across subjects)

What is a multiple treatment design?

- interactive designs:
--> A B BC (simplest): allow analysis of combined (BC) as well as seperate (B) treatment effects
- variations on this theme: ex: A BC B BC etc
- can potentially isolate impact of individual treatment components
* similar to interaction designs but view treatments as distinct rather than 2 parts of a package

What are 4 components to scientific methodology?

Operational Specificity
Repeated Measures
Interobserver agreement
External validity

What is Operational Specificity?

- researcher needs to explain things well enough so that people can understand
- should be able to replicate the study by reading it

How do establish operational specificity for the IV (treatment)?

- write out the instructions given to the participant
- stimuli/materials
- expected response
- feedback/reinforcement
- scheduling & amount of treatment

What is treatment fidelity evidence?

what evidence is there that they implemented the treatment they way they said?

How we obtain operational specificity for DV (outcome measure)?

- trial scoring (% correct - need obligatory context) --> so they know when they are wrong
- rate measures
- frequency measures
- interval and time sampling
*must set criterion level of performance - when you;ll say goal was mastered (not 100%, normally 70-80%)
*more likely to evaluate outcomes in multiple ways than in group designs (no issue of type 1 error)
- participant characteristics --> can get more info b/c less participants
- interventionist characteristics --> who provided intervention? what training do they have?
- setting characteristics --> what was the setting?
*these last 3 help us to apply to clinic

What is the repeated measures component of scientific methodology?

- done in consistent way
- at regular intervals (not 1 week between, then 3 weeks between)
- more frequently than in group design (easier to test, so able to do more)

What is a big concern with repeated measures?

- testing effects are a big concern here --> we aim to get baseline to get all testing effects out before you start treatment

What is important about the inter & intra-observer reliability?

- should have someone different doing testing and doing the intervention
- or if interventionist is collecting data, then having someone who is blind analyzing the data

How do we obtain external validity with single subject?

- get it with replication
- well described participants define who to extend findings to

What are the 2 ways to analyze single subject designs?

visual
statistical

What are two ways we do visual inspection?

- compare adjacent
1. level: value of dependent measure
- last point of previous phase to 1st point of following
2. trend: direction of change in a phase (3 patterns: accelerating, decelerating, flat)
- slope: rate of change

What are 2 ways we statistical analysis?

- compare slopes
- compare overlap in data points

What is a celeration line?

- tries to explain/estimate performance in both phases
"line of best fit"

What are 2 ways to compare slopes?

- extend baseline celeration line into treatment phase and count number of data points above celeration line
OR
- make a slope for baseline, and one for intervention and compare them

What do we compare slopes on?

level
trend
slope
variability

What is the two standard deviation band method of analysis?

- calculate mean and SD for the baseline data
- draw baseline +/- 2 SD and extend through treatment phase
- examine how many data points fall outside of the SD range

What is the percentage non-overlapping data method of analysis?

- look at range in each phase and determine percentage of data points that fall within overlapping ranges

How do we establish good vs. bad social validation with single subject design?

social validation: establishing the importance of the treatment effect
good: patient preferred, high comfort, high safety, low cost, simplicity of application
bad: intrusive, threatening, obscure, impractical