appraising quantitative research design

key questions of quantitative research design

will there be an intervention? what types of comparisons will be made? how will confounding variables be controlled? will blinding be used? how often will data be collected? when will effects be measured relative to potential causes

key features of quantitative designs

intervention, comparisons, control over confounding variables, blinding, rime frames, relative timing, location

causality

many if not most quantitative research questions are about causes and effects. research questions that seek to illuminate causal relationships need to be addressed with appropriate designs

counterfactual

what would have happened to the same people exposed to a cause if they simultaneously were not exposed to the cause

effect

represents the difference between what actually did happen when exposed to the cause and what would happen with the counterfactual condition

key criteria for making causal inferences

temporal, relationship, confounder

temporal

the cause must precede the effect in time

relationship

there must be a demonstrated association between the cause and the effect

confounder

the relationship between the presumed cause and effect cannot be explained by a third variable or confounder, another factor related to both the presumed cause and effect cannot be the real cause

biological plausibility

the causal relationship should be consistent with evidence from basic physiologic studies

different designs are…

appropriate for different questions. therapy, prognosis, etiology/ harm and description

experimental designs offer…

the strongest evidence of whether a cause (an intervention) results in an effect (desired outcome). thats why they are high on evidence hierarchies for questions about causes and effects

therapy cause probing hierarchy of design

rct/ experimental design→ quasi experimental→ cohort study→ case control→ descriptive/ correlational

prognosis cause probing hierarchy of design

cohort study→ case control→ descriptive/ correlational

etiology/ hard cause probing hierarchy of design

rct/ experimental design→ quasi experimental→ cohort study→ case control→ descriptive/ correlational

within/ between group comparison design considerations

can use either or both in a study

within groups

comparisons about measurements made with same subjects at one or more points in time

between groups

comparisons made with more than one group of subjects at one or more points in time

within group comparison schema

O1 X O2. group 1 observation 1 then there is an intervention and then group 1 observation 2

between group comparisons schema

Experimental Group: (O1 X O2) → Change after treatment. intervention used

Control Group: (O3 O4) → Change without treatment. no intervention

intervention

the researcher does something to some subjects. introduces an intervention or treatment. pre and post tests

control

the researcher introduces controls, including the use of a control and experimental groups

randomization

the experimenter assigns participants to a control or experimental condition on a random basis. the purpose is to make the groups equal with regard. to all other factors except receipt of the intervention

design symbols- schema

r- randomization

o- measurement at a point in time (observation, data collection)

x- intervention or treatment (can sometimes be listen as t)

post test only (or after only) design experimental design

outcome data collected only after the intervention

post test only (or after only) symbolic representation

R X O

R O

pretest- posttest (before- after) design

outcome data collected both at baseline and after the intervention

pretest- posttest (before- after) design symbolic representation

R O X O

R O O

crossover design

subjects are exposed to 2+ conditions in random order. subjects serve as their own control

crossover design symbolic representation

R     O     X_A      O      X_B      O

R     O     X_B      O      X_A      O

experimental condition proxy

must be designed with sufficient intensity and duration that effects might reasonably be expected. researchers describe the intervention in formal protocols that stipulate exactly what the treatment is. attention must be paid to intervention fidelity

intervention fidelity

treatment fidelity. weather the treatment as planner was actually delivered and received

control group condition (counterfactuals)

no intervention is used. control groups get no treatment at all. usual care or standard or normal procedures is used to treat patients. an alternative intervention is used. a placebo or pseudointervention presumed to have no therapeutic value is used. attention control condition and delayed treatment (wait listed)

attention control

extra attention but not the active ingredient of the intervention

delayed treatment (wait listed controls)

the intervention is given at a later date

delayed treatment representation

R     O     X       O               O

R     O              O      X       O

advantages of experiements

most powerful for detecting cause and effect relationships

disadvantages of experiments

often not feasible or ethical, hawthorne effect (knowledge of being in a study may cause people to change their behavior), too expensive

quasi experiments

involve an intervention but lack either randomization or control group

2 main categories of quasi experimental

nonequivalent control group designs and within subjects designs

nonequivalent control group designs

those getting the intervention are compared with a nonrandomized comparison group

within subjects designs

one group is studied before and after the intervention

nonequivalent control group designs mehtod pretest posttest

if the pre intervention date are gathered then the comparability of the experimental and comparison groups at the start of the study can be examined

nonequivalent control pretest posttest design symbolic representation

O₁     X      O₃

O₂             O₄

nonequivalent control group design post test only

without pre intervention data it is risky to assume the groups were similar at the outset. nonequivalent control group posttest only is much weaker

nonequivalent control group posttest only symbolic representation

X      O₁

         O₂

within subjects one group pretest posttest designs

typically yield extremely weak evidence of causal relationships.

symbolic representation: O1 X O2

time series designs

gather preintervention and postintervention data over a longer period

symbolic representation: O₁ O₂ O₃ O₄ X O₅ O₆ O₇ O₈

advantages and disadvantages of quasi experimentals

may be easier and more practical than true experiments but they make it more difficult to infer causality and usually there are several alternative rival hypotheses for results

nonexperimental studies

if researchers do not intervene by controlling independent variable the study is non experimental

not all independent variable of interest to nurse researchers can be….

experimentally manipulated. for example gender and smoking cannot be manipulated bc of ehtics

correlational designs

cause probing questions for which manipulation is not possible are typically addressed with a correlational design

correlation

an association between variables and can be detected through statistical analysis

correlational studies are..

weaker than RCTs for cause probing questions but different designs offer varying degrees of supportive evidence

prospective correlational design

a potential cause in the present is linked to a hypothesized later outcome. this is called a cohort study by medical researchers

prospective designs are stronger than…

retrospective designs in supporting causal inferences- but neither is as strong as experimental designs

retrospective correlational design

an outcome in the present is linked to a hypothesized cause occuring in the past

case control

a retrospective design in which cases are compared to controls on prior potential causes

not all research is…

cause probing

the purpose of descriptive studies is to

observe, describe and document aspects of a situation

descriptive correlational

the purpose is to describe whether variables are related without ascribing a cause and effect connction

disadvantage of nonexperimental research

does not yield persuasive evidence for causal inferences. this is not a problem when the aim is description but correlational studies are often undertaken to discover causes

advantage of nonexperimental research

efficient way to collect large amounts of data when intervention and or randomization is not possible

cross sectional design

data are collected at a single point in time

longitudinal design

data are collected two or more times over an extended period

follow up studies

longitudinal designs are better at showing patterns of change and at clarifying whether a cause occurred before an effect. a challenge in longitudinal studies is attrition or the loss of participants over time

controlling external factors

achieving constancy of conditions, control over environment setting time, control over intervention via a formal protocol: intervention fidelity

controlling participant factors

randomization, homogeneity, matching, statistical control (analysis of covariance)

statistical conclusion validity

the ability to detect true relationships statistically

internal validity

the extent to which it can be inferred that the independent variable caused or influenced the dependent variable

external validity

the generalizability of the observed relationships across samples, settings or time

construct validity

the degree to which key constructs are adequately captured in the study

threats to statistical conclusion validity

low statistical power, weakly defined cause- independent variable not powerful, unreliable implementation of a treatment- low intervention fidelity

threats to internal validity

temporal ambiguity, selection threat, history threat, maturation threat, mortality threat

selection threat

biases arising from preexisting differences between groups being compared. this is the single biggest threat to studies that do not use an experimental design

history threat

other events co occurring with causal factor that could also affect outcomes

maturation threat

processes that result simply from the passage of time

mortality/ attrition threat

differential loss of participants from different groups. typically a threat in experimental studies

history threat reduction

random selection or assignment

maturation threat reduction

control group, random assignment, baseline data

testing

multiple testing might influence subjects response on subsequent testing

testing threat reduction

post test only design, control group tested same number times or prolong length of time between tests

selection threat reduction

random sampling, random assignment

mortality threat reduction

collection of demographic variables from all subjects for future comparison those who complete, communicate clear instructions and expectations regarding participation in the study

instrumentation

inconsistency of data

instrumentation threat reduction

comprehensive training of data collectors, reliability and validity of instruments

threat to external validity

person, time, place. inadequate sampling of study participants, unfortunately enhancing internal validity can sometimes have adverse effects on external validity

threats to construct validity

Is the intervention a good representation of the underlying construct?

Is it the intervention or awareness of the intervention that resulted in benefits?

Does the dependent variable really measure the intended constructs?