research design exam 1

program evaluation

provides information to decision makers to help them make judgements about the effectiveness of a program and help improve it

quality improvement

provides evidence to improve clinical practice and patient outcomes within the setting that it is generated

evidence-based practice (EBP) model

components:

• external evidence

• practitioner experience

• client’s situation and values

research ethics

• a code of guidelines on how to conduct scientific research in a morally acceptable way

• principles of standards that help researchers to uphold the value and standards of knowledge construction

Nuremberg Code 1948

the voluntary informed consent of the human subject is absolutely essential.

Declaration of Helsinki 1964

first set of ethical principles for experimentation

according to the belmont report what are the 3 basic ethical principles upon which human subject protections are based?

• respect for persons

  • the autonomy of the individual must be acknowledged, informed consent, voluntary participation

  • protect vulnerable populations

• beneficence

  • do no harm

  • minimize risks —> risks to subjects must be reasonable in relation to anticipated benefits (risk-benefit ratio)

• justice

  • fair distribution of burden and benefits

coercion

the use of threat or force (such as loss of services) is intentionally presented in order to obtain compliance

undue influence

the use of persuasion or manipulation, authority figures, or the offer of an excessive or inappropriate reward in order to obtain compliance

informed consent

• is an ongoing process throughout the study, includes providing new info that may impact participant’s decision to continue

• free to withdraw consent at any time

• special consideration given to vulnerable subjects

privacy vs confidentiality

• privacy = the control over the extent, timing, and circumstances of sharing oneself with others

• confidentiality = the treatment of information that an individual discloses in a relationship of trust, involves the expectation that info will not be divulged to others without permission

institutional review board (IRB)

• review research proposals and rules on approval or deny approval

• IRB evaluates

  • scientific merit

  • competence of investigators

  • risks to subjects

  • feasibility based on identified resources

  • looks at all procedures

primary outcomes

• directly related to study hypothesis

• determines success/failure of study

• designed to be detected with high degree of statistical power

secondary outcomes

• usually exploratory

• provides supporting evidence for primary outcome

• leads to new research directions

digital object identifier (DOI)

• unique and never-changing string assigned to online articles, books etc

• facilitate retrieving works

• must be included in APA style

grey literature

• research information available through formats other than customary journals

• examples: OT magazine, white papers, abstracts, websites, dissertations, etc.

• not peer reviewed (weakness)

clinical question vs. research question

clinical question

• purpose: to determine solutions to specific clinical problems, understood within a theoretical framework

• application of literature, clinical/professional judgement and client preferences to generate course of action

• plan of care design, implementation and evaluation of change

• process ends with a solution

• outcomes do not contribute to understanding beyond immediate situation

research question

• purpose: broader questions about recurrent phenomena and used to obtain knowledge generalizable beyond individual situations

• process ends with more questions

• goal: to contribute to scientific understanding

concept and constructs

concepts: symbolically represents observations and experience (not directly observable; i.e. work)

construct: represents a model of relationships among 2 or more concepts (i.e. QoL, composed of multiple concepts)

CONSORT

*explanatory designs

• RCTs

• single-subject design

• quasi-experiments

STROBE

*exploratory designs

• case-control studies

• cohort studies (retrospective, prospective)

• cross-sectional study

• correlational and predictive research

exploratory designs

• explore data about personal, environmental, behavioral, or genetic influences that may relate to health outcomes

  • used to predict effect of 1 variable on another OR to test relationships

  • prospective vs. retrospective

  • longitudinal vs. cross-sectional

cross-sectional

• studies a group of subjects at 1 point in time

• exposure and outcome variables measured concurrently

• strengths

  • provides insight into current health status of a population

  • helps inform decisions about how to best allocate public resources

  • efficiency

• weaknesses

  • reverse causation

  • difficult to document temporal sequence

longitudinal

strengths

• documenting change and establishing sequence of events

• confirms if suspected risk factor preceded outcome

prospective

• strengths

  • useful when target outcome can be expected to occur fairly frequently/without excessively long monitoring

• limitations

  • takes time

  • not practical for rare or slowly developing conditions

  • funds, resources

  • attrition, can lose patients over time

  • internal validity challenges

    • measurement

    • confounding variables influence data

retrospective

• strengths

  • cheaper/faster

  • more efficient for investigating disorders with long latency periods

• limitations

  • difficulty in defining variables

  • records may be incomplete

  • biased in reporting or assessment (aware of outcome already)

  • researcher unable to control procedures used to previously measure target variables

secondary analysis

• retrospective research often involves secondary analysis

• researcher uses existing data set

  • re-examine variables

  • answer questions other than for which data was originally collected

• may explore subsets of variables or participants, new relationships b/w variables

cohort studies (exploratory)

• cohort = a group of individuals followed over time

• cohort study = follow-up study

  • longitudinal investigation of an identified group of subjects who do not yet have an outcome of interest

  • exposed and unexposed cohort members monitored for period of time to see if they develop outcome

strengths/limitations of cohort studies

• strengths

  • allows for establishing correct temporal sequence of events

• limitations

  • not appropriate for studying rare conditions

    • cannot know at start of study if targeted outcome will occur with sufficient frequency for statistical comparisons

  • bias

  • attrition

case-control studies (exploratory)

• cases = those with target condition

• controls = those without target condition

• case-control study = method of observational research

  • 2 groups of individuals purposely selected on basis of if they do/don’t have health condition being studied

  • examine if the 2 groups differ in their exposure history or presence of a risk factor

  • comparison of cases can be done cross sectionally and retrospectively

  • almost ALWAYS RETROSPECTIVE

strength and limitations of case-control studies

strengths

• useful for studying rare conditions with long latency or incubation periods

limitations

• selection bias

• observation bias

• interviewer bias (aware of case/controls)

• recall bias

• confounding factors

correlational studies

• purpose: describe existing relationships among variables

• provide rationale for clinical decisions or generation of hypotheses

• several variables are often examined at once to determine those that are related

predictive correlational studies

• purpose: predict a behavior or response based on an observed relationship

• used to develop models to serve as basis for clinical decision making to understand factors that influence intervention success

• used for validation of diagnostic/prognostic information

strengths and limitations of correlational/predictive studies

strengths

• serves as first steps in sorting causal relationships

• examination of relationship of multiple variables

limitations

• without experimental control, cannot rule out possibility 2 variables may be related by their association

• no causation

methodological research

• the development and testing of measuring instruments for use in research and practice

• establishing reliability and validity of instruments is critical

• important in deciding tools to best demonstrate effectiveness of our services

reliability

• a measure’s consistency in score production

  • internal consistency; t-test

  • intrarater, interrater reliability

validity

• a measure’s ability to measure the characteristic or feature that it is intended to measure

  • content validity: bigger picture, does the assessment contain everything included in the construct

  • criterion validity: comparing the content of one measure to another/gold standard

  • construct validity: individual components, are they defined appropriately, do they measure what they’re intended to measure

MDC and MCID

minimal detectable change

• by therapist, minimal change required to exceed SEM

minimal clinically important difference

• by client, recognizes/notices difference

parametric vs. non-parametric variables

parametric

• normal distribution

• generally interval and ratio levels of measurement

non-parametric

• does not assume a normal distribution

• nominal and ordinal variables

descriptive statistics

• used to provide a description of a given variable

• ex. mean, median, mode, SD

• helps recognize outliers

• describes shape, central tendency, variability within the data

• can be displayed on a histogram (bar graph)

inferential statistics

• allow us to estimate unknown population characteristics from sample data

• ex. CI, t-test analysis, regression analysis

null hypothesis

there is no difference in the results, any difference is due to random error

alternative hypothesis

there is a statistical difference in the results

• this is termed the research hypothesis

Type I error vs. Type II error

Type I

• we reject the null hypothesis that is true (false positive)

Type II

• we fail to reject the null hypothesis when it is false (false negative)

p-value

probability value

• a measure of the likelihood that the results are due to chance alone

• results are significant if p<a

• typically, p<.05

correlation

• expressed as “r”

• varies between -1.0 and 1.0

  • 1.0 = very high correlation

linear regression

• one or more than one variable is used to try and predict the change of the other variable

• the closer R² is to 1 = the better the prediction

• does not represent causation

odds ratio

• used to express the odds of having an outcome of interest vs. the odds of not having an outcome of interest

  • OR = 1 —> exposure does not affect the odds of an outcome

  • OR>1 —> exposure associated with higher odds of outcome

  • OR<1 —> exposure associated with lower odds of outcome

risk ratio (RR)

• RR>1 —> increased risk in exposed group

• RR<1 —> reduced risk in exposed group

confidence interval

• a range of scores within specific boundaries (confidence limits) that should contain the population mean

  • the percentage of “confidence” that the true population mean will fall within the interval

  • typically 95%

effect size

• describes the magnitude or strength of a relationship

• Cohen’s d is commonly reported and interpreted according to common suggestions:

  • small, d=.2

  • medium, d=.5

  • large, d=.8

searchable clinical questions

contain 3 main elements:

• patient characteristics

• patient management

  • intervention, diagnosis

• outcome of interest

efficacy of an intervention

• intention: to help therapists make clinical decisions about implementing interventions

• PICO/PIO format

• common research designs associated:

  • RCT, nonrandomized control trials, pretest/postest without control group

• in infants, what is the efficacy of swaddling (versus no swaddling) for reducing crying?

PICO

P: patient/population

I: intervention (exposure or test)

C: comparison (if relevant)

O: outcome

usefulness of an assessment

• intention: to understand usefulness of assessment and determine if measures being used have sufficient reliability/validity

• common research designs

  • psychometric methods, reliability studies, validity studies, sensitivity studies

• What is the best assessment for measuring improvement in ADL function?

• What methods increase the validity of the health-related QoL assessment?

description of a condition

• intention: to better understand the individuals you work with as therapists

• common designs:

  • incidence and prevalence studies

  • group comparisons

  • surveys/interviews

• What motor problems are associated with CP?

• are anxiety disorders a common comorbidity among clients with COPD?

prediction of an outcome

intention: to understand what factors contribute to the prognosis, response or outcomes in a specific condition

• to understand associations between factors

common designs

• correlational and regression studies

• cohort studies

examples

• what childhood conditions are related to stuttering in children?

• what factors are associated with adherence and attendance for pulmonary rehabilitation?

lived experience of a client

• intention: to understand the evidence from the client’s perspective

• SPIDER format

  • sample

  • phenomenon of…

  • interest

  • design

  • evaluation

  • research type

• common designs

  • qualitative studies

  • ethnography

  • narrative

• what is the impact of MS on parenting?

• how do athletes deal with career-ending injuries?

CAT

critically appraised topic

CAP

critically appraised papers

independent variable

• is manipulated to make comparisons in study

• may have multiple levels

• also called predictor variable

• can have more than one IV

dependent variable

• measures the result of the manipulation

• also called outcome variable

• study may look at more than one DV

exploratory vs. explanatory studies

exploratory

• IV and DC often measured together to determine if they have a predictive relationship

• interested in cause and effect

explanatory

• different conditions are compared to investigate causal relationships

• IV is controlled and DV is measured

other variables

• control

  • can affect the outcome of the study but remain constant

  • are “controlled” by design/statistical procedures used

• extraneous

  • can influence outcome of study and are tracked so influence can be examined later

• all variables must be identified and operationalized (defined)

hypotheses

• declarative statement made between IV and DV

• hypotheses are NEVER proven or disproven

• can only be supported or show lack of evidence to support it

non-directional hypothesis

directional hypothesis

results that are statistically significant are not…

necessarily clinically significant.

• factors that can help determine clinical significance:

  • sample size

  • MCID

  • CI

  • effect sizes

  • clinical utility

data analysis

• data analysis method is determined by the type of data being used

  • continuous vs categorical

  • dichotomous or not?

  • parametric vs. non-parametric

    • parametric = mean

    • non-parametric = median

explanatory designs

• seek to evaluate cause and effect between set of IV and DV

• seek to create generalizable knowledge

  • RCTs

  • quasi-experimental designs

  • single-subject designs

RCTs

• answer the question: can this intervention or treatments work under ideal conditions?

• compares a treatment vs. comparison group

  • comparison group may be control group

strategies for increasing study rigor

• large sample size

• random assignment

• use of comparison group

• selection of robust outcome measures

• blinding of intervention administrators and participants

external validity

• the extent to which study results can be generalized to other person, settings, or times

• usefulness of the findings outside experimental situations

• strategies to enhance external validity:

  • the type of subject tested (studies may restrict who participates/exclusion criteria)

  • setting in which the experiment is carried out

  • time in history when study is done (technology, interventions that are timely, etc).

internal validity

• methodological components of the study

• the extent to which the study captures evidence of a cause-and-effect relationship between the IV and DV

• does one cause the other or alternative explanation?

• strategies to enhance:

  • internal factors such as time, selection, attrition, testing, instrumentation

  • if individual is repeatedly exposed to a measure it could affect results

  • social factors such as response of subjects to treatment and research adherence protocols

strengths and limitations of RCTs

strengths

• methods reduce bias via use of comparison group, randomization, and blinding

• sample and interventions are clearly identified

limitations

• only tell us about benefits to a group (not generalizable to everyone)

• experimental conditions markedly differ from real-life situations

• participants may differ from real-life clients

• expensive

pragmatic clinical trial

• no control group

• answers the question: can this intervention or treatment work under usual conditions?

• compares: the comparative effectiveness of real-world alternatives

quasi-experimental designs

• “kind-of” designs

• many similarities to experimental designs

  • manipulation of an IV without either random assignment, comparison groups or both

• NOT a TRUE experiment

strengths and limitations of quasi-experimental designs

• strengths

  • accommodate limitations of natural settings where logistics of treatment conditions and random assignment are often difficult, impractical or unethical

• limitations

  • only tell us about the benefits to a group

  • cannot rule out biases like RCTs

  • groups may differ from each other in ways other than treatment condition (other variables are not controlled for)

single-subject designs

• answer the question: can this intervention or treatment work for an individual?

• involves serial observations of individuals before, during and after interventions

  • participant is both control and treatment group

  • only one variable is changed at a time

• ABA design

  • baseline period (A), intervention period (B), and return to baseline period (A)

• allows researchers to look at individual outcomes

steps of single-subject design

• step 1: operationalize target behavior

• step 2: collect baseline data

• step 3: conduct intervention

• step 4: reassess at specified time intervals

• step 5: shift course as needed

strengths and limitations of SSDs

• strengths

  • helps understand those who responded favorably to treatment from those who did not improve

  • more directly contributes to decision-making about individual clients and practice

• limitations

  • results cannot be generalized beyond individual

    • will changes occur with other individuals?

    • will improvement be sustained?

efficacy vs. effectiveness

efficacy

• performance of an intervention under ideal and controlled circumstances

• high internal validity

• can overestimate an intervention’s effect once it is implemented in clinical practice

effectiveness

• performance of an intervention under “real-world” conditions

  • i.e. moving intervention to Magee Rehab to see results

• accounts for external patient-provider and system level factors that may moderate an intervention’s effects

studies that summarize other studies

• systematic review

• meta-analysis

• scoping review

clinical practice guidelines

• systematically developed statements

• include recommendations to optimize patient care

  • not fixed protocols, guidelines to consider

• informed by a systematic review of evidence

  • integrates assessment of the benefits and harms of alternative care options

high quality clinical guideline

• institute of medicine set out standards for developing trustworthy clinical practice guidelines

  • covers areas like managing conflicts of interest, review by stakeholders, and updating over time

• Equator network’s AGREE guidelines

  • instead of CONSORT, STROBE, etc.

strengths and limitations of clinical practice guidelines

• strengths

  • can improve quality of care and decision-making

  • encourage use of supported interventions and discourage use of others

  • can enhance consistency of care around world

• limitations

  • misleading, lacking, or misinterpreted articles may lead to incorrect guideline recommendations

  • recommendations are influence by opinions and clinical experience of development group

  • patients may not always be included or priority

scoping review

• exploratory projects that systematically map the literature available on a topic, identifying key concepts, theories, sources of evidence and gaps in research

• useful for:

  • clarifying working definitions and conceptual boundaries of a topic

  • when a body of literature has not been comprehensively reviewed or is large/complex

• helps determine the value of undertaking a full systematic review

strengths and limitations of scoping reviews

• strengths

  • useful for understanding available evidence without a summary answer

  • describes evidence on a broad question

  • can inform future research, including systematic review

• limitations

  • does not formally evaluate quality of evidence

  • results are broad and provide an overview of literature, not synthesized answer

systematic reviews

• scientific approach to addressing a research question through synthesis of existing research vs. collecting new data

  • summarizes findings of multiple studies on a topic

• includes same sections as other research articles

• no statistical analysis/synthesis

• Cochrane collaboration

sections of a systematic review

• background and rationale

• methods

  • selection criteria

  • search strategy

  • screening process

  • data extraction

• results

  • PRISMA flow diagram (documents process of identifying or excluding relevant studies)

  • assessing methodological quality

    • PEDro Scale

  • data synthesis

    • assessing degree to which studies have similar findings

• discussion/conclusions

  • focus on reporting overall results across studies not summaries of each individual study

  • implications

strengths and limitations of systematic review

• strengths

  • comprehensive analysis and summary of high quality articles on topic

    • gain insight into body of literature of interest

    • identify primary studies that can provide more detail to guide practice

    • time efficient

• limitations

  • still need to critically review process used

  • do not usually provide details to guide specifics for intervention

  • publication bias

  • study heterogeneity

meta-analysis

• extension of systematic reviews… comprehensive study that summarizes findings of multiple studies

• involves statistical analysis of data

• advantages:

  • increased power by pooling data from several samples, improving estimates and effect sizes and generalization

meta-analysis effect size

• estimates from individual studies are combined to reflect an overall size of the effect of the IV

• intervention studies: standardized mean difference (SMD)

  • difference between group means divided by their pooled standard deviation

• observational studies: risk ratio

• weighting effect size

  • helps determine how much each study’s effect size contributes to the overall estimate —> relative precision of each study

meta-analysis forest plots

• graphical representation of studies included

meta-analysis heterogeneity

• statistical heterogeneity

  • degree to which studies are similar in design and treatment effect

  • Cochran’s Q

  • I² statistic

meta-analysis sensitivity analysis

• assess if findings change when key assumptions differ

  • ex. studying subgroups of studies that meet different exclusion and inclusion criteria

• involves re-analyzing data using different statistical approaches or accounting for inconsistencies in reporting results in individual studies

meta-analysis: funnel plots

• a scatter plot of treatment effect against a measure of precision or variability for each study

  • if data points are more scattered —> points to less chance of bias

meta-analysis GRADE system

• GRADE

  • grade of recommendation, assessment, development and evaluation system

  • used to assess the quality of the body of evidence in systematic reviews and meta-analyses

strengths and limitations of meta analysis

• strengths

  • increased statistical power

  • estimate of magnitude of experimental effects

  • increased insight of relationship between variables

  • systematic comparison of differences between studies

• limitations

  • complications in statistical analysis with combining results

    • combined studies need to be similar in population, sample, intervention and outcome measures

    • can only combine results when studies have clinical homogeneity

AMSTAR-2

• tool to formally appraise a systematic review or meta-analysis

• 16 item checklist to assess adequacy of reporting each component of the review

paired sample t-test

• also called: one sample t-test, dependent sample t-test

• when one sample/group is measured twice under different conditions

  • same group measured at 2 different time points or regarding 2 different DVs

• interval/ratio data

independent samples t-test

• also called: two sample t-test

• when two different samples are measured regarding the same outcome variable

  • compares the differences between 2 different groups on one DV

• interval/ratio data

linear regression

• compares the relationship between IV and DV

• interval/ratio data

• multiple regression =

  • when comparing relationship between 2 or more IVs and 1 DV (DV is continuous)

  • if DV is categorical —> multiple logistic regression