Research I Midterm

Rubin Ch. 1-6, articles, & class activities

What research level?: Forty individuals in a long-term care facility receive strength and balance training to reduce falls. The number of falls for a 1-month period before the intervention is compared with the number of falls for a 1-month period after the intervention.

3: Nonrandomized before-after design

What research level?: Twenty individuals on one wing of a long-term care facility receive strength and balance training, while 20 individuals on another wing of the same long-term care facility receive usual care. The number of falls for the two groups is compared for 1 month before the intervention and 1 month after the intervention.

2: Nonrandomized controlled design

What research level?: The results of three randomized controlled trials examining the efficacy of strength and balance training to reduce falls in long-term care facilities are analyzed and synthesized.

1: SR

What research level?: A resident in a long-term care facility receives individualized strength and balance training. The number of falls for this individual is determined at different time points and presented in a report.

5: Case report

What research level?: Individuals in three long-term care facilities are randomly assigned to receive either strength and balance training or usual care. The number of falls for the two groups is compared for 1 month before the intervention and 1 month after the intervention.

1: RCT

Threat to internal validity?: A researcher is interested in a study comparing two different teaching methods. One method is used  at School A and the other method is used at School B. During the course of the study, the principal at School B is fired and replaced by a new principal.

History

Threat to internal validity?: Children recently enrolled at a nursery school are having trouble adjusting to the teacher and the  scheduled activities. The head teacher designs a program of story telling that is intended to help them adjust. Four  weeks later, observations are made and the children are better adjusted than they were before the program began.

Maturation

Threat to internal validity?: A researcher wants to assess the effectiveness of a college energy conservation program. This  program included an energy conservation campaign as well as an improved method for monitoring the college’s energy  usage. The researcher recorded the amount of energy use based on archival sources for the 2 years prior to the  program and for the 2 years following the end of the program. The researcher found a decrease in energy use at about  the time when the program was initiated.

Instrumentation

Threat to internal validity?: An educational psychologist designs a demanding but fair spelling test that can be administered to  grade school children. The test is found to be valid and reliable. A researcher selects the test to help evaluate a  “cognitive” approach to teaching spelling. A 5^th grade class is tested prior to being taught the new method and then  again after the new method is taught. The children show improvement on their spelling scores from the first to the  second test administration, suggesting that the program was effective in helping children to learn to spell.

Testing

Threat to internal validity?: All students planning to enroll in a small Midwest college are given math aptitude tests prior to  registration for classes. Students scoring the lowest are selected for a 1-week math enrichment program to be taken  prior to registration for classes. Following participation, students are tested again. Scores on the math aptitude test  following the enrichment program show that students have improved. Credit for this change is given to the program.

Regression to the mean

Threat to internal validity?: A researcher is interested in the impact that a new science curriculum has on boys and girls’ long term interest in science. He matches 4^th grade boys and girls on their overall interest in science and tracks them until  the 12^th grade. The researcher found that by the 12^th grade, the boys had significantly more interest in science than the  girls. The researcher concluded that the new science curriculum was only effective in increasing boys’ long-term interest  in science.

Selection by maturation interaction

Threat to internal validity?: A psychologist is trying to figure out whether males or females know more about political issues.  The psychologist stops people on the street, asking them to participate in a survey on political issues. The results of his  survey found that men know much more about politics than women do.

Selection of subjects

Threat to internal validity?: A social scientist is hired by a large company to assess the effect of an employee training program.  Volunteers are sought from the employees and a modest bonus offered for participation. Thirty-six employees agree to  participate and are randomly assigned to the program condition (n = 18) or to a waiting list control group (n = 18). The  employees on the waiting list are scheduled to receive the training program following the first group’s training;  meanwhile, these employees serve as a control group for the group trained first. Training requires that the employees  attend sessions after normal work hours and also that they complete brief written assignments at home. Twelve of the  employees finish the training program, and an analysis of appropriate job performance measures reveals that their  scores are better than of those the employees in the control group.

Mortality

Identifying variables?: Dementia and Parkinson’s disease?

Control

Identifying variables?: Individuals with aspiration of thin liquids?

Control

Identifying variables?: Degree of aspiration?

Dependent

Identifying variables?: Three treatment options?

Independent

Identifying variables?: Patient preference?

Dependent

Identifying variables?: Patient’s anxiety level?

Extraneous

Why use EIP?

Guides practice decisions & interventions + better equipped to understand others (manage care companies)

Age of accountability

People want to know WHY and HOW, not just take OTPs word

Barriers to EIP?

Not enough/gaps in research, attitudes, demand/time, cost

EIP=

best evidence + OTP’s expertise + client’s values, attributes, preferences, & circumstances

Practice decisions are…

Informed by research evidence

EIP Question #1

What are the factors that best predict desirable & undesirable outcomes?

What is the best intervention?

An occupation-based intervention

EIP Question #2

What can I learn about clients, service delivery, & targets of intervention from experinces of others?

What is one of the most important factors influencing intervention effectiveness

Quality of practitioner-client relationship

EIP Question #3

What assessment tool should be used?

EIP Question #4

Which intervention, program, or policy has the best effects?

EIP is a…

Process for making practice decisions in which OTPs integrate best evidence available with their expertise, and client factors (attributes, values, preferences, circumstances)

EIP 5-Step Process:

1. Form question

2. Find best evidence

3. Critically appraise evidence

4. Select intervention based on appraisal + expertise + client factors

5. Monitor client progress

All knowledge is…

Subject to refutation/change

EIP Question #5

What are the costs of interventions, policies, and tools?

What must OTPs know regarding EIP question #5?

The resources available in practice context and their cost (ratio to costs to benefits)

EIP Question #6

What are the potential harmful effects of interventions, policies, & tools?

Contraindicated

Not indicated for one particular diagnosis or symptom; cannot do interventions with certain individuals b/c they could cause significant harm

Using PICO framework, P stands for…

Patient, population, problem

Using PICO framework, I stands for…

Intervention (or instrument)

Using PICO framework, C stands for…

Comparison

Using PICO framework, O stands for…

Outcome

How can you overcome feasibility obstacles?

• cut corners

• work with a team

• refer client to well-prepared agency

Dodo Bird Verdict?

Idea that all interventions are effective if the practitioner-client relationship is strong

Dr. Trudell doesn’t like “this proves X”. What should you say instead?

“supports X”

Qualitative designs…

Employ flexible designs & subjective methods w/ small sample size

Quantitative designs…

Produce precise & objective statistical findings; generalizable results (geared at testing if hypotheses about causes really produce effects)

AOTA Hierarchy: Level I consists of…

Systematic reviews, meta-analyses, randomized controlled trials

AOTA Hierarchy: Level II consists of…

Nonrandomized controlled designs (cohort) & case control designs

AOTA Hierarchy: Level III consists of…

Nonrandomized before-after (pretest-posttest) & cross-sectional designs

AOTA Hierarchy: Level IV consists of…

Single-subject design

AOTA Hierarchy: Level V consists of…

Case report

Other research designs below Level V?

• Anecdotal case reports

• Pretest/posttest w/o control group

• Qualitative descriptions of client experiences during & after treatment

• Surveys

Systematic reviews (1)…

Look at every study/report then appraise them

Meta-analyses (1)…

Combine data from only randomized experiments, find common effect, & emerge w/ final assessment for common effect (more statistical subset of SRs)

Randomization

Increases likelihood to have one group that is similar in terms of the other group in an experiment; greatly reduces chance of confounding variables (motivation)

Level I Diagram

Participants→Randomization→ Experimental group & Control group→Outcome

Nonrandomized controlled designs (Cohort design)…

Group of people share common experience within period of time; prospective

Level II Diagram

Participants→Exposure to intervention & No exposure→Outcome

Case control design…

Compares people with specific condition(case) with people who don’t(control) and explores what makes groups different retrospectively

Nonrandomized Before-After design (pretest-posttest)…

Evaluate group involved in an intervention; no control so difficult to conclude what responsible for outcome

Nonrandomized Before-After Diagram

Participants→Assessment→Intervention→Outcome

Cross-sectional design…

Looks at one group at a single point in time; not generalizable

Cross-Sectional Diagram

Participants→Measurement of outcomes & other factors at same time

Single-subject design…

Repeated evaluation of clients for outcomes both before baseline and after intervention

Why are single-subject designs considered plausible?

Collect many data points which reveals/predicts trends

Level IV Diagram

Client→Baseline→Intervention→Evaluation

Case report…

Descriptive info about relationship between both a particular intervention(exposure) and an outcome

Level V Diagram

Participants w/ condition of interest→Info about clinical outcome

How do interventions differ from assessments?

During interventions clients work toward outcomes to improve what was assessed

Best research design for EIP question #1?

Correlational, case control, qualitative

Best research design for EIP question #2

Qualitative

Best research design for EIP question #3

Correlational: test-retest reliability, differences in averages of 2 groups validity, pretest-posttest no control sensitivity

Best research design for EIP question #4

Experimental, quasi-experimental, single-subject, correlational

Selectivity bias

Doubt between comparability of groups

What is one of the key factors that cause high selectivity bias?

Motivation

Example of measurement issue of participants altering their behavior?

Over reporting the amount of vegetables I eat on a daily basis to please my PCP

Example of measurement issue of self-reporting bias?

A participant does not perceive the fact that they drink 1 beer a day as a negative habit, so they underreport the amount they drink

Example of measurement issue of recall bias

A student rates a professor poorly on a course survey because of a negative incident that occurred the previous week, despite having had consistently positive interactions with the professor throughout the rest of the semester

Maturation

Physiological processes occurring within the participants that could account for any changes in their behavior

Regression to the mean

Tendency that participants who receive extreme scores when tested, tend to have less extreme scores on subsequent retesting even in the absence of any treatment effects

Selection of subjects

Any bias in selecting and assigning participants to groups that results in systematic differences between the participants in each group

Selection by maturation interaction

Treatment and no-treatment groups, although similar at one point, would have grown apart (developed differently) even if no treatment had been administered

Mortality

Differential dropping out of some subjects from the comparison groups before the  experiment is finished, resulting in differences between the groups that may be unrelated to the treatment effects

Instrumentation

Changes in the measurement procedures may result in differences between the  comparison groups that are confused with the treatment effects

Testing

When participants are repeatedly tested, changes in test scores may be more due to practice or  knowledge about the test procedure gained from earlier experiences rather than any treatment effects

History

Extraneous events occurring during the course of the experiment that may affect the  participants’ responses on the dependent measure

Small sample size

Fewer than 80 participants

Internal validty

Was the intervention the most plausible cause of the outcome?

External validity

Do the study’s participants, intervention procedures, & results seem applicable to your practice context?

Strong internal validity means…

The results are confidently attributed to the effect of the IV

Time order

Causal variable comes first, then the effect variable

Threats to internal validity (alternative explanations)

• History

• Passage of time (or maturation)

• Regression to the mean

• Testing

• Instrument decay

• Selectivity bias

Regression to the mean

Elevated or very low scores in the pretest move closer to the mean once tested again

Testing

Pretest changes participants behavior

Instrument decay

Over time, observer gains skill or gets bored

Best way to avoid selectivity bias?

Random assignment

__ controls for statistical regression, history, and passage of time and eliminates influence of individual characteristics

Random assignment

Reliable measures are…

Consistent & precise

Masked tx approach

Uninformed as to purpose of study or desired results

Statistically significant?

p < 0.05 (5x out of 100)