CR Exam 4

What is reliability and what concept is it tied to?

Extent to which a measurement is consistent and free from error; tied to measurement error

What is the best reliability coefficient for continuous data?

Intraclass correlation coefficient (ICC)

What is the best reliability coefficient for discrete (ordinal/nominal) data?

Kappa

ICC's measures degree of ___________ and ___________ simultaneously

relationship; agreement

ICC can be used with ______ raters or ratings

>2

What other measurement is ICC often reported in conjunction with?

Standard error of measurement (SEM)

ICCs give "standardized" (__________) estimates of reliability

relative (ex: no units)

SEM gives "unstandardized" (____________) estimates of reliability

absolute (ex: in the units of measurement)

What two type of numbers define ICC?

Model (1st number), Form (second number)

How many types of ICC types are there? What are they dependent on?

6; purpose of study, design of study, type of measurement taken

What is model 1 in the ICC?

Each subject is measured by a different set of rater; raters are randomly chosen (rarely used in clinical research)

What is model 2 in the ICC?

Each subject measured by same raters; raters considered randomly chosen and representative of rater population (common for inter-rater or test-retest reliability)

What is model 3 in the ICC?

Each subject measured by same raters; raters are only ones of interest; can be for inter-rater if study raters only ones of interest (common for intra-rater reliability)

What model of the ICC can results be generalizable?

Model 2

What model of the ICC can results not be generalizable?

Model 3

What do the forms mean in ICC?

Number of observations used to obtain reliability estimate; k is a placeholder and the # should be specified in manuscripts

If generalizing to only one observation per subject per rater, what is the form number?

1

If generalizing to multiple observations averaged to get single number for analysis, what is the form number?

Number of observations averaged

What ICC/Kappa number range means near complete agreement?

0.81-1.00

What ICC/Kappa number range means moderate agreement?

0.41-0.60

What ICC/Kappa number range means poor agreement?

0.00-0.20

What ICC/Kappa number range means fair agreement?

0.21-0.40

What ICC/Kappa number range means strong agreement?

0.61-0.80

Will ICC estimates based on "average measures" or "single measures" be substantially higher?

Average measures

What is the null value in CI for ICC interpretations?

"0," if CI includes it is not significant

What are reliability for categorical scales based on?

Frequency tables, agreements on diagonal

What is percent agreement in categorical scales?

How often raters agree, dividing number of agreements by total of all possible agreements

Problem with percent agreement?

Does not account for agreement due to chance, tends to overestimate reliability

What is kappa coefficient for categorical scales?

Proportion of agreement between raters after chance agreement has been removed (will be lower than % agree)

What is a weighted kappa?

Can choose to make a "penalty" worse for larger disagreements

Why is internal consistency used?

Looking at how well the questions correlate with total score and if different questions measure the same concept or indicator

When is internal consistency often used?

To construct and evaluate scales/questionnaires

What is Cronbach's alpha?

Represents correlation among items and correlation of each individual item with the total score

What is the recommended Cronbach's alpha?

Between 0.70-0.90

What does Cronbach's alpha help with?

Eliminating items from test/questionnaires that are not homogenous to the set or not contributing unique information

What is alternate/parallel forms looking at?

Comparing different methods of testing same phenomenon with different instruments

What is limits of agreement?

Range that includes approximately 95% of differences when comparing alternate forms

When comparing alternate forms, what does the mean of dots in a bland-altman plot represent?

Evaluates if there is bias when comparing difference between 2 measures, if scores are consistently above/below the line

SEM is __________ measure of reliability

Absolute

What is minimal detectable difference/change (MDC)?

Amount of change in a variable that must be achieved to reflect a true change/difference (mathematical multiple of SEM)

What is epidemiology aimed at?

Measuring risk; aimed at studying determinants of disease, injury or dysfunction in populations

Risk in physical therapy can be expressed in terms of:

- Experiencing an adverse outcome

- Patients not improving with treatment

- Requiring more invasive or expensive subsequent interventions in spite of treatment

Risk usually uses what kind of designs and variables?

Observational designs with dichotomous variables

In case-control and cohort studies, what kind of variables are the IV and DV usually?

Dichotomous variables

In case-control studies, subjects are based on what?

Whether or not they have the disorder

Typically, case-control studies are _______

retrospective (works well for rare conditions)

In cohort studies, subjects are based on what?

On exposure or not

Typically, cohort studies are _________

prospective (doesn't work well for rare diseases)

What are the two measures of association and risk?

Relative risk (RR) and Odds ratio (OR); both quantify strength of association between "exposure" and "disease"

When are relative risks used?

In cohort studies

When are odds ratios used?

In case-control studies

If RR or OR is 1...

Null value, no association between exposure and a disease

If RR or OR is > 1...

Positive association between an exposure and a disease (exposure considered harmful)

If RR or OR is < 1...

Negative association between an exposure and a disease (exposure is protective)

Relative risk ratio:

Incidence of disease with exposed/Incidence of disease with unexposed

Odds ration risk:

Odds of exposure among cases/odds of exposure among controls

What is experimental event rate (EER)?

Percentage of patients in experimental group with bad outcome

What is control event rate (CER)?

Percentage of patients in control group with bad outcome

Relative risk equation with CER and EER

RR = EER/CER

What is relative risk reduction?

Chances of reduction of bad outcome (in relative terms) when using treatment option compared to control

Relative risk reduction equation

RRR = (CER-EER)/CER

What is absolute risk reduction?

Chances of reduction of bad outcomes (in absolute terms) when using treatment in comparison to control

Absolute risk reduction equation

ARR = CER-EER

What is number needed to treat (NNT)?

How many patients you have to provide treatment to in order to prevent one bad outcome (closer to 1 the better)

If the ARR is 0.....what is the NNT

Infinity

The relative risk reduction doesn't tell us anything about ________________

the actual size of effect

What is the absolute risk increase (ARI)?

When treatment poses additional risks to the patient (gives us the number needed to harm (NNH = 1/ARI))

What is the best concept to use when looking at risk?

ARR

When should you be skeptical when reading about risks?

If only the relative risk reduction is reported and no other risks are

Systematic reviews focus more on what kind of questions?

Foreground questions

Scoping reviews focus more on what kind of questions?

Background questions

What are the three types of literature reviews?

narrative, systematic, meta-analysis

Downside to narrative reviews?

Do not follow strict systematic methods like other literature reviews (prone to bias)

What are narrative reviews?

Summarize in general what is in the literature on a given topic, good source of background information

What are systematic reviews?

Rigorous process of searching, appraising, and summarizing the existing information on a selected topic

What is the goal of systematic reviews?

Aim to find all studies addressing the review's question using an objective and transparent process

First step in process of systematic reviews

Formulation of the review question

Second step in process of systematic reviews

Doing a comprehensive, systematic search and selection of primary studies

Third step in process of systematic reviews

Critical appraisal of included studies for quality and data extraction

Two grading tools to evaluate the quality of studies

PEDro and QUADAS

Fourth step in process of systematic reviews

Summarizing results

What is meta-analysis?

The statistical technique involving in extracting and combining data to produce a summary result (optional part of SR)

When can/should you do a meta-analysis:

- When more than one study has estimated an effect

- When there are no differences in the study characteristics that are likely to substantially affect outcome

- When the outcome has been measured in similar ways

What type of visual is used for meta-analysis?

A forest plot

What effect size should be used with continuous data?

Cohen's d

What effect size should be used when study's outcome measure is dichotomous?

Odds ratio (OR)

When pooling data, how does it effect sample size?

Increases sample size, which reduces the potential for type II error

If the 95% CI on a forest plot crosses the line of no effect is the study statistically significant?

No

What is "weighting" dependent on with meta-analysis?

The sample size and study quality, increase in either increases weight