Week 1: Appraising Evidence about Diagnostic Tests & Clinical Measures

Diagnosis

A process of integrating and evaluating the data that are obtained during the examination to describe the individual condition in terms that will guide the physical therapist in determining the prognosis and developing a plan of care.

Differential Diagnosis

A systematic process for distinguishing among two or more possible causes of a clinical disorder or condition.

Index test or measure

The diagnostic test or clinical measure of interest, the utility of which is being evaluated through comparison to a gold (or reference) standard test or measure.

Sensitivity (Sn)

The proportion of individuals with the condition of interest that have a positive test result.

Specificity (Sp)

The proportion of individuals without the condition of interest who have a negative test result.
Ability of the test to correctly identify (- test result) in someone without the disorder

Positive predictive value (PPV)

The proportion of patients or clients with a positive test result who have the condition of interest.
Ability of the test to correctly determine the % of people with the disorder from all of the people with positive test results

Prevalence

The proportion of individuals with a condition of interest at a given point in time.

Positive likelihood ratio (LR+)

The likelihood that a positive test result will be obtained in an individual with the condition of interest as compared to an individual without the condition of interest.

Negative predictive value (NPV)

The proportion of patients or clients with a negative test result who do not have the condition of interest.
Ability of the test to correctly determine the % of people without the disorder from all of the people with a negative test result

Negative likelihood ratio (LR-)

The likelihood that a negative test result will be obtained in a patient or client with the condition of interest as compared to a patient or client without the condition of interest.

Posttest probability

The odds (probability) that an individual has a condition based on the result of a diagnostic test.

Pretest probability

The odds (probability) that an individual has a condition based on clinical presentation before a diagnostic test is conducted.

Concurrent Validity

A method of criterion validation that reflects the relationship between a test or measure of interest and a measure with established validity, both of which have been applied within the same time frame.

Instrumentation

A threat to research validity characterized by problems with the tools used to collect data that may influence the study's outcomes.

Diagnosis does what

Labels patients

Classifies a problem

Determines prognosis

Determines and guides intervention

Purpose of diagnostic tests

Focus the examination on a particular body region or symptom

Identify potential problems requiring referral to a physician or other specialist

Assist in the classification process

Key thresholds for diagnostic tests

Test threshold and treatment threshold.

Normal

The range of results beyond which the target disorder becomes highly probable.

Test threshold

"The probability below which a diagnostic test will not be ordered or performed because the possibility of the diagnosis is so remote”

Treatment threshold

The probability above which a diagnostic test will not be ordered or performed because the possibility of the diagnosis is so great that immediate treatment is indicated"

Diagnostic tests in PT

Clinical examination techniques and tests performed and/or interpreted by others (e.g., radiographs, lab tests).

Measures in Physical Therapy

Techniques we perform to quantify a patient's impairment, functional limitation, or disability.

Assessment of study credibility

-Does the study of the given test compare to a reference test? (gold standard)

-Were practioners blinded to subject grouping?

-Are important details adequately described?

-Are measures of reliability discussed?

-Are measures of Validity discussed?

Did the investigators include subjects with all levels or stages of the condition being evaluated by the index test (measure)?

Determines the usefulness of the test (measure) with respect to distinguishing between important gradations in the diagnosis or impairment

-Does the study of the given test compare to a reference test? (gold standard)

Determines the usefulness of the test (measure) with respect to distinguishing between important gradations in the diagnosis or impairment

Did the investigators evaluate (or provide a citation for) the reliability of the index diagnostic test (measure)?

A necessary precondition for validity; may be tested in a separate investigation

Was there a comparison to a "gold" (reference) standard test (measure)?

Verifies the test (measure) does what it's supposed to do

Did all subjects undergo the test (measure) of interest and the gold standard test (measure)?

Addresses the issue of bias due to manipulation of the sample

Were the individuals interpreting each test's (measure's) results unaware

Addresses the issue of measurement bias

Was the time between application of the index test (measure) and the "gold standard" comparison diagnostic test (measure) short enough to minimize the opportunity for change in the subjects' condition?

Addresses the potential for misclassification (inaccurate quantification) due to natural changes in the subjects' status (e.g., healing)

Was the study repeated on a second set of subjects?

Addresses the reproducibility of the results

Additional Considerations for diagnostic studies

Is there a detailed description of the:

Clinical Setting

Inclusion criteria

Exclusion criteria

Protocol for test(s)

Reliability

The degree to which a test produces stable and consistent results.

Sources of variability for reliability

Observer

Subject

Instrument

Environment

Types of reliability measures

Standard Error of Measurement (SEM)

Correlation coefficient (e.g., Pearson r, ICC, Kappa, etc.)

ICC AND Pearson's r are for

interval and ratio data

Spearman's rho is used for

ordinal data

Validity

Measure represents what it's supposed to represent.

Types of measurement validity

All instruments

Face validity

Criterion validity

Concurrent validity

Discriminant validity

Predictive validity

Types of measurement validity

Survey instruments: everything plus...

Content validity

Construct validity

SpPin

High specificity indicates that a positive test result can rule in a disorder.
High specificity + Positive test result = Rule in disorder

Sensitivity (SnNout)

High sensitivity indicates that a negative test result can rule out a disorder.

Predictive values depend on

prevalence

LRs are independent of

Prevalence

LRS can be applied to

individuals

Formula for sensitivity

Patients with the disease who test positive/all patients with the disease

a/a+c

Formula for specificity

Patients without the disease who test negative/ all patients without the disease

d/b+d

Formula of PPV

Patients with the disorder who test positive/ all patients who test positive

PPV = a/(a+b), where a is true positives and b is false positives.

Formula for NPV

Patients without the disorder who test negative/ all patients who test negative

NPV = d/(d+c), where d is true negatives and c is false negatives.

Formula for PLR

sensitivity/(1-specificity)

(a/a+c)/(1-(d/b+d))

Formula for NLR

(1- sensitivity)/specificity

(1-(a/a+c))/(d/b+d)

Receiver operating characteristic curve

A graphic way to evaluate different test scores with respect to the number of true-positive and false-positive results obtained at each threshold or cut point

Y-axis of ROC curve

Sensitivity

X-axis of ROC curve

1 − Specificity

Area under the curve (AUC)

Reflects diagnostic usefulness

Pre-test probability

What you think the odds are that the patient has the disorder before you conduct the diagnostic test

Post-test Odds

What you think the odds are that the patient has the disorder after you conduct the diagnostic test

Post-test Probability

Probability of the disorder once the test results are obtained

LR+ > 10 or LR - < 0.10 indicates

large and conclusive change

LR+ = 5-10 or LR- = 0.10-0.20

moderate change

LR+ = 2-5 or LR- = 0.20-0.50

Small but sometimes important change

LR+ = 1-2 or LR- = 0.50-1.0

Negligible change in pre-test probability

Nomograms

Visual tool to convert: Pretest probability → Posttest probability

P-value

The probability that the result occurred due to chance

95% Confidence Interval (C.I.)

The range of values within which the true value is estimated to lie within a 95% probability

If an LR = 1.0 it represents a

50:50 chance of increasing or decreasing the probability of a diagnosis

Should you use this evidence?

Is the study high quality (e.g., does the design minimize bias)?

Are the results important enough to use?

Is the test or measure of interest available, practical, and safe for application in the clinical setting?

Was your patient or client represented in the study sample?

Can you estimate the pre-test probability of the disorder?

Patient's or client's values and preferences re:

Diagnostic tests purpose in PT

Help focus the examination process to a particular body region or system

Fatal flaw in research quality

If not all subjects receive the reference standard, accuracy is inflated

Limitations of Traditional Test Interpretation

Sensitivity and specificity are population-based, not patient-specific

They do not tell you how much probability changes for an individual patient

Preferred diagnostic test measurement

Likelihood ratios

Decision thresholds

Below test threshold → don't test; Above treatment threshold → treat

Tests with LRs close to 1.0

Are essentially useless

Core purpose of diagnostic tests

Should not be used in isolation; they refine clinical judgement but don't replace it

Strategies for control of unwanted factors (minimizing bias)

Clear inclusion/exclusion criteria

Subjects suspected to have condition of interest

Clear performance protocols

Include criteria for positive/negative test

Verify examiner's competence

Verify presence/absence of condition of interest via comparison test

Masking of examiners

Observed score (X)

X = true score (T) and an error component (E).

Sources of measurement error

Rater, instrument used, patient/subject being measured, challenge when response is inherently unstable.

Intra-rater

Measurement error within individuals.

Inter-rater

Measurement error between individuals.

Pearson's product moment correlation (r)

Statistical method used for interval/ratio data to evaluate reliability of interval or ratio data

ICC

Intraclass correlation coefficient, used to assess reliability of interval or ratio data

Spearman's rho (p)

Statistical method used for ordinal data to evaluate reliability

Kappa (k)

Statistical method used to assess agreement between raters of nominal data

kappa 0-0.25

little to no change

kappa 0.26-0.5

fait degree of relationships

kappa 0.51-0.75

moderate to good degree of relationship

kappa 0.76-1.00

good to excellent degree of relationship

2x2 tables

chi square statistic indicated an association between two variables; simpler analysis than linear regression --> easier to explain to policymakers but loose a lot of information through simplification

High sensitivity means

high rate of false positives

High specificity means

higher false negatives

Receiver Operating Characteristic (ROC) curves

Used to determine cut points for sensitivity and specificity.

Sensitivity and specificity have some limitations

Indicate a test's performance in individuals whose status is known

Reduce information to two options based on cut point for positive/negative

ROC Y-axis

true positive/ sensitivity

ROC X-axis

1-specificity

true negatives

Likelihood ratios

Mathematical calculations that reflect a diagnostic test's ability to provide persuasive information.

P value

Probability that result occurred due to chance.

Advantages of LRs

Ratios can be calculated for all levels of test results

Not dependent on prevalence of the condition

Can be applied to individuals

LR of 1.0

Value of 1.0 = no added info, no better than chance; coin flip

Confidence interval

Range of scores within which the true score is estimated to lie within a specified probability.

Clinical measures

Used to quantify and describe a patient's impairments in body functions and structures.