PH Lecture 5: Validity and Reliability

What can medical tests help to do? (select all that apply)

1. Detect a condition 2. Determine a diagnosis 3. Plan treatment and monitor treatment effectiveness 4. Monitor a condition over time

Do screening tests, in general, prevent disease?

No - screening tests in general will not prevent disease (they detect it early)

On whom are screening tests conducted?

Generally healthy people before symptoms appear

What kind of results do screening tests provide?

Yes/no or positive/negative results

Do screening tests diagnose disease?

No - they identify patients in whom further testing should be conducted (a positive screen means they MAY have it)

What 3 qualities must a screening test have to be useful?

1. Cost-effective 2. Accurate 3. Reliable

What are examples of screening tests? (select all that apply)

1. Ultrasound 2. Lab tests 3. Genetic testing 4. Routine mammograms 5. Screening colonoscopy

According to Maxim, Niebo & Utell (2014), what is a major objective of most screening tests?

To reduce morbidity or mortality in the screened population by early detection, when treatment may be more successful

What is the mortality rate formula (screening lecture)?

# of total deaths during a specified time period / defined population at risk of dying

What is the morbidity rate formula (screening lecture)?

# of new disease cases in a specified time period / # of people in the population at risk

What is lead-time bias in cancer screening?

Screening diagnoses cancer earlier (e.g., age 60 vs. 67) so survival appears longer (5-yr survival 100% vs 0%), even though the patient dies at the same age (70)

When applying evidence-based screening guidelines, what should recommendations match?

The patient's age, risk factors, clinical context, and preventive-care setting

When applying screening guidelines, what balance should be considered?

Expected benefit vs. potential harm, test burden, false-positive results, and downstream procedures

What are diagnostic tests used for?

To help determine the cause of symptoms - designed to detect disease (may follow a positive screening test)

What are examples of diagnostic tests? (select all that apply)

1. X-ray 2. Biopsy 3. CT 4. MRI 5. Diagnostic mammogram

What factors determine how feasible a test is? (select all that apply)

1. Cost 2. Time required 3. Training required to administer 4. Probability it will give the correct results (most important)

What is the most important factor in a test's feasibility?

The probability that it will give the correct results

What is the PURPOSE of a screening test vs. a diagnostic test?

Screening = early disease detection; diagnostic = determine presence/absence of disease

What is the TARGET POPULATION of a screening test?

Asymptomatic patients at increased risk of developing a disease

What is the TARGET POPULATION of a diagnostic test?

Symptomatic patients (to determine a diagnosis) or asymptomatic patients with a positive screening result

What do screening test results indicate vs. diagnostic test results?

Screening = high or low probability of disease; diagnostic = more likely to provide a definitive diagnosis

What is a 'gold standard' test?

A diagnostic test regarded as definitive for a disease - the best-available, most reliable, and most accurate benchmark

Is the gold standard test always 100% accurate?

No - it may not be 100% accurate

What should be used if no gold standard exists for a condition?

A reference standard (still very accurate but slightly imperfect)

Type 2 diabetes: standard test vs. gold standard test?

Standard = fasting glucose; gold standard = HbA1c

Breast cancer: standard test vs. gold standard test?

Standard = mammogram; gold standard = FNA (fine needle aspiration)/biopsy with histopathology

Tuberculosis: standard test vs. gold standard test?

Standard = tuberculin test; gold standard = sputum culture

Pulmonary embolism: standard test vs. gold standard test?

Standard = D-dimer; gold standard = CTA of chest

According to the NIH, what concepts must be mastered to correctly interpret a diagnostic test?

Sensitivity, specificity, prevalence, positive and negative predictive values

How is a candidate (new) test evaluated?

A group (N) with and without disease is given both the candidate test and the gold standard test, then compared in a 2x2 contingency table

In a perfect test, what do FN and FP equal?

FN = FP = 0

⭐ In the 2x2 contingency table, what is cell 'a'?

True positive (TP) - test positive, disease present

⭐ In the 2x2 contingency table, what is cell 'b'?

False positive (FP) - test positive, disease absent

⭐ In the 2x2 contingency table, what is cell 'c'?

False negative (FN) - test negative, disease present

⭐ In the 2x2 contingency table, what is cell 'd'?

True negative (TN) - test negative, disease absent

⭐ In the 2x2 table, what do a+c and b+d represent?

a+c = all people WITH the disease (gold standard positive); b+d = all people WITHOUT the disease

⭐ In the 2x2 table, what do a+b and c+d represent?

a+b = all positive test results; c+d = all negative test results

⭐ In the 2x2 table, what is a+b+c+d?

N (total number tested)

In the zombie vs. baby problem, what is the 'gold standard'?

The subject's true identity

What is the key question when evaluating a test (zombie example)?

Not 'Is the test perfect?' but 'How does it fail?'

⭐ What is the sensitivity formula?

Sensitivity = TP / (TP + FN) = a / (a + c)

⭐ What question does sensitivity answer?

Of all people who truly HAVE the disease, how many does the test correctly flag as positive?

⭐ What does high sensitivity mean?

Few false negatives

⭐ What is the 'dangerous miss' that sensitivity guards against?

A false negative - an actual case labeled as not having the disease

⭐ What does the mnemonic SnNout mean?

When a highly SeNsitive test is Negative, it helps rule OUT disease

⭐ What is the specificity formula?

Specificity = TN / (TN + FP) = d / (b + d)

⭐ What question does specificity answer?

Of all people who truly do NOT have the disease, how many does the test correctly identify as negative?

⭐ What does high specificity mean?

Few false positives

⭐ What does the mnemonic SpPin mean?

When a highly SPecific test is Positive, it helps rule IN disease

⭐ What does sensitivity condition on vs. specificity?

Sensitivity conditions on disease status (those with disease); specificity conditions on absence of disease

⭐ CALCULATE (zombie screen): 100 beings - 20 zombies, 80 babies. Test flags 19 zombies and 8 babies as positive. What is the sensitivity?

TP = 19, FN = 1 -> 19/20 = 95%

⭐ CALCULATE (zombie screen): 100 beings - 20 zombies, 80 babies. Test flags 19 zombies and 8 babies as positive. What is the specificity?

TN = 72, FP = 8 -> 72/80 = 90%

⭐ CALCULATE (zombie screen): TP = 19, FP = 8, FN = 1, TN = 72. What is the PPV?

19 / (19 + 8) = 19/27 ≈ 70%

⭐ CALCULATE (zombie screen): TP = 19, FP = 8, FN = 1, TN = 72. What is the NPV?

72 / (1 + 72) = 72/73 ≈ 99%

What are the sensitivity and specificity of a conventional PAP smear for cervical cancer?

Sensitivity 50-70%; specificity 90-95%

What are the sensitivity and specificity of an HPV test for cervical cancer?

Sensitivity 90-95%; specificity 90-95%

What are the sensitivity and specificity of PSA (cut-off 4 ng/mL) for prostate cancer?

Sensitivity 70-80%; specificity 90-95%

What are the sensitivity and specificity of mammography for breast cancer?

Sensitivity 75-87% (up to 97% in at least 1 study); specificity 85-90%

What are the sensitivity and specificity of a rapid influenza diagnostic test?

Sensitivity 40-70%; specificity 90-95%

Which common test has the LOWEST sensitivity (40-70%)?

Rapid influenza diagnostic test

What is a Type 1 error in screening?

A false positive - the test indicates disease when the person is actually healthy (rejection of a true null hypothesis)

What are the consequences of a Type 1 error (false positive)? (select all that apply)

1. Unnecessary treatment 2. Emotional distress 3. Wasted resources/financial burden

How can Type 1 errors be minimized? (select all that apply)

1. Lower the significance level (α) 2. Use larger sample sizes 3. Conduct replication studies

What is a Type 2 error in screening?

A false negative - the test fails to detect disease that is actually present

What is the consequence of a Type 2 error (false negative)?

A person is falsely told they are healthy and does not receive necessary treatment

How can Type 2 errors be minimized? (select all that apply)

1. Increase statistical power 2. Increase sample size 3. Increase the significance level (α) 4. Use a more sensitive test

What is positive predictive value (PPV)?

The proportion of patients with positive test results who actually have the disease - if a person tests positive, the probability they have it

What is the PPV formula?

PPV = TP / (TP + FP) x 100% = a / (a + b)

What is negative predictive value (NPV)?

The proportion of patients with negative test results who do not have the disease - if a person tests negative, the probability they don't have it

What is the NPV formula?

NPV = TN / (FN + TN) x 100% = d / (c + d)

Which group had the highest PPVs for mammography?

Older women with a family history of breast cancer

What do likelihood ratios estimate?

How much a test result changes the odds of disease

What is the LR+ formula?

LR+ = Sensitivity / (1 - Specificity)

What is the LR- formula?

LR- = (1 - Sensitivity) / Specificity

Do larger or smaller LR+ values provide stronger evidence FOR disease?

Larger LR+ values

Do larger or smaller LR- values provide stronger evidence AGAINST disease?

Smaller LR- values

What are likelihood ratios useful for?

Moving from pretest to posttest probability (they combine sensitivity and specificity)

How is pretest probability estimated?

Using disease prevalence plus patient-specific history, examination, and risk factors

What makes a test clinically useful (pretest -> posttest)?

It changes probability enough to alter management

CALCULATE (lecture practice): 1,000 tested; 427 positive, 573 negative; 369 of the positives had disease; 558 of the negatives did not. Fill in the 2x2 table.

TP = 369, FP = 58, FN = 15, TN = 558

CALCULATE (lecture practice): TP = 369, FP = 58, FN = 15, TN = 558. What is the sensitivity?

369 / (369 + 15) = 369/384 ≈ 96%

CALCULATE (lecture practice): TP = 369, FP = 58, FN = 15, TN = 558. What is the specificity?

558 / (558 + 58) = 558/616 ≈ 91%

CALCULATE (lecture practice): TP = 369, FP = 58, FN = 15, TN = 558. What is the PPV?

369 / 427 ≈ 86%

CALCULATE (lecture practice): TP = 369, FP = 58, FN = 15, TN = 558. What is the NPV?

558 / 573 ≈ 97%

What is prevalence?

The proportion of individuals in a population having the disease

What is the prevalence formula from the 2x2 table?

Prevalence = (TP + FN) / N x 100%

Do sensitivity and specificity depend on disease prevalence?

No - sensitivity and specificity do not depend on the true prevalence

As prevalence decreases, what happens to PPV and NPV?

Prevalence ↓ -> PPV ↓ and NPV ↑

Why does PPV decrease as prevalence decreases?

There will be more false positives for every true positive ('needle in a haystack' - the bigger the haystack, the more things are mistaken for a needle)

Why does NPV increase as prevalence decreases?

There will be more true negatives for every false negative (a false negative is unlikely when the disease is rare)

In what kind of samples is a test's PPV higher?

Samples in which disorder prevalence is greater

On what patients should diagnostic tests be evaluated?

Patients with the same prevalence as those for whom the test will be used in the future

OCT vs. FA example: TP = 6, FP = 21, FN = 9, TN = 51. What is the sensitivity?

6/15 = 40% of those with disease were diagnosed correctly

OCT vs. FA example: TP = 6, FP = 21, FN = 9, TN = 51. What is the specificity?

51/72 = 71% of negatives were correctly identified

OCT vs. FA example: TP = 6, FP = 21, FN = 9, TN = 51. What is the PPV?

6/27 = 22% of positives actually have disease

OCT vs. FA example: TP = 6, FP = 21, FN = 9, TN = 51. What is the NPV?

51/60 = 85% of negatives are not diseased

OCT vs. FA example: TP = 6, FP = 21, FN = 9, TN = 51 (N = 87). What is the prevalence?

15/87 = 17%

What is a common strategy to show that measurements are reliable?

Repeat measurements and see how much they agree with each other

What is intrarater reliability?

Agreement when ONE person observes the same subject/specimen twice

What is interrater reliability?

Agreement when TWO or more people observe the same subject/specimen