Diagnostic Accuracy Studies – 9.1

Diagnosis: Concepts and Definitions

  • Diagnosis = systematic process of determining the nature of a disorder or disease. This involves synthesizing various data points: patient-reported symptoms (subjective experiences), clinician-observed signs (objective findings), comprehensive patient history (including medical, social, and family history), and, when necessary, objective laboratory or imaging results.

  • Differential diagnosis = a methodical process of ruling out alternative conditions that present with similar clinical features. This involves considering all plausible diagnoses before narrowing down to the most likely one.

  • Clinically grounded in medicine, these diagnostic principles are broadly applicable and generalise across various healthcare disciplines, including nursing and allied-health professions, where accurate identification of health conditions is crucial for effective care.

Diagnosis vs. Assessment & Measurement

  • Diagnosis is a critical component within a broader, comprehensive patient assessment process. It's the specific determination of a condition's presence or absence.

  • Purposes:

    • Rule-in health conditions actually present: confirming the existence of a specific disease or disorder.

    • Rule-out conditions not present: excluding other potential conditions that might mimic the symptoms.

  • Accurate diagnosis requires structured observation (e.g., physical examination, gait analysis), detailed interview techniques (e.g., symptom characterisation, patient narrative), and the judicious use of specific tests (e.g., blood tests, imaging, functional assessments). The evidence gathered guides the formulation of appropriate management plans.

  • Measurement level:

    • Nominal/binary: The simplest form, classifying a condition as either present or absent (e.g., 'True' or 'False' for a disease).

    • Sometimes ordinal: For conditions with varying severity, diagnosis might involve a spectrum or scale (e.g., mild, moderate, severe autism spectrum disorder).

  • Validity imperative: Every diagnostic decision is fundamentally a measurement of whether a condition exists. Therefore, for a diagnosis to be trustworthy, the diagnostic process and the tests used must be both reliable (consistent results upon repeated measurement) and valid (accurately measuring what they intend to measure).

Sources of Evidence in Diagnosis

  • Subjective: Information reported by the patient or their caregivers, reflecting their internal experiences. Examples include pain level ("It hurts"), fatigue, dizziness, or emotional states. These are crucial but dependent on accurate patient recall and self-perception.

  • Objective: Measurable and observable data collected by the clinician or through diagnostic tests. This includes clinician-observed signs (e.g., rash, swelling, abnormal reflexes, fever measured by a thermometer), results from functional tests (e.g., range of motion, muscle strength), quantitative data from imaging (e.g., X-rays, MRI scans), and laboratory pathology results (e.g., blood cell counts, biopsy findings).

  • Not always one-to-one mapping between a single sign/symptom and a specific disorder: Many conditions share common symptoms, leading to diagnostic uncertainty. For example, a headache can be a symptom of many different conditions, from minor tension to severe neurological issues. This necessitates a differential diagnosis process.

Importance & Consequences of Diagnostic Accuracy

  • Diagnostic accuracy directly dictates the selection of correct treatment strategies, appropriate ongoing management plans, and reliable prognosis for the patient.

  • An incorrect diagnosis can lead to several serious consequences:

    • Wrong or missed treatment: Instituting therapies for a condition not present, or failing to treat the actual condition, which can worsen patient outcomes.

    • Wasted resources: Unnecessary tests, procedures, and medications, placing burdens on both the patient and the healthcare system.

    • Potential harm: Including adverse drug reactions from inappropriate medications, complications from unnecessary procedures, or progression of an untreated disease.

  • Epidemiological impact of diagnostic error (Scott & Crock 2020):

    • An estimated 8\%-15\% of US hospital admissions involve some form of diagnostic error. These errors can stem from cognitive biases (e.g., anchoring bias, premature closure), system failures (e.g., inadequate communication, workflow issues), or lack of clinician knowledge.

    • In Australia, approximately 140{,}000 diagnostic errors occur annually, leading to about 21{,}000 cases of serious harm and 2{,}000{-}4{,}000 avoidable deaths. These figures highlight the significant public health burden of diagnostic inaccuracy.

    • Up to 50\% of malpractice claims against General Practitioners (GPs) involve diagnostic error, with more than 80\% of these errors deemed preventable through better diagnostic processes and systemic improvements.

Scope & Aims of Module

  • This module does not aim to teach "how to diagnose" specific medical conditions (e.g., pneumonia or diabetes).

  • Instead, it focuses on the universal principles that underpin diagnostic accuracy, including core concepts in research design pertinent to diagnostic studies, statistical indices used to quantify test performance, methods for interpreting diagnostic results in clinical practice, and their practical and ethical implications.

Diagnostic Accuracy Research: Core Ideas

  • The fundamental clinical question, "Is this test valid and useful for diagnosis?" translates directly into a research question that can be investigated through specific study designs.

  • Gold-standard design = Diagnostic Accuracy Study: This is the strongest design for evaluating a new diagnostic test. It involves comparing:

    • Reference (gold-standard) test: An established, highly accurate diagnostic method that is accepted as the true measure of a condition's presence or absence. It represents the "ground truth."

    • Index test: The new or existing test under evaluation whose diagnostic performance is being assessed.

  • Agreement between the results of the index test and the reference test quantifies the accuracy of the index test. High agreement indicates high accuracy.

  • Diagnostic Yield Study: A weaker form of evidence where there is no established reference test available or used. These studies typically report the proportion of positive test results in a given population without verifying false positives or negatives, offering limited insight into accuracy.

Terminology: Case/Control vs Positive/Negative

  • Reference test labels: Determined by the gold standard.

    • Case: An individual who truly has the condition (condition present).

    • Control: An individual who truly does not have the condition (condition absent).

  • Index test labels: The result of the test being evaluated.

    • Positive: The index test result predicts the presence of the condition (predicts case).

    • Negative: The index test result predicts the absence of the condition (predicts control).

Hierarchy of Evidence for Diagnostic Studies

This hierarchy ranks diagnostic study designs by their strength in minimizing bias and providing reliable evidence:

  1. Level I – Systematic Review (SR) of Level II studies: Highest quality, combining findings from multiple rigorous primary studies.

  2. Level II – Prospective, consecutive cohort study: Involves following a group of subjects over time, enrolling them consecutively (to avoid selection bias), and applying both the index and reference tests independently and blinded (to prevent incorporation and observer bias). This is considered the gold standard for primary diagnostic accuracy research.

  3. Level III-1 – Non-consecutive cohort study: Similar to Level II but susceptible to selection bias if subjects are not enrolled consecutively.

  4. Level III-2 – Diagnostic case-control study: Compares groups already known to have or not have the condition. Prone to verification bias and can lead to inflated accuracy estimates because the prevalence of the disease is artificially set by the researchers, not reflecting real-world prevalence.

  5. Level IV – Diagnostic yield study: Weakest evidence as it lacks a reference test, only reporting how many times a test is positive for a condition without confirming true positive/negative rates. No direct measure of accuracy.

  • Most diagnostic studies employ a cross-sectional design, measuring the presence of a condition and test results at a single point in time. In contrast, prognostic studies are longitudinal, following individuals over time to predict future outcomes.

Threats & Biases in Diagnostic Studies (Hoffmann et al.)

Bias can systematically distort the perceived accuracy of a diagnostic test:

  • Incorporation bias: Occurs when the results of the index test are used, even implicitly, as part of the reference standard definition or interpretation, leading to an artificially inflated agreement between the two tests.

  • Verification bias: A significant bias where only a subset of the participants (often those with positive index test results or clinically suspicious cases) receives the costly or invasive reference test, while others do not. This can lead to an overestimation of sensitivity and an underestimation of specificity.

  • Non-consecutive sampling: Recruiting participants in a way that is not representative of the target population for the test, introducing selection bias and potentially altering the observed prevalence or spectrum of disease, which in turn affects test performance metrics.

  • Partial verification when reference test is invasive/harmful: A specific type of verification bias. If the reference test is painful or risky, it may only be applied to symptomatic or index test-positive individuals. This often leads to missing asymptomatic controls in the study, which can result in an inflated sensitivity (Sn) and a decreased specificity (Sp), as true negatives may be misclassified or missed due to lack of verification.

Two-by-Two Contingency Table

This table is fundamental for calculating diagnostic accuracy measures:

Reference Case

Reference Control

Index Positive

True Positive (TP)

False Positive (FP)

Index Negative

False Negative (FN)

True Negative (TN)

  • True Positive (TP): The index test correctly identifies the condition as present when it is actually present.

  • False Positive (FP): The index test incorrectly identifies the condition as present when it is actually absent (Type I error).

  • False Negative (FN): The index test incorrectly identifies the condition as absent when it is actually present (Type II error).

  • True Negative (TN): The index test correctly identifies the condition as absent when it is actually absent.

Core Measures & Formulae

These metrics quantify different aspects of a diagnostic test's performance:

  • Sensitivity (Sn): The ability of a test to correctly detect cases; the proportion of true positives among all individuals who truly have the condition.

    Sn = \frac{TP}{TP+FN}

  • Specificity (Sp): The ability of a test to correctly identify controls; the proportion of true negatives among all individuals who truly do not have the condition.

    Sp = \frac{TN}{TN+FP}

  • Positive Predictive Value (PPV): The probability that an individual truly has the condition, given a positive index test result.

    PPV = \frac{TP}{TP+FP}

  • Negative Predictive Value (NPV): The probability that an individual truly does not have the condition, given a negative index test result.

    NPV = \frac{TN}{TN+FN}

  • Positive Likelihood Ratio (LR+): Indicates how many times more likely a positive result is among those with the disease compared to those without the disease. High LR+ (>>1) strongly suggests the presence of the disease.

    LR^+ = \frac{TP \; / \; (TP+FN)}{FP \; / \; (FP+TN)} = \frac{Sensitivity}{1-Specificity}

  • Negative Likelihood Ratio (LR–): Indicates how many times more likely a negative result is among those with the disease compared to those without the disease. Low LR- (<<1) strongly suggests the absence of the disease.

    LR^- = \frac{FN \; / \; (TP+FN)}{TN \; / \; (FP+TN)} = \frac{1-Sensitivity}{Specificity}

  • Conventional interpretive cut-points for Likelihood Ratios:

    • LR^+ > 10 or LR^- < 0.1 are generally considered excellent for ruling in or ruling out a disease, respectively, leading to large shifts in post-test probability.

    • LR^+ > 2 or LR^- < 0.5 suggest a good ability to rule in or rule out, causing moderate shifts in probability.

Interpreting Measures

  • High Sn (Sensitivity): Indicates that the test rarely misses individuals with the disease (few False Negatives). This makes it a good "rule-out" test: if the test is negative, it's highly likely the person does not have the condition, as it detects most cases (think of a sensitive screening test).

  • High Sp (Specificity): Indicates that the test rarely incorrectly identifies individuals without the disease as having it (few False Positives). This makes it a good "rule-in" test: if the test is positive, it's highly likely the person truly has the condition, as it's specific to the disease (think of a confirmatory diagnostic test).

  • High PPV (Positive Predictive Value): Means that most individuals who test positive genuinely have the disease. It's profoundly impacted by disease prevalence: in low prevalence settings, even a highly specific test can have a low PPV.

  • High NPV (Negative Predictive Value): Means that most individuals who test negative genuinely do not have the disease. Less influenced by prevalence than PPV, but still affected.

  • Trade-off: There is an inherent inverse relationship between sensitivity and specificity when adjusting the diagnostic threshold of a continuous test. Raising the cut-off point (making it harder to test positive) will increase specificity but decrease sensitivity, and vice-versa. Clinicians must balance the risks of false positives against false negatives based on the clinical context.

ROC Curves

  • Receiver Operating Characteristic (ROC) curves visually represent the trade-off between sensitivity and specificity across all possible cut-off points for a diagnostic test with continuous results.

  • They plot the True Positive Rate (Sensitivity) on the y-axis against the False Positive Rate (1{-}Specificity) on the x-axis.

  • Area Under Curve (AUC): A single scalar value derived from the ROC curve that quantifies the overall diagnostic accuracy of the test:

    • 0.5 = Indicates a test that performs no better than chance (diagonal line).

    • 1 = Represents a perfect test (a curve that goes straight up to the top-left corner and then across).

  • A curve closer to the upper-left corner of the plot indicates a superior test, as it achieves higher sensitivity for any given level of false positive rate, or higher specificity for any given level of sensitivity.

Decision Thresholds, Prevalence & Trade-offs (Key Principles)

  • Lower cut-off: Setting a lower diagnostic threshold (e.g., a lower blood sugar level defining diabetes) makes it easier for an individual to test positive. This will generally increase Sensitivity (more true cases detected) but decrease Specificity (more healthy individuals might be misclassified as positive).

  • Higher cut-off: Conversely, setting a higher threshold makes it harder to test positive, which tends to decrease Sensitivity (more true cases missed) but increase Specificity (fewer healthy individuals misclassified).

  • Prevalence influences PPV & NPV directly: In populations with high disease prevalence, the PPV of a test will naturally be higher because the 'prior probability' of the disease is already high. Conversely, in low prevalence populations, even good tests can have low PPV and high NPV. Understanding prevalence is crucial for interpreting individual test results.

  • Optimal threshold: The ideal cut-off point is often chosen by balancing various factors. Youden’s J statistic (=Sn+Sp-1) is a common metric that identifies the point on the ROC curve farthest from the chance line, aiming to maximize both sensitivity and specificity. However, the optimal threshold can also be determined by considering specific cost/benefit implications (e.g., the cost of a false positive vs. a false negative) or the patient’s preferences and risk tolerance.

Demonstration Case Study: BMI, Abdomen & Ankle (n = 252 males)

  • Population characteristics: Approximately 50\% of the male participants were classified as overweight (BMI\ge25), 10\% as obese (BMI\ge30), and less than 1\% as morbidly obese (BMI\ge40).

  • Correlations:

    • Abdomen circumference vs. BMI: This showed a very strong positive correlation (r=0.92), indicating that as BMI increased, abdomen circumference also consistently increased, suggesting abdomen circumference is a good proxy for BMI.

    • Ankle circumference vs. BMI: Exhibited only a moderate correlation (r=0.50), implying a less consistent relationship and thus a poorer proxy for BMI.

  • Testing overweight (using BMI as the reference standard):

    • Ankle cut-off at lower 25% (22 cm): This threshold yielded a high Sensitivity of 90\% (good at detecting overweight individuals) but very low Specificity of 34\% (many healthy individuals were incorrectly flagged as overweight, leading to many False Positives). This would be suitable for a screening test where missing cases is undesirable.

    • Ankle median cut-off: A more balanced performance with Sensitivity of 70\% and Specificity of 68\%. While better balanced, it's still not highly accurate.

    • Abdomen median cut-off (\approx90 cm): This test demonstrated the best overall performance with high Sensitivity (87\%) and high Specificity (88\%). The calculated LR+ of 7.3 (good for ruling in) and LR– of 0.14 (excellent for ruling out) confirm its strong diagnostic utility. This indicates that abdominal circumference is a much more effective and balanced diagnostic indicator for overweight status than ankle circumference in this population.

  • Testing obesity:

    • Using the same abdomen threshold (\approx90 cm) that was effective for overweight status produced perfect Sensitivity (it caught all obese individuals) but had low Specificity (many non-obese individuals were flagged as obese). This highlights that a diagnostic threshold must be specifically calibrated for the condition and population being diagnosed. For obesity, the threshold for abdomen circumference needed upward adjustment (a higher cut-off) to improve specificity and reduce false positives.

  • Lesson: This case study clearly demonstrates that the validity of a diagnostic test (e.g., abdomen circumference being superior to ankle circumference as an indicator of BMI) combined with an appropriately calibrated threshold for a specific condition yields the highest diagnostic accuracy.

Screening vs Diagnostic Testing

These are two distinct phases of identifying health conditions, differing in their purpose, target population, and acceptable error rates:

  • Screening: Aims for early detection of potential disease indicators in a large, generally asymptomatic (healthy-appearing) population. The priority is to maximize Sensitivity (minimize False Negatives) to ensure as many cases as possible are identified, even if it means tolerating a higher rate of False Positives. Individuals with a positive screen then proceed to more definitive diagnostic testing.

  • Diagnostic testing: Performed to confirm or rule out a specific disease in symptomatic individuals or those who have screened positive. The priority here shifts towards maximizing Specificity and achieving definitive classification, as these results directly influence treatment decisions.

  • Operational differences (as often summarized in Healthknowledge.org table):

    • Population: Screening targets entire asymptomatic populations (e.g., mammograms for all women over 40); Diagnostic testing targets symptomatic individuals or those with positive screening results (e.g., biopsy after abnormal mammogram).

    • Invasiveness: Screening tests are typically non-invasive, low-cost, and easy to administer; Diagnostic tests can be more invasive, costly, and complex.

    • Cost: Screening is cost-effective for large populations because it's widespread and initial tests are cheap; Diagnostic tests are typically more expensive per individual but are justified by the need for definitive answers.

    • Acceptable FP/FN ratios: Screening prioritizes minimizing FN (missing cases, high Sn) so high FP is more acceptable; Diagnostic testing prioritizes minimizing FP (misdiagnosing healthy, high Sp) as a false diagnosis can lead to harm.

Over-diagnosis & Threshold Creep

  • Over-diagnosis refers to diagnosing a disease that would never have caused symptoms or harm during a person's lifetime. This can occur due to expanding disease definitions or increasingly sensitive tests.

  • Expanding disease definitions (e.g., lowering the cut-off for hypertension (HTN) or chronic kidney disease (CKD)) can artificially inflate disease prevalence. This "threshold creep" creates a larger pool of "patients" who might not benefit from treatment and could instead be harmed by unnecessary interventions (over-treatment) or experience psychological distress and anxiety from being labeled with a chronic condition.

  • There is a critical need for transparent, evidence-driven threshold setting, free from vested interests (e.g., pharmaceutical companies, medical device manufacturers) that might benefit from expanding disease definitions.

Emerging Example: Infra-red Thermometers for COVID-19

  • Early concerns (Aw 2020) highlighted that infra-red thermometers had a low Sensitivity (approx. 29\%), meaning they missed a significant proportion of infected individuals (many False Negatives). This was particularly problematic given that fever is not universally present or can be transient in COVID-19, and individuals could be infectious while asymptomatic or afebrile.

  • Subsequent meta-analysis (Aggarwal 2020) showed "reasonable" accuracy, especially concerning their Negative Predictive Value (NPV) during the pandemic. While not perfect diagnostically, a negative temperature reading had a good chance of truly indicating no fever, which was useful in high-prevalence settings for initial triage. However, the study also stressed that operator technique (e.g., correct measurement distance, environmental factors) was critical to maintaining accuracy.

Practical & Ethical Implications

  • Cut-offs should be empirically evidence-based, derived from robust, large-scale diagnostic accuracy studies that reflect the target population and clinical context.

  • Clinicians must carefully consider the patient harm hierarchy: Which type of error (False Positive vs. False Negative) carries greater detriment to the patient? For life-threatening but treatable conditions (e.g., certain cancers), minimizing False Negatives (high Sensitivity) is often prioritized, even if it means accepting more False Positives (leading to further testing). For conditions where treatment carries significant risks, minimizing False Positives (high Specificity) might be prioritized.

  • Clinician bias or conflict of interest (e.g., financial incentives to over-diagnose or order more tests) can severely undermine the validity and ethical application of diagnostic principles, leading to patient harm and resource waste.

  • Proper test administration techniques and targeting the test to the correct population (i.e., the population for which the test's accuracy has been validated) are essential to preserve the diagnostic accuracy established in research settings. Misapplication of tests can lead to erroneous results regardless of the test's inherent properties.

Key Takeaways

  • Diagnostic accuracy is a complex interplay involving the inherent validity of the test, the quality of its execution, the chosen decision thresholds, and the prevalence of the condition within the population being tested.

  • Four primary metrics—Sensitivity (Sn), Specificity (Sp), Positive Predictive Value (PPV), and Negative Predictive Value (NPV)—along with Likelihood Ratios (LR+, LR-) and ROC curve analyses, are used to quantify test performance. Each metric offers distinct clinical insights into a test's strengths and limitations.

  • The research hierarchy, typically ranging from high-confidence Level II prospective cohort studies to less reliable Level III-2 case-control studies and Level IV diagnostic yield studies, guides the confidence clinicians should place in published diagnostic evidence.

  • Various biases, including incorporation bias, verification bias, and sampling bias, can significantly inflate the perceived accuracy of a diagnostic test reported in studies, leading to misleading conclusions.

  • Optimal clinical practice involves selecting highly valid diagnostic tests, applying empirically determined and context-appropriate thresholds, and consciously balancing the risks associated with False Positives versus False Negatives in a patient-centred manner.