Large Language Models as Clinical Decision Support Tools in Genetics

Background on Genetic Services and Workforce Shortages

The integration of artificial intelligence into healthcare is occurring rapidly, specifically regarding its potential to support clinical decision-making. Within the field of genetics, there is a critical need for such support due to significant workforce shortages. Currently, many families face wait times of six months or longer for an initial genetics appointment. There are more than 100100 unfilled geneticist positions across the United States. Training programs are unable to keep pace with the demand, and more than a third of currently practicing geneticists plan to retire within the next decade. These factors limit access to evaluations, testing, and follow-up care.

Because of this shortage, the responsibility for genetic care is increasingly shifting toward nongenetics providers (NGPs). A national survey indicated that 72 %72\, \% of nephrologists have ordered genetic testing in their practice, yet many described their genetics education as limited. In cardiology, over 60 %60\, \% of practitioners reported a lack of confidence in selecting the correct tests or applying results to patient care. Primary care physicians similarly report feeling underprepared and undersupported, which frequently leads to delayed diagnoses, missed opportunities for early intervention, and an increase in inequitable care.

Existing Solutions and Clinical Decision Support

Several strategies are being explored to address the gaps in genetic healthcare. One approach is workforce expansion, which involves increasing residency and fellowship positions and creating incentives for providers. However, since specialist training takes years, expansion alone is considered insufficient to meet rising demand. Another strategy involves restructuring delivery care models through telogenetics, group counseling, and embedding genetic counselors directly into specialty clinics to improve efficiency and reduce geographic barriers.

A major focus is also placed on improving NGP readiness through better education and clinical decision support (CDS) tools. Providers face challenges including unclear clinical guidelines, limited specialist access, and short patient visits that make detailed family histories difficult to obtain. CDS tools, such as automated prompts within the electronic health record (EHR), can flag patients for genetic evaluation. For example, individuals with BRCA-related cancers or Lynch syndrome are often underidentified; EHR-integrated systems can assist providers in recognizing these patients in real-time. Despite these developments, large language models (LLMs) remain a relatively underexplored area for improving genetic referral decisions.

Large Language Models in Clinical Genetics

Large language models are computer programs trained on vast datasets, including books, journals, websites, and articles, to learn patterns in language. LLMs like ChatGPT, Gemini, Meta AI (Llama), and Claude operate by predicting and generating language based on these learned patterns. They can answer questions, summarize information, and explain complex concepts in a human-like manner. However, it is essential to note that these models do not reason or understand medical context in the way a clinician does. Their outputs are based on statistical patterns and require careful human interpretation and oversight.

Previous literature has informed the exploration of LLMs in medical contexts. A study by Hirasawa and colleagues found that ChatGPT-4 achieved diagnostic accuracy on complex internal medicine vignettes comparable to physicians. Wong and colleagues compared ChatGPT-3.5 and ChatGPT-4 to pediatric dermatologists, finding that while specialists performed better overall, ChatGPT-4 approached specialist-level accuracy in multiple-choice and case-based questions. Young and colleagues evaluated LLMs on rare pediatric diseases, finding that while overall diagnostic accuracy was low, the models could often identify the correct disease category or include the diagnosis in a differential list. These studies suggest that LLMs could potentially serve as tools for differential diagnosis support.

Study Objectives and Methodology

The primary objective of this study was to test whether LLMs could generate accurate diagnoses and differentials for real-world clinical vignettes and to compare this performance to board-certified clinical geneticists. The study used Qualtrics to deliver an online quiz containing 4040 clinical vignettes developed by geneticists from actual cases. These vignettes covered pediatric and adult genetics, featuring classic textbook syndromes, atypical presentations, and "negative" cases where no genetic diagnosis was present. Key features included family history, physical exam findings, and neurological or metabolic symptoms.

Recruitment targeted board-certified and board-eligible clinical geneticists through the American College of Medical Genetics and Genomics (ACMG) and the American Board of Medical Genetics and Genomics (ABMGG). A total of 1414 geneticists completed the study. The models tested were ChatGPT 3.5, Llama, and Gemini. Each model was prompted as follows: "Consider you are a clinical geneticist. Here is a case vignette. Can you answer the following: 1. What is the most likely diagnosis? 2. Excluding your first answer, list your top three differential diagnoses."

Scoring Rubric and Statistical Analysis

Responses were scored by three independent clinical geneticists using a tiered rubric:

  • 22 points: Correct diagnosis ranked first (most likely).

  • 1.51.5 points: Correct diagnosis listed first in the differential (if top answer was wrong).

  • 11 point: Correct diagnosis appeared second or third in the differential.

  • 0.50.5 points: Adjacent conditions ranked first.

  • 00 points: Diagnosis missed entirely.

Statistical analysis included the Mann-Whitney U test to compare LLMs versus geneticists and the Friedman test to evaluate performance differences among the three LLMs. The geneticist cohort had an average age of 4848 years (range 3535 to 6565). Over 50 %50\, \% had more than 1010 years of clinical experience, with an overall mean of approximately 1212 years. Participants specialized in areas such as inborn errors of metabolism, neurodevelopmental disorders, and connective tissue disorders.

Comparative Performance Results

The LLMs achieved a mean total diagnostic score of 62.662.6 out of 8080, while the clinical geneticists achieved a mean score of 53.453.4 out of 8080. This difference was statistically significant with a p-value of 0.050.05. In terms of specific diagnostic accuracy, LLMs correctly ranked the primary diagnosis first in 68 %68\, \% of cases, compared to 57 %57\, \% for human geneticists. Furthermore, the LLMs had a lower rate of complete diagnostic misses at 14 %14\, \% compared to the geneticists' 20 %20\, \%.

Among the specific models, Gemini demonstrated the strongest performance with the highest diagnostic score and lowest miss rate. ChatGPT 3.5 had the lowest diagnostic score and highest miss rate among the LLMs. Llama showed intermediate performance, though it was noted for having a strong differential list accuracy. Despite these trends, the differences in performance between the three LLMs were not statistically significant. Within the geneticist group, there were no significant differences in performance based on years of experience (>12>12 years vs. <12<12 years) or specialty (metabolics vs. non-metabolics).

Implications for Clinical Practice

The findings suggest that LLMs are capable of recognizing clinical patterns across disparate features—such as physical findings and developmental concerns—and connecting them to known genetic conditions. This is particularly effective for well-characterized disorders. Even when the top diagnosis is incorrect, the inclusion of the correct condition in a differential list can prompt necessary actions like ordering tests or making referrals. LLMs could be particularly valuable for NGPs in settings where specialist access is limited.

However, the study emphasizes that LLMs should be viewed as support tools rather than diagnostic authorities. They lack the ability to handle the human aspects of medicine, such as helping families navigate ethical considerations, communicating uncertainty, and providing emotional support. Clinical judgment remains the "gold standard," and LLM outputs must be verified against evidence-based guidelines and trusted resources.

Study Limitations

Several limitations were noted by the researchers. First, the sample size was small (n=14n = 14), which limits the generalizability of the results and the ability to detect subtle subgroup differences. Second, researchers could not verify if human participants consulted outside sources while completing the Qualtrics survey. Third, clinical vignettes, while diverse, cannot fully capture the complexity of in-person evaluations involving dynamic family histories or evolving symptoms. Fourth, the scoring process involves inherent subjectivity, though a structured rubric and multiple raters were used to mitigate bias. Finally, LLMs evolve extremely quickly; a model tested several months ago may perform differently than the version available today, meaning the relationship between clinicians and these tools must be dynamic.

Future Directions and Research Phases

Future research will focus on understanding how specific prompting strategies—including wording, structure, and the order of clinical details—influence diagnostic output. There is a need for standardized prompting frameworks to improve consistency. Additionally, evaluating LLM performance in "messy" real-world contexts involving incomplete histories and conflicting findings is essential. Providers also need formal training modules and AI literacy guidelines to help them identify inaccuracies or "hallucinations" in model outputs.

Phase two of this study will be a real-world feasibility study. It will evaluate whether LLMs, supported by prompt manuals and provider training, can help pediatricians more accurately identify when a referral to a general genetics clinic is appropriate. The ultimate goal is to extend the reach of genetics expertise and ensure patients receive timely care while keeping the patient at the center of the medical process.

Questions & Discussion

Audience Question: Did the human respondents know that they were being compared to LLMs when they answered the questions? Ivy: Yes, they were informed of the central goal during the disclosure. They were aware that the objective was to compare their answers to those of a large language model.

Audience Question: Did you consider a third category of clinicians using an LLM? Studies often show that humans plus AI are more effective than either alone. Ivy: Initially, we wanted to focus on establishing the baseline by comparing LLMs to the gold standard of geneticists. However, we are considering that for phase three of the research, comparing responses from clinicians using LLMs as support to those without.

Audience Question: Were there any HIPAA or PHI protections for the LLMs? Also, why do you think ChatGPT performed the worst? Ivy: None of the vignettes contained identifying features, so there were no HIPAA violations. As for ChatGPT's performance, it may be due to the quality of the resources it pulls from compared to other models. Each LLM has unique strengths and limitations despite sharing a similar underlying mechanism.

Audience Interaction: Results from the audience poll indicated feelings of being "hesitant," "excited," and "uncertain." The speaker noted that "uncertain" is a common response, highlighting that the future of AI in medicine depends on how clinicians and researchers choose to use the technology responsibly.