L7 - Medical Information from the Internet and AI

What are some tips to identifying quality websites?

Use common sense criteria such as determining if it has authoritative authorship, up to date information, accuracy of information, little bias, etc.

What are some characteristics of online medical information intended for healthcare professionals over patients?

Professional websites (e.g., UpToDate, Lexicomp, PubMed) use technical language, detailed clinical data, and are aimed at helping providers make evidence-based decisions.

What are some characteristics of online medical information intended for patients over healthcare providers?

Patient-focused websites (e.g., Mayo Clinic, MedlinePlus, WebMD) use simpler language, are more accessible, and focus on helping patients understand conditions, treatments, and preventive care.

What questions can you ask to determine how current the information is on a website?

When was the original information posted?

How frequently is it updated?

When was the last content update?

Give examples of different types of information a website could be providing

Promotion, advertising, and serious content

What is something that should be done to verify information on a website?

Verify the information using other trusted sources and explore citations used referenced in the website

Evidence resources

Provides citations for original research, systematic reviews, etc. in order for the reader to verify the claims presented

Information resources

May or may not provide evidence to support claims presented. The author is relaying (common) information with no evidence listed

What questions can you ask to determine the level of evidence used in a particular webiste?

What evidence is provided?

Is the evidence from a respectable, peer-reviewed medical publication?

Do the studies referenced back up what the author is saying?

What is some information to be aware of when examining links used in a website?

Watch for outdated links, should be current/live/accessible, and determine if other sites provide links to the one you're reviewing

What do outdated links indicate when evaluating the credibility of information on a website?

They are not keeping up with current information

What is some information to be aware of when examining references used in a website?

What kind of references are listed, the kind of information being referenced, where the information came from (journal, newspaper, etc.), etc.

What is the DISCERN instrument?

A questionnaire with 3 sections used to evaluate the quality of written health information, especially about treatment options. For each question, you choose a number from 1 to 5 based on how well the material meets that specific criterion.

Name four General Purpose AI sources

Google Gemini, Microsoft Copilot, Anthropic Claude 2, and Google LaMDA

Name two Healthcare Purpose AI sources

Your.MD and InPharmD

Your.MD

Triage symptoms and connects them to healthcare resources

InPharmD

Uses a combination of AI and human expertise to answer complex questions about medications

Name five common biases seen in AI responses

Selection, Confirmation, Recall, Group Attribution, and Implicit Biases

Selection Bias

Occurs when the training data used to develop an AI system is not representative of the real-world population or scenarios it will be applied to. This can lead to inaccurate, unfair, or unreliable outcomes when the AI is used in practice.

Give an example of selection bias seen in AI

If an AI model for skin cancer detection is trained mostly on images of light-skinned individuals, it may perform poorly on darker skin tones

Confirmation bias

Occurs when the system tends to prioritize or reinforce patterns that already exist in the training data, potentially ignoring alternative patterns or explanations.

Give an example of confirmation bias seen in AI

If an AI used for hiring decisions is trained on data from a company that historically hired mostly men for leadership roles, it may "learn" to favor male candidates—reinforcing the existing pattern rather than evaluating all candidates fairly.

Recall bias

Occurs during data labeling, where inconsistencies creep in due to subjective interpretations of the data

Give an example of recall bias seen in AI

A tool for sentiment analysis might struggle to distinguish sarcasm or cultural nuances in text if the training data relied on subjective labeling of emotions

Group Attribution bias

Occurs when the system makes assumptions about an individual based on characteristics or patterns associated with the group they belong to, rather than assessing them as a unique individual.

Give an example of group attribution bias seen in AI

An AI system used in credit scoring may lower the creditworthiness of a person simply because they belong to a zip code or demographic group that, on average, has lower credit scores

Implicit bias

Refers to unconscious prejudices that are either embedded in the training data or introduced by the developers themselves.

Give an example of implicit bias seen in AI

An AI resume screening tool might favor resumes with traditionally masculine language over those with more neutral wording.