CHAI-Assurance-Standards-Guide-6-26-2024

Assurance Standards Guide

Overview

• The Coalition for Health AI (CHAI) has published an Assurance Standards Guide aimed at establishing comprehensive guidelines for the development and deployment of AI solutions in healthcare.

• The document is protected under copyright laws, and reproduction or distribution requires prior written permission from CHAI.

Draft Versions Released

• v0.1 Draft 1: Released on 5/8/2024, incorporating feedback from Editorial/Workgroup Leads.

• v0.2 Draft 2: Released on 6/14/2024, with feedback from the Independent Review Group and HHS/ONC.

• v0.3 Draft 3: Released for public comment on 6/26/2024, incorporating feedback from the Board of Directors.

• More drafts will follow until a final version is released.

Purpose of the Guide

• This comprehensive Guide seeks to integrate best practices for stakeholders involved in AI solutions in healthcare. Future editions will focus on:

  1. Identifying actions and responsibilities in the AI lifecycle.

  2. Distinguishing between AI models developed internally and those sourced externally.

  3. Illustrating considerations within existing regulations, clarifying overlaps with current guidelines.

Contributors

• Writing Team: Includes members such as Nicoleta Economou, Matthew Elmore, and Alison Callahan.

• Contributions come from various workgroups focusing on usefulness, fairness, safety, transparency, and privacy in AI.

• Diverse independent reviewers also provided feedback for validation and accuracy.

Table of Contents Overview

• The document includes several sections, including a Preface, Summary, Introduction, the AI Lifecycle, Core Principles, Independent Review, Best Practices for Trustworthy Health AI, Pathway for Continuous Learning, References, Glossary, and Appendices.

Preface

• The Guide is designed to be a comprehensive playbook for ethical and quality assurance in AI development in healthcare.

• It aims to educate all stakeholders on the considerations for developing and deploying AI technologies effectively.

• A companion document, the Assurance Reporting Checklist (ARC), is also available for detailed evaluations.

Core Principles for Trustworthy Health AI

• The guide is founded on core principles that include:

  1. Usefulness, Usability, and Efficacy: AI solutions must provide benefits for patient care and healthcare operations.

  2. Fairness and Equity: AI must not introduce biases; should result in parity across subgroups.

  3. Safety and Reliability: Ensure safety in AI predictions to avoid risks to health and care.

  4. Transparency, Intelligibility, and Accountability: The workings of AI solutions, including guidelines and outcomes, must be understandable and transparent.

  5. Security and Privacy: Must protect patient data effectively against unauthorized access.

The AI Lifecycle

Stages of the AI Lifecycle

• Stage 1: Define Problem & Plan: Understand the issue AI aims to address, evaluate needs, and set expectations.

• Stage 2: Design the AI System: Establish technical requirements and ensure user-focused design.

• Stage 3: Engineer the AI Solution: Create an AI solution that performs as expected with tested data.

• Stage 4: Assess: Validate AI solutions pre-deployment through rigorous evaluations.

• Stage 5: Pilot: Conduct preliminary tests to assess real-world effectiveness.

• Stage 6: Deploy & Monitor: Broader implementation followed by continuous monitoring and evaluation.

Importance of Independent Review

• Independent quality assurance is crucial before AI solutions are deployed in healthcare to ensure safety and efficacy, addressing potential biases and errors effectively.

Implementing Best Practices for Trustworthy Health AI

• Best practices encompass evaluating AI systems during each lifecycle stage based on the core principles established, ensuring comprehensive risk management and stakeholder alignment.

Pathway for Continuous Learning

Emphasizes the need for ongoing adaptation of AI solutions to maintain alignment with patient needs and ethical standards, enhancing usability, and fostering trust.

Conclusion

• The CHAI Assurance Standards Guide serves as a crucial framework for AI development and deployment in healthcare, striving to enhance patient care through trustworthy and ethical AI practices.