Case Studies in AI Governance and the Rebuttal to Robinson
David Robinson’s Vision for AI Governance
During a presentation regarding a kidney donation algorithm, David Robinson detailed a specific vision for the governance of AI systems.
Central to this vision is the concept of the "whiteboard," which represents the execution plan developed during the creation of an AI system.
The whiteboard contains:
The intended functions of the AI system.
Specific technical details regarding the algorithm's construction and logic.
Robinson posits that effective governance is achieved when regulators or oversight bodies gain access to ‘the room where it happens’ to review the contents of this whiteboard.
This approach is modeled after the perceived success and transparency of the kidney donation algorithm.
Critique of the Whiteboard Assumption
The feasibility of using the ‘whiteboard’ method for universal AI governance depends on two critical assumptions that are not always applicable in real-world scenarios:
Organizations developing AI must be willing to trust outside entities with access to their internal development environments and strategic ‘whiteboards.’
The information discovered on the whiteboard must be demonstrably helpful or actionable for those outside actors.
There is a significant legal and structural divide between public and private entities:
Public services or city governments may be legally compelled to reveal their decision-making algorithms.
Private entities face a more complex situation where transparency may conflict with business interests.
Transparency, Profit, and Auditing
Private companies are frequently disincentivized from sharing algorithmic details because their primary goal is to generate profit from the products enabled by those algorithms.
There are high-stakes risks associated with the exposure of powerful systems, such as large language models like GPT; releasing such systems without rigorous consideration could result in widespread misuse.
Third-party algorithmic auditing is a proposed solution to balance transparency and proprietary secrets:
Auditors can check for harmful biases.
This process does not require the company to expose its proprietary code to the general public.
Criteria for Beneficial AI Systems
AI systems should be required to pass specific tests and meet criteria established by both experts and the public to ensure they are beneficial to humanity. Key evaluation criteria include:
Accuracy: Maintaining high performance across the specific groups of people the algorithm might affect.
Robustness: Ensuring the algorithm does not fail or degrade significantly when its operating environment changes.
Real-World Efficacy: Verifying that the system works properly and as intended in practical, real-world applications.
The Problem of Explicability in AI
Nigam Shah suggests that the requirement for an explanation depends on the context of the AI's use:
Systems used for hiring decisions require deep, detailed explanations to ensure fairness.
Systems assisting doctors in medical decision-making may only need to prove they are effective, regardless of whether the internal mechanisms are fully understood.
Explaining AI reasoning is inherently difficult due to:
Extreme technical complexity.
The lack of a standardized, universal definition of what constitute an ‘explanation.’
AI and humans do not share a common language for experience:
An algorithm trained to distinguish between cats and dogs perceives specific image properties that assist its decision.
These properties are often difficult for a human to decipher or understand in human terms.
Adrian Weller and the BBC suggest that it is often more important for an algorithm to operate reliably, safely, and without discrimination than for it to be explicable.
If a company is unable to explain an algorithm to the satisfaction of its stakeholders, the situation enters a ‘grey area’ that may necessitate resolution through the court system.
Case Study 1: Algorithmic Infrastructure Allocation in City I
Scenario: City I attempted to use an algorithmic system to distribute a limited annual budget for infrastructure spending across various city sectors.
Motivation:
The city government lacked confidence in its existing manual allocation schemes.
They believed software would be more cost-effective than hiring a team of experts.
They sought to rectify a historical legacy of racial and social discrimination in resource distribution.
Implementation:
The city contracted a local company, I Software, to build the system.
The system utilized city-provided data and specifications regarding public transportation and road improvements.
A government task force served as the liaison between I Software, the government, and the public.
Because the algorithm did not involve proprietary intellectual property owned by the firm, the data and implementation details were shared fully with the task force.
Primary Metric: The task force instructed I Software to design the algorithm to maximize economic output, specifically measured by Gross Domestic Product (GDP).
Limitations and Failures of the GDP Maximization Algorithm:
Initial results showed an overall increase in GDP, but the task force discovered the budget was being allocated almost exclusively to areas that already possessed robust infrastructure.
Public Feedback: During a public conference, citizens argued that neglected areas represented unrealized economic potential that the algorithm ignored.
Data and Temporal Flaws: The algorithm utilized a short-term metric, measuring the increase in GDP exactly year after a project began.
Skewed Data: There were very few data points regarding the economic output of investments in neglected areas, leading to a biological loop of underinvestment.
New Findings:
Historical data revealed that consistent investment in fixing roads in neglected areas produced a return on investment that exceeded other parts of the city when measured over a period of years.
Causality and Human Factors:
The algorithm processed historical data for correlations but was incapable of detecting causality.
It failed to account for the human factor: improved infrastructure (such as parks, roads, and playgrounds) improves citizen morale, which leads to higher productivity and increased local economic engagement.
Conclusion: The city concluded that their goal was too multifaceted for the algorithm, which could not incorporate the deep research required for long-term health. They determined that hiring human experts was a better investment than the unreliable algorithm, despite the higher cost.
Case Study 2: Proprietary Lung Cancer Detection
Scenario: Company A developed a proprietary algorithm for identifying lung cancer, claiming the system was more accurate than a panel of human experts.
Partnership: Hospital B wanted to implement the system but required a deep understanding of its performance, even though the code was proprietary.
Performance Metrics: The hospital focused on two critical error types:
False Positives: Incorrectly predicting cancer, which leads to unnecessary, invasive surgery.
False Negatives: Incorrectly predicting a lack of cancer, which is more dangerous because it misses a life-threatening condition.
Testing Method: Company A provided an API (Application Programming Interface). This allowed doctors to test the algorithm on their own internal patient data without accessing or seeing the proprietary internal code.
Validation Process:
Doctors established baseline true positive and true negative rates.
They created a representative dataset that included various races, genders, and health conditions.
The algorithm performed reasonably well, nearly meeting the specific thresholds set by the doctors.
Privacy Regulations and HIPAA Challenges:
Hospital B intended to provide patient data to ‘fine-tune’ the algorithm, which raised concerns regarding the Health Insurance Portability and Accountability Act (HIPAA).
Regulatory Gap: HIPAA was established in and does not explicitly account for modern AI healthcare applications.
The Department of Health and Human Services (HHS) has stated that more guidance is required regarding de-identification methods for AI use.
Resolution: Company A obtained an exemption by proving the effectiveness of their de-identification process through a third-party vendor. This vendor attempted to retrieve sensitive information from the scrubbed data and failed, verifying the security of the process.
Conclusion: While the specific case between Company A and Hospital B was resolved, it highlights the need for outdated policies like HIPAA to catch up to the realities of modern AI to facilitate beneficial use while maintaining data security.