IRB and AI: Adapting Research Practices for the Future
IRB and AI: An Interview
Background and Experience
The interviewee has decades of experience in the field of regulatory research.
They have worked at Ball State for fifteen years.
They have been in the research field for approximately 24-25 years.
Their original career background was in military and medicine.
Role at Ball State
The interviewee's office manages a wide array of regulatory fields related to research, excluding financial matters.
Responsibilities include:
Conflict of interest
Biosafety
Biosecurity
HIPAA (as the university's privacy officer)
Export control officer
Several other roles related to research regulations
AI as a Tool
AI is viewed as a new tool, similar to past technological advancements such as laptops and PCs.
Initially, such tools are novel and highly sought after, but eventually become commonplace.
Examples of technological advancements include:
Pens and pencils
Mimeograph machines (early photocopiers, using a drum to roll out copies)
Gutenberg Bible and the printing press (first mass-produced book)
Before the printing press, everything was handwritten.
Technology has consistently advanced humanity, initially being new and later becoming second nature before turning into history.
The Rapid Pace of Technological Advancement
The current rate of technological advancement, particularly with AI, outstrips our ability to keep up.
Modern technology improves rapidly, rendering top-of-the-line products outdated within months.
When considering AI, it's important to recognize that we are currently in the early stages of its evolution.
Computing Power
Modern cell phones possess more computing power than all the computers used in the Apollo moon landings.
The Apollo missions used large, fundamental computers that occupied entire rooms.
Technology consistently evolves and poses new challenges.
AI: A Double-Edged Sword
Technology, including AI, is inherently a tool, with its utility determined by its application.
It can be used to:
Save lives
Advance society
Devastate as a weapon
Just like a hammer can build a house or harm someone.
Examples in Sci-Fi:
R2-D2 and C-3PO (positive)
Skynet (negative)
Consider nanomachines:
Microscopic machines programmed for a specific purpose (e.g., cancer cell elimination)
If reprogrammed, it can be a bio-weapon.
Perspective and Usage
When evaluating AI, consider the perspective and how it is being used.
AI, like any tool, can be helpful or problematic depending on its application.
Virtual meetings have become common, changing how interactions occur rather than what is discussed.
AI has the potential to greatly advance research.
AI in Research
AI can be used to conduct simple surveys and interviews, potentially using a human-like persona.
This could be compared to faculty hiring graduate assistants for surveys.
could increase participant numbers, especially among college students, by allowing participation at any time.
AI and Language
AI could translate questions into numerous languages, increasing the reach of research.
AI's effectiveness depends on the quality of input it receives.
"Garbage in, garbage out" principle applies to AI.
AI relies on machine learning and requires substantial, high-quality input.
AI programmed by professional linguists will produce more natural-sounding language.
Predictive Algorithms
The quality of input determines the output of predictive algorithms.
Example: AI program to predict shark behavior:
Input from horror movies (e.g., Jaws) results in skewed, aggressive predictions.
Input from shark experts produces a more balanced, accurate algorithm.
AI tools may not always be programmed by individuals with sufficient knowledge.
Avoid simply feeding AI everything and hoping it will figure things out.
AI Modeled After Humans
AI is inspired by the human mind, but it requires proper input to develop positively.
Researchers should be aware of the sources used to train AIs (programming, purpose, etc.).
Researchers need to verify and double-check all results obtained from AI.
AI's Limitations
A molecular biologist demonstrated that AI, while useful, can be incorrect.
When asked to create a heart cell, the AI initially produced a heart shape, only later incorporating biological cells while maintaining the heart shape.
AI learns over time, like humans in school.
Human input is essential for AI to learn and for the interpretation of results.
AI Counter-Defense
AI can be used to discern between AI-generated and real images.
Companies offer services that use counter-defense AI to determine the authenticity of pictures, movies, and sound bites.
AI vs. AI
AI can be used to combat AI threats, similar to warfare tactics.
These systems identify and attempt to neutralize threats.
Human input remains crucial, as AI may not detect certain elements.
Honeypots (traps for AI) can be effective, but human oversight is still necessary.
Human analysis (e.g., survey completion time, canned responses) is essential in identifying illegitimate submissions.
Evolution of AI Deception
AI will become more sophisticated, making it harder to detect.
Researchers should adapt their thinking and approaches.
The two-email verification system is one adaptation.
Researchers should educate themselves on available tools and technologies.
IT security experts can provide valuable assistance.
Sharing detection methods is crucial in helping each other.
Integrity in Research
Integrity is essential for meaningful research.
AI can speed up analysis, but it is not infallible.
Learning from mistakes is crucial.
Guidance and education can help prevent future attacks.
Combining AI with human skills provides the best outcome.
The IRB Perspective
The IRB aims to promote awareness and provide tools to combat bot attacks and educate researchers on the risks of new technology, bot attacks, and vulnerabilities.
This aims to prevent the devaluation of participant anonymity.
It is crucial to learn from each other's experiences to enhance research integrity and prevent similar attacks.
Financial Incentives
Incentives (e.g., gift cards) can be a significant draw for malicious actors.
In your example, paying out $10 for 3,000 responses is a significant financial opportunity for malicious actors.
Even with a low success rate (e.g., 1 out of 10), malicious actors can still profit significantly.
If a study looking at one hundred thousand participants uses an incentive, malicious actors and bots in attacks may slip through the cracks, still drawing significant money.
Clinical trials can be very expensive, but the potential return means researchers are very careful in this phase.
Anonymity, Confidentiality, and Identifiability
Anonymity, confidentiality, and identifiability are complex and situational concepts.
In-person, recorded interviews are not anonymous.
Mailed responses in unlabeled envelopes offer a higher degree of anonymity.
Emailed responses are not truly anonymous due to email addresses.
The IRB reviews protocols to ensure data collection minimizes identifiable characteristics.
If not necessary to collect, location data and IP addresses are frequently turned off completely.
Truly anonymous data in today's world is a fallacy.
It is the task of review boards is to make connecting information extremely difficult for potential anonymity breaches.
Data Collection
Minimize data collection to only what is needed, avoiding the "shotgun approach".
In Qualtrics, turn off metadata collection if not needed.
Balancing the need for identifiable characteristics to verify responses with the need to protect anonymity.
Informed consent forms with IT security information can provide additional legal protection.
Overly long, technical informed consent forms can deter participation.
Risk Assessment
When evaluating data collection, consider the ultimate risk to participants (e.g., potential harm from leaked data).
Risk assessments change over time.
What is the risk if identifiable information does get leaked or breached or so on?
Researchers frequently only steal a few data points because they are lazy.
There will normally be follow-up intimidation to harvest financial incentives.
Safeguards
Implement reasonable safeguards and best practices.
A major problem is lazy safeguards.
Protect participants by implementing reasonable safeguards.
You can't control what other people do. You can control what you do and try to mitigate what they do.
AIs are always going to be around, and it's best to learn how to mitigate potential risks.
Ethical Considerations and Future Implications
Training AIs to be participants is theoretically possible.
Research has always been about the impossible being made possible.
If you think it, you can make it happen.
There are existing computer and weather programs modeled on a lot of historical data, facts about weather patterns, and temperatures, all trying to predict what happens.
There is no guarantee of 100% accuracy in any case. Predictive may work, but not definitively.
You may never be able to claim that this is a %.
You could never claim complete certainty with AI data that acts like humans.
Watch the remake of Battlestar Galactica about the ethics and practical concerns of the integration of AI and technology in society.
Machines do not have moods or emotional drives that inform behavioral shifts.
Humans can be irrational and change their minds.
It makes the study inaccurate to depend on pure models. Humans are never 100%. Models are never 100%.
Wisdom comes from asking questions.