1/190
Looks like no tags are added yet.
Name | Mastery | Learn | Test | Matching | Spaced | Call with Kai |
|---|
No analytics yet
Send a link to your students to track their progress
Moral Responsibilities in Science
Responsibilities in the Practice of Science
No fraud
Avoid QRPs
Careful reporting of data
Careful development of criticisms
Responsibilities of the Scientific Community
Respectful debate
Supporting diverse members of the scientific community
Mentoring obligations
Responsibilities to Society
Good treatment of human subjects
No new destructive capacities?
Don’t make the world worse!
Responsibilities in the Practice of Science (Moral Responsibilities in Science)
No fraud
Avoid QRPs
Careful reporting of data
Careful development of criticisms
Responsibilities of the Scientific Community (Moral Responsibilities in Science)
Respectful debate
Supporting diverse members of the scientific community
Mentoring obligations
Responsibilities to Society (Moral Responsibilities in Science)
Good treatment of human subjects
No new destructive capacities?
Don’t make the world worse!
The Research Agenda
What do scientists choose to pursue?
Inside the “academic bubble.”
Implications of research for broader society?
What are the forces shaping the research agenda?
Beliefs about responsibilities, scientists' choices
Funding, incentives, prizes
What do scientists choose to pursue? (The Research Agenda)
Inside the “academic bubble.”
Implications of research for broader society?
What are the forces shaping the research agenda? (The Research Agenda)
Beliefs about responsibilities, scientists' choices
Funding, incentives, prizes
The Nature of Responsibility
Causal vs. Moral
Causal: What did I cause?
Moral: What is praise-worthy and blame-worthy?
Many causal responsibilities do not involve moral responsibilities
E.g., events over which one has no control
E.g., events over which no one could have decided better
Some moral responsibilities involve a lack of direct causal action
E.g., a failure to provide care
Causal vs. Moral (The Nature of Responsibility)
Causal: What did I cause?
Moral: What is praise-worthy and blame-worthy?

Many causal responsibilities do not involve moral responsibility (The Nature of Responsibility)
E.g., events over which one has no control
E.g., events over which no one could have decided better
Some moral responsibilities involve a lack of direct causal action (the Nature of Responsibility)
E.g., a failure to provide care
Moral Responsibility
What are we morally responsible for?
What we intend
What is foreseeable
Recklessness
Negligence
Foreseeability bounds
Reasonable person standard
Part of a community’s discussion
Anyone should have seen…
NOT sci-fi speculation
Grounded plausibility

What is foreseeable (Moral Responsibility)
Recklessness
Negligence
Foreseeability Bounds (Moral Responsibility)
Reasonable person standard
Part of a community’s discussion
Anyone should have seen…
NOT sci-fi speculation
Grounded plausibility

Moral Responsibility for Societal Impact in Science
Suppose a scientist creates a virus that escapes the lab and kills people
When are they morally responsible?
If they create it with the intent to kill
If they created the virus under conditions that failed to meet biosafety standards (recklessness)
If they created the virus and accidentally released it because they were sloppy (even with biosafety measures in place)
If they created the virus but didn’t think it would be dangerous
If they accidentally created the virus and accidentally released it

Changes to Freedom and Responsibility in Science
1940-2000
Freedom for scientists meant, in part, freedom of responsibility
Particularly for basic research (vs. applied)
Responsibility for societal impact was thought to hamper the pursuit of knowledge
The attainment of knowledge was worth any impact
2000 - today
Freedom only comes with responsibility
1940-2000 (Changes to Freedom and Responsibility in Science)
Freedom for scientists meant, in part, freedom of responsibility
Particularly for basic research (vs. applied)
Responsibility for societal impact was thought to hamper the pursuit of knowledge
The attainment of knowledge was worth any impact

2000 - today (Changed to Freedom and Responsibility in Science)
Freedom only comes with responsibility

Freedom from —> with Responsibility
AAAS 1975 Edsall Report
For basic researchers: no responsibilities for implications/impacts of work (for the 20th century)
For applied researchers: responsibilities met by institutional constraints
By 2010, the grounds were shifting
Too many exceptions were rising
Dual-use concerns arising across science
On Being A Scientist (2009, 3rd ed.)
“The standards of science extend beyond responsibilities that are internal to the scientific community. Researchers also have a responsibility to reflect on how their work and the knowledge they are generating might be used in the broader society.”
AAAS 1975 Edsall Report (Responsibility from —> with science)
For basic researchers: no responsibilities for implications/impacts of work (for the 20th century)
For applied researchers: responsibilities met by institutional constraints
By 2010, the grounds were shifting (Responsibility from —> with science)
Too many exceptions were rising
Dual-use concerns arising across science
On Being A Scientist (2009, 3rd ed.)
“The standards of science extend beyond responsibilities that are internal to the scientific community. Researchers also have a responsibility to reflect on how their work and the knowledge they are generating might be used in the broader society.”
International Science Council (2021)
“Scientific freedom must come with social responsibility, for scientists in all areas of research.”
Freedom and Responsibility in the 21st Century
Freedom (autonomy) comes with responsibility
The more responsibility a scientist bears, the more freedom/autonomy they have
The more freedom a scientist bears, the more responsibility they have

Bridgman
scientists are NOT responsible for the impacts of their findings
free to do whatever
study what you want to study
RESPONSIBILITY INHIBITS SCIENCE
basic scientists - not worry about impacts
applied scientists - worry about impacts
this is NOT real
Douglas
scientific freedom ONLY comes with responsibility
the more freedom that a scientist has, the more responsibility they hold
responsible for reasonable + foreseeable implications
avoid recklessness
avoid negligence
not seeing / recognizing clear happenings

Responsibility vs. Accountability
Moral responsibility:
Duties and obligations
Floor AND Ideals
Accountability:
What you can be held accountable for
What you are asked to give an account for
Clear mechanisms for implementation
About floors, not ideals
Moral Responsibility (Responsibility vs. Accountability)
Duties and obligations
Floor AND Ideals
Accountability (Responsibility vs. Accountability)
What you can be held accountable for
What you are asked to give an account for
Clear mechanisms for implementation
About floors, not ideals

What are the floors and ideals for the societal impacts of science
Individual minimum floor:
Don’t make the world worse
Individual Ideal:
Make the world better
Communal minimum floor:
Make the world better
Communal Ideal:
Make the world sustainably better
Ways to help scientists with responsibilities
Oversight/accountability mechanisms
Responsibility is partially or fully offloaded to someone else
Problem of checklist culture
Research Ethics Consultations (RECs)
Socio-technical integration protocols (STIR)
Better responsible conduct of research (RCR) education
More than internal to science issues
More than human and animal subject protections
Learning to think about impact in the shaping of the research agenda
Oversight/accountability measures (ways to help scientists with responsibilities)
Responsibility is partially or fully offloaded to someone else
Problem of checklist culture

Better responsible conduct of research (RCR) education (ways to help scientists with responsibilities)
More than internal to science issues
More than human and animal subject protections
Learning to think about impact in the shaping of the research agenda

Wallach’s Research to Practice Pipeline
Researchers
Let's see what we can discover
Someone else will worry about turning work into a product (application)
Applied Scientists
Let’s see if we can make this practice work
Surely the researchers already thought about the impacts of the work
Product Teams
Let’s see if we can turn this into a product
People have already put so much work into making this possible
Marketing
Let’s see who is interested in this
It’s our job to market the product, not to think about the possible harmful effects
Consumers
Let’s see what this does
Surely they won’t market something problematic!
Problem:
No one is taking responsibility for impact; no foresight is exercised
Few are trained to think about the societal impact(s) of their work!
Researchers (Wallach’s Research to Practice Pipeline)
Let's see what we can discover
Someone else will worry about turning work into a product (application)
Applied Scientists (Wallach’s Research to Practice Pipeline)
Let’s see if we can make this practice work
Surely the researchers already thought about the impacts of the work
Product Teams (Wallach’s Research to Practice Pipeline)
Let’s see if we can turn this into a product
People have already put so much work into making this possible
Marketing (Wallach’s Research to Practice Pipeline)
Let’s see who is interested in this
It’s our job to market the product, not to think about the possible harmful effects
Consumers (Wallach’s Research to Practice Pipeline)
Let’s see what this does
Surely they won’t market something problematic!
Problem (Wallach’s Research to Practice Pipeline)
No one is taking responsibility for impact; no foresight is exercised
Few are trained to think about the societal impact(s) of their work!

Dual-Use vs. Dangerous Research
Dual-Use Research
Two uses:
Beneficial discovery
Harmful use
Research is not intended to be harmful, but it could be used to cause harm
Not weapons research!
Requires a malicious actor who uses research for harmful purposes
Dangerous Research:
Research that brings a substantial risk if accidents occur
Does not require malicious actors
Response: manage risks with procedures and materials
Dual-Use Research
Two uses:
Beneficial discovery
Harmful use
Research is not intended to be harmful, but it could be used to cause harm
Not weapons research!
Requires a malicious actor who uses research for harmful purposes

Dangerous Research
Research that brings a substantial risk if accidents occur
Does not require malicious actors
Response: manage risks with procedures and materials

20th Century: rDNA Discovery - Dangerous Research and Biosafety
1970s:
Gene-editing abilities arise
Recombinant DNA (rDNA) technology
They were not very precise (compared with CRISPR)
How should this new technology be governed?
Paul Berg’s Early Experiments
Combining the virus that caused cancer in mice with a bacteriophage
Was going to put it into E. coli
Should he?
1972 Asilomar Conference - Dangerous Research and Biosafety
1974: The National Academy of Sciences raised concerns about rDNA
Should there be a moratorium on such research?
1975: Paul Berg organizes a conference to discuss the issue at Asilomar, CA
The main discussions focus on containment and risk
What kinds of (micro)organisms are being used?
What kinds of changes are being introduced?
Levels of Risk: Minimal, Low, Moderate, High
Different practices and infrastructure are needed at different levels
The Main Discussions Focus on Containment and Risk (1975 Asilomar Conference)
What kinds of (micro)organisms are being used?
What kinds of changes are being introduced?
History of Lab Leaks (a sampling)
1967: Marburg virus outbreak in Germany (31 deaths)
1966-1978: Smallpox lab infections (less than a dozen deaths)
1978: Foot and mouth disease was released from the Plum Island Animal Disease Center in US
2003-2004: SARS infection from laboratory exposure
2004: Another foot and mouth disease outbreak from Plum Island
2007: Foot and mouth disease lab leak outbreak in the UK leads to culling of 2000 animals
2013: H5N1 needlestick accident in the US
2014: Dengue needlestick outbreak in South Korea
History of Dual-Use Research
2001: US Anthrax Attacks
The idea that someone with technical ability could deliberately intend to harm
2005: The National Security Advisory Board on Biosafety (NSABB) formed
Tasked with providing advice on dual-use research in biosciences
Advisory body
2010: The US Select Agent List was created
Studies on particular toxins or pathogens
Studies that could make these entities more dangerous
Biosafety level concerns
Need to get approval for work with “select agents”

Dual-Use Research - Policy Issues
Dual-Use Research → Dual-Use Research of Concern (DURC)
Risks of substantial harm
Risks of easy use
Attempt to circumscribe areas of concern
E.g., Select agent lists in the US
This has not been a successful delineation

Failure of Delineation
Within research on pathogens, what might be dual-use research can shift quickly
As shown with the H5N1 case
No listing of coronavirus before 2019
Lots of other areas of science also have dual-use concerns
Mathematics and cryptography
Quantum computing and cryptography
AI and pathogen creation
Physics and Isotope Separation
Chemistry and chemical weapons
Scientists can quickly discover that their area of research can be weaponized
Within research on pathogens, what might be dual-use research can shift quickly (Failure of Delineation)
As shown with the H5N1 case
No listing of coronavirus before 2019
Many other areas of science have dual-use concerns (Failure of Delineation)
Mathematics and cryptography
Quantum computing and cryptography
AI and pathogen creation
Physics and Isotope Separation
Chemistry and chemical weapons
H5N1 and gain-of-function research
H5N1 Avian Flu
Widespread problem in bird populations
Jumps to humans in close contact with infected poultry
High human mortality rate (up to 60%)
Gain-of-Function Research
Funded by the United States
Trying to make the virus gain a function
Use ferrets (they sneeze)
Similar nasal passages to humans
Can contract the Flu
Result:
Yes, H5N1 could be transmissible
What should be done with this result?
H5N1 Avian Flu (H5N1 and gain-of-function research)
Widespread problem in bird populations
Jumps to humans in close contact with infected poultry
High human mortality rate (up to 60%)
Gain-of-function Research (H5N1 and gain-of-function research)
Funded by the United States
Trying to make the virus gain a function
Use ferrets (they sneeze)
Similar nasal passages to humans
Can contract the Flu
H5N1 and the NSABB
Researchers Fouchier & Kawaoka submit papers to Science and Nature
With details of mutations making the virus transmissible
Journal Editors as NSABB for advice
Should we publish these papers?
Should we withhold some parts?
NSABB’s first decision (2011): Redact some details
Journals accept advice
Moratorium on gain-of-function research self-imposed
NSABB’s second decision (2012): Publish it all
How should we share details with researchers who need to know?
Published papers in full in 2012
Another moratorium on gain-of-function research
Imposed by the NIH
Motivated by the Biosafety protocol lapses in 2014
Patchwork of Current Regulatory Mechanisms
Granting Agency Oversight
Institutional Biosafety Committees (IBCs)
Select Agent Lists
Dual-Use Publication Committees (journals)
The National Science Advisory Board on Biosafety (NSABB)
Export Controls
Gain-of-function oversight?
Or should it be enhanced by potential pandemic pathogens (ePPP) research?
Note: IRBs cannot do this research!
It's not human subject research
How Oversight Can Fail - Horsepox Reconstruction
Professor David Evans at the University of Alberta reconstructs Horsepox from mail-ordered DNA bits
“Did not require exceptional biochemical knowledge or skills, significant funds or significant time” (2016 report)
Funded by a private company, headed by an Edward Jenner enthusiast
Motivation - to create safer vaccines (but they already did exist!)
Evans checked with the University of Alberta and the Canadian government before proceeding
Not illegal to reconstruct Horsepox (it is to reconstruct smallpox!)
Horsepox is not on any select agent list
Dual-Use Dangers
Horsepox is very similar to smallpox
Instructions for reconstructing one are likely to work very well for the other
The genetic sequence for smallpox is already published
Professor David Evans at the University of Alberta reconstructs Horsepox from mail-ordered DNA bits (How Oversight Can Fail - Horsepox Reconstruction)
“Did not require exceptional biochemical knowledge or skills, significant funds or significant time” (2016 report)
Funded by a private company, headed by an Edward Jenner enthusiast
Motivation - to create safer vaccines (but they already did exist!)
Evans checked with the University of Alberta and the Canadian government before proceeding (How Oversight Can Fail - Horsepox Reconstruction)
Not illegal to reconstruct Horsepox (it is to reconstruct smallpox!)
Horsepox is not on any select agent list
Dual-Use Dangers (How Oversight Can Fail - Horsepox Reconstruction)
Horsepox is very similar to smallpox
Instructions for reconstructing one are likely to work very well for the other
The genetic sequence for smallpox is already published
Reasons Scientists Give for Pursuing Dual-Use Research
Harmful use is not what I intended
Importance of moral responsibility - recklessness and negligence
Not just about researchers’ intentions
Responsibility is bound by foreseeability!
The work is inevitable
May not be true
Inevitability is not exculpatory generally
It is someone else’s job to ensure bad outcomes don’t happen
Current oversight mechanisms do not ensure this
If you can’t say whose job it is, it is your job!
It is worth the risks
Who should make this decision?
Harmful Use is Not What I Intended (Reasons Scientists Give for Pursuing Dual-Use Research)
Importance of moral responsibility - recklessness and negligence
Not just about researchers’ intentions
Responsibility is bound by foreseeability!
The Work is Inevitable (Reasons Scientists Give for Pursuing Dual-Use Research)
May not be true
Inevitability is not exculpatory generally
It Is Someone Elses Job to Ensure Bad Things Don’t Happen (Reasons Scientists Give for Pursuing Dual-Use Research)
Current oversight mechanisms do not ensure this
If you can’t say whose job it is, it is your job!
It is worth the risk (Reasons Scientists give for Dual-Use Research)
Who should make this decision?
Challenges for Dual-Use Governance
Scientists have not yet fully embraced societal responsibility
Underlying belief that more knowledge is generally good
Under the presumption that knowledge should be shared
Regulatory mechanisms are NOT built for fully responsible research
Lots of dual-use research does not involve select agent list entities
Lots of dual-use research is NOT bioscience
Publication oversight is often misguided or too late (e.g., horsepox reconstruction)
Restricted sharing systems not in place (10+ years after H5N1)
Science involves Genuine Discovery
We don’t know where the next case of dual-use research will occur
Scientists have not fully embraced societal responsibility (Challenges for Dual-Use Governance)
Underlying belief that more knowledge is generally good
Under the presumption that knowledge should be shared
Regulatory Mechanisms are NOT built for fully responsible research (Challenges for Dual-Use Governance)
Lots of dual-use research does not involve select agent list entities
Lots of dual-use research is NOT bioscience
Publication oversight is often misguided or too late (e.g., horsepox reconstruction)
Restricted sharing systems not in place (10+ years after H5N1)
Science involves Genuine Discovery (Challenges for Dual-Use Governance)
We don’t know where the next case of dual-use research will occur
Weapons Research
Deliberately attempting to produce a weapon
Why would a scientist do this?
Current International Law
Biological Weapons Convention (1972)
Chemical Weapons Convention (1925, 1997)
Radioactive Decay
Radioactive decay products
Alpha (He nucleus)
Beta (an electron)
Gamma (electro-magnetic wave)
Radioactive Decay - What Can You Use It For?
Radioactive decay products to irradiate things
Radium for cancer treatment
Radium for watch dials
Radioactive decay to explore the structure of the atom (and nucleus)
Used alpha particles to discover the presence of the nucleus in 1905
Using neutrons to change the isotope - and potentially produce decay
All small bits, very hard to control
Radioactive Decay Products to Irradiate Things (Radioactive Decay - What Can it Be Used For?)
Radium for cancer treatment
Radium for watch dials

Radioactive decay to explore the structure of the atom (and nucleus) (Radioactive Decay - What Can it Be Used For?)
Used alpha particles to discover the presence of the nucleus in 1905
Using neutrons to change the isotope - and potentially produce decay
Discovery of Fission (1938) - A Field Turns on a Dime
Hahn & Strassman notice an odd chemical signature
Meitner & Frisch interpret the result
Practical extraction of nuclear energy becomes possible
Words spread (among nuclear physicists)
Even in an age without commercial air travel or the internet
When to stop publishing? - The Discovery of Fission (1938)
Backdrop of the coming WWII
Hitler annexed Austria in March of 1938
War will officially begin in September of 1939, but everyone knows it's coming
There was a debate in 1939 on whether or not to stop publishing
Neutron multiplication
Cross-section of carbon (a potential moderator)
Crucial!
Backdrop of the coming WWII (When to stop publishing? - Discovery of Fission, 1938)
Hitler annexed Austria in March of 1938
War will officially begin in September of 1939, but everyone knows it's coming
There was a debate in 1939 on whether or not to stop publishing (When to stop publishing? - Discovery of Fission, 1938)
Neutron multiplication
Cross-section of carbon (a potential moderator)
Crucial!
Nuclear Weapons Primer
Fission - A nucleus splits, releasing massive energy
Key issue: whether or not a chain reaction will occur
Fissionable Isotopes U235 and Pu239
Hanford produces Pu239
Oak Ridge produces U235
Critical Mass
Slow chain reaction (nuclear power)
Using a moderator to slow neutrons down
Fast chain reaction possible (bomb)
Bomb designs
U235 and the gun design
Pu239 and the compression sphere
Fusion - Two nuclei fuse together (H+H = He)
Hydrogen bombs, triggered by fission weapons
Unlimited amount of power
Fission - A Nucleus Splits, Releasing Copious amounts of energy (Nuclear Weapons Primer)
Key issue: whether or not a chain reaction will occur
Fissionable Isotopes U235 and Pu239 (Nuclear Weapons Primer)
Hanford produces Pu239
Oak Ridge produces U235
Critical Mass (Nuclear Weapons Primer)
Slow chain reaction (nuclear power)
Using a moderator to slow neutrons down
Fast chain reaction possible (bomb)
Bomb Designs (Nuclear Weapons Primer)
U235 and the gun design
Pu239 and the compression sphere
Fusion - Two nuclei fuse together (H+H=He) (Nuclear Weapons Primer)
Hydrogen bombs, triggered by fission weapons
Unlimited amount of power

Niels Bohr (1855-1962)
Danish physicist
Founded the Institute for Theoretical Physics at the University of Copenhagen in 1920
Wins the Nobel Prize in 1922 for work on the structure of the atom
Quantized orbitals for electrons around the nucleus
Developed ideas of complementarity, the Copenhagen interpretation of quantum mechanics

Margarethe Bohr (1890-1985)
Studying to be a French teacher
Married Niels Bohr in 1912
Key collaborator, transcriber, and editor for her husband’s work
Had six sons
Harald died at 10 (meningitis)
Christian died at 18 (boating accident)

Werner Heisenberg (1901-1976)
German physicist
Meets Bohr in June of 1922
Appointed as a professor at Leipzig in 1927
Wins the Nobel Prize in 1932 for the development of quantum mechanics

Quantum Mechanics Basics
At a subatomic level, stuff gets weird
Wave-Particle Duality
Light as a photon, as a wave
Electron as a particle, as a wave
Uncertainty Principle (Heisenberg)
Can’t precisely know the location AND speed of an electron
Detecting either requires an interaction (e.g., a photon)
Wave-Particle Duality (Quantum Mechanics Basics)
Light as a photon, as a wave
Electron as a particle, as a wave
Uncertainty Principle (Heisenberg) (Quantum Mechanics Basics)
Can’t precisely know the location AND speed of an electron
Detecting either requires an interaction (e.g., a photon)
Copenhagen’s Story
Heisenberg goes to Copenhagen in September 1941
Denmark is under Nazi German occupation
Heisenberg is in charge of the Nazi bomb program
What does he want to speak to Bohr about? What is said?
The Uncertainty of the Bomb
How much fissile material (U235 and Pu239) was needed to make the bomb?
Question of “critical mass”
How much plutonium is needed critically changes how the bomb is made
Could that much Plutonium even be made?
How big would the explosion be?
Question of the speed of fission, speed for fast neutrons
How feasible would it be to separate U235 from U238?
How much Pu239 could be made in a U238 reactor?
Need a moderator to slow neutrons down
Heavy water or carbon
The cross-section of carbon was kept secret (Germans couldn’t get it right)
Need for cadmium to stop the reaction!
Drop into the reactor to stop the neutrons and stop the reaction from happening
Question of “Critical Mass” (Uncertainty of the Bomb)
How much plutonium is needed critically changes how the bomb is made
Could that much Plutonium even be made?
How much Pu239 could be made in a U238 Reactor? (Uncertainty of the Bomb)
Need a moderator to slow neutrons down
Heavy water or carbon
The cross-section of carbon was kept secret (Germans couldn’t get it right)
Need for cadmium to stop the reaction!
Drop into the reactor to stop the neutrons and stop the reaction from happening
Problems with Interpreting History
Why did Heisenberg go to see Bohr?
Heisenberg only wrote about the event after WWII was over
Would be considered treason in the time of the war
Evidence is spotty and incomplete (for good reasons)
Historians still debate what the evidence means
Parallels with quantum mechanics
Uncertainty
Measurement and key intervention
Why did Heisenberg go see Bohr? (Problems with Interpreting History)
Heisenberg only wrote about the event after WWII was over
Would be considered treason in the time of the war
Evidence is spotty and incomplete (for good reason) (Problems with Interpreting History)
Historians still debate what the evidence means
Parallels with quantum mechanics
Uncertainty
Measurement and key intervention
The Fate of the German Bomb Project
Led by Heisenberg
Incorrect critical mass amount
Weird, given how good a mathematician Heisenberg was
Incorrect understanding of carbon as a moderator
Loss of heavy water from Norway
Loss of talent to the Allies
Aerial bombing pressure
Lack of drive among the scientists left?
Did Heisenberg Control the Project so it Could Fail?
Ensuring Failure?
Asking for too little money
Never telling the Nazis about Plutonium
Not getting an accurate calculation of the critical mass needed
Heisenberg was an INCREDIBLE physicist; he should have gotten it right?
While staying in control of the project
Trying to keep the physicists occupied
Not on the battlefields
Trying to ensure slow failure
Trying to succeed in making a reactor, not a bomb
Ensuring Failure? (Did Heisenberg Control the Project so it Could Fail?)
Asking for too little money
Never telling the Nazis about Plutonium
Not getting an accurate calculation of the critical mass needed
Heisenberg was an INCREDIBLE physicist; he should have gotten it right?