Lecture 12, Feb 12

Existential Risks

Definition of existential risk

Existential risk: Event that threatens the destruction of humanity’s potential, either by making humanity go extinct or by putting humanity on a path toward a dystopian future with no way back. (Ord, 2020)

1) Existential risk is not only extinction

• Dystopian futures

  2) Not reaching technological maturity is an existential risk (a dystopian future)

• Living in a preagricultural state forever would also be an existential risk

• Space settlement / digital consciousness could multiply trillions of times the number of people

  3) Global catastrophes are not existential risks

• A temporary catastrophe (e.g., 4 billion deaths) is not an existential risk

• If we can bounce back from it, it is not an existential threat

4) Two sources of risks

Natural risks (Almost negligible)

• A supervolcanic eruptions

• A large asteroid impacts

• The explosion of a supernova

  Anthropogenic risks: risks stemming from human technologies (1/6 risk this century)

• Engineered pandemics

• AI takeover

• Climate change

• Nuclear wars

Tech: Danger & Solution

Technology is the central theme for longtermism

Main source of danger:

• Techno-determinism

• Technophobes and technophiles agree

Main source of solution

• Techno-conservative vs techno-progressive

• Speeding up technological development of protective measures

Growth of moral wisdom and technology

• Our moral development did not follow the rapid development of technology

Climber analogy:

Imagine a free-climber on a wall (not attached). The climber feels

that she might be in danger. In this case, just stopping to climb would make her

eventually fall anyway. She cannot climb down the wall, it is even harder to do so

safely. The only way to safety is up. Humanity is in a similar situation. If we remain at

the same level of development, we will just destroy humanity with the level of

technology we have already developed. Going down is also not an option because

when a dangerous technology has been invented, it is always available for people to

use it for bad. We thus have to focus on new defensive technologies.