What are the main sources of AI existential risk?

There are several broad dynamics that seem like plausible contributors to the risk of an AI-caused existential catastrophe.

There are a number of ways that AI could end up behaving dangerously:

  • Training processes could produce unintended outcomes, for example a “misaligned mesa-optimizer”
  • We could misspecify our goals, producing an AI that pursues goals we don't want
  • AIs could be misused by people intentionally trying to cause harm

Additionally, there are features of the world that could make avoiding a disaster harder:

  • Insufficient time to solve open technical problems, especially around AI alignment.
  • A lack of coordination between the most important actors, like AI labs and national governments.
  • The acceleration of progress through cheaper computing hardware, algorithmic progress and increased investment

One could also look at different kinds of dangerous uses AI could be put to, like locking in undesirable values or inventing powerful weapons. Different types of errors could persist in an AI even as its capabilities became highly advanced, like incorrect assumptions about metaethics, decision theory, or metaphilosophy.

A post-AGI world could end up with different broad patterns where human values lose influence, like new competitive pressures or concentration of power.



AISafety.info

We’re a global team of specialists and volunteers from various backgrounds who want to ensure that the effects of future AI are beneficial rather than catastrophic.

© AISafety.info, 2022—1970

Aisafety.info is an Ashgro Inc Project. Ashgro Inc (EIN: 88-4232889) is a 501(c)(3) Public Charity incorporated in Delaware.