Isn't the real concern misuse?

Misuse of advanced AI is an important concern, but catastrophes may happen even without misuse.

Transformative technologies often have a wide range of applications. Just as fire can be used to cook food or to burn down a village, and nuclear reactions can be used to produce electricity or to power terrible weapons, AI technology can be used to achieve both good and bad aims.

Misuse of AI has the potential to cause harm at the individual, societal, and civilizational levels. For example:

One of the aims of AI governance is to find ways to coordinate and prevent such harms.

However, misuse is not the only problem. Even when used with the best of intentions, technology can cause accidents: a fire can spread to burn down a city, and a reactor can melt down. One way in which advanced AI differs from other technologies is that it could act on its own goals, and we do not currently know how to reliably direct sufficiently advanced systems toward any specified goal at all. With future advances, AI could become powerful enough to outmaneuver humanity, and accidents resulting from a failure to solve the alignment problem could threaten human civilization as a whole.

In other words, for things to go well, we would need to both:

  1. Coordinate to avoid misuse by bad actors.

  2. Ensure powerful AIs are aligned.