Why would a misaligned superintelligence kill everyone?

While AI is unlikely to be malevolent towards humanity, we might still die as a result of the AI doing instrumental reasoning, with our deaths being either 1) an intentional goal or 2) a side-effect of some other goal:

• As an intentional goal: The AI might see a risk of humans interfering with its goals, e.g., by trying to turn it off or by building a rival superintelligent AI. The AI might attempt to remove this risk by killing us.

• As a side-effect: Whatever a superintelligent AI’s goals are, it’s unlikely that they would be best served by keeping the Earth broadly in its current state, unless the AI specifically values preserving the status quo (e.g., for the sake of humans and other life). Just like we flood large areas to build reservoirs for dams, fundamentally changing the ecosystem, an AI might undertake large-scale projects that make the world unlivable for humans, or that use the materials lying around as resources. This material could include humans themselves: “The AI does not hate you, nor does it love you, but you are made out of atoms which it can use for something else.” But in such a scenario, our environment would probably have already stopped supporting human life.

In the extreme, this might lead to absolutely everybody dying, i.e., human extinction.