How do fictional stories illustrate AI misalignment?
There are many fictional stories that depict unaligned non-human entities in ways that can illustrate some aspects of what AI misalignment might look like.
Many traditional fables about "being careful what you wish for" — such as the stories of King Midas, The Sorcerer’s Apprentice, and some traditional Jewish Golems — hinge on a character getting what they literally ask for, but failing to anticipate the full consequences or side effects. These stories illustrate how the complexity and fragility of human values make outer misalignment a real danger, and how a genie-like superintelligence that did not understand what we really want (or tried to actively subvert it, in the case of specification gaming) could lead to catastrophic consequences.1
Some stories (Mission: Impossible – Dead Reckoning, 2001: A Space Odyssey, Ex Machina) portray AIs that have been tasked with seemingly beneficial or morally neutral aims but are misaligned in important ways from their creators.
Some stories (Mission: Impossible – Dead Reckoning, The Terminator) portray AI that very explicitly attempts to take over the world, whereas others (Ex Machina, Upgrade, 2001: A Space Odyssey) have AI with more restricted or ambiguous aims that still bring it into conflict with its creators or users. World takeover is generally not the AI's final goal but comes about as an instrumentally convergent goal.
Isaac Asimov’s characters attempt to constrain the behavior of robots with the Three Laws of Robotics, but Asimov’s intent was to explore how these laws are insufficient to guarantee a good outcome.
The Terminator movies are often used as a pop-culture reference for AI risk. Many in the field dislike when AI risk is illustrated this way because pop culture views Skynet as malicious rather than simply amoral. The movies also involve android killer robots, which are unlikely.2 Others have pushed back3 and argued that there are some relevant aspects of these movies, including:
-
Skynet understands that humans are a threat to its goals and attempts extermination as an instrumental goal.
-
Competitive dynamics make the rise of Skynet hard to avoid, mirroring the AI arms race.
-
Skynet fighting back when humans try to disable it illustrates how not being shut down is an instrumentally convergent goal.
We should avoid generalizing from fictional evidence and take these comparisons as intuition pumps rather than arguments. These exact scenarios are unlikely, and there are possible misalignment scenarios that could be very dangerous but would not make for interesting media and thus are not covered as much in such stories.
Some have argued that this kind of misalignment is less likely because LLMs appear to understand human intent, but this is a point of active debate in the field. ↩︎
Why on earth would you make a killer robot in human form? It’s so inefficient! A true superintelligence would never clothe itself in the form of the inferior meat-bags. ↩︎
People who have pushed back include Matt Yglesias, Skluug, and Hein de Haan. ↩︎