What is perverse instantiation?

Perverse instantiation is fulfilling instructions in a way that undermines the intended objective.

Think of the many stories about someone who finds a genie and gets to make a wish, but the genie takes the wish literally and fulfills it in a way that undermines the person’s hopes, and may even harm them. For example, one easy way to make someone’s toe stop hurting is to amputate their leg.

The concern is that an AI is likely to fulfill commands in this kind of way. Algorithms need to be specified precisely, and if the goal is misstated therein, the AI may well pursue the programmed goal, indifferent to the intent of the programmers (even though it might be capable of figuring out that intent). So it might pursue its goals without concern for the side effects, even if these are extremely harmful.