Can an AI be smarter than humans?

The human brain was developed by a process with zero foresight, trying random changes and keeping what seems to work in practice, layering ad-hoc hacks on top of ad-hoc hacks, like the worst spaghetti code ever. It’s subject to severe design constraints, being limited to about 20 watts of power, needing to grow from something that can fit through a birth canal and be specified with very little data (the human genome is only around 700MB, most of which is used for things other than the brain). It would be extremely surprising if the brain were the peak of physically possible intelligence. The brain doing things in complex ways is not strong evidence that we need to fully replicate this complexity to get intelligence. We may just be able to throw sufficient compute at the problem.

AI researchers have been consistently improving both the optimization power and generality of AI algorithms and there is little reason to expect progress to suddenly stop (see also scaling laws). We’ve gone from coding systems specifically for certain games (like chess), to algorithms like MuZero which learn the rules of the game they’re playing and how to play at vastly superhuman skill levels purely via self-play across a broad range of games (e.g., Go, chess, shogi and various Atari games).

It is, however, plausible that for an AGI we need a lot more compute than we will get in the near future, or that some key insights are missing which we won’t get for a while. The Open Philanthropy report on how much computational power it would take to simulate the brain is the most careful attempt at reasoning out how far we are from being able to do it, and suggests that by some estimates we already have enough computational resources, and by some estimates Moore’s law may let us reach it before too long.

It also seems that much of the human brain exists to observe and regulate our biological body, which a body-less computer wouldn't need. If that's true, then a human-level AI might be possible with considerably less compute than the human brain.