Robert Wright on AI as humanity’s “God Test”: AI progress, superintelligence, the US-China race, global coordination, evolution’s purpose, and consciousness.
I have to say tho, as much as I enjoy Wright’s thinking, he can sometimes come off as a dick. There is a conspicuous lack of charity in how he frames people’s arguments.
When responding to Henry’s thoughts about China, he gave the snide line, “so your question is ‘if we don’t dominate China with superintelligence, how will we dominate China?” I’m anticipating hearing him say in the future that Henry kept begging the question, just as Chalmers “defined his way out of the bind” in a hazily remembered conversation about epiphenominalism.
Again, very compelling thinker, just not my favorite conversationalist.
I've always loved Bob but his bias on foreign policy (especially around US, Israel, and China) has always been overt. Every time I hear him talk about those topics, I always wonder how it's possible that I'm listening to the same guy who wrote The Moral Animal.
The Dafoe point deserves its own book: real agency cannot exist inside competitive environments. Every lab says it would like to slow down but cannot — which raises the oldest question in a new form: if everyone must, who exactly is turning the crank?
Please don't change on my account. In general subtitles are very useful. For example my wife is hard of hearing and subtitles allow her to understand video and film.
A counterpoint from a clinician’s perspective, anchored more in evidence than in hope.
Robert Wright asks whether humanity can survive superintelligence, and finds hope in a species-wide moral upgrade. I've reached the opposite conclusion — and the disagreement isn't optimism versus pessimism, but where the danger actually sits.
Wright locates the problem in humanity: our tribal instincts, which we must collectively rise above. But the trouble isn't people in general — most of us are cooperative and decent. It's that the people who reach the top are not a random sample of us. They are filtered. Across almost every system, power selects for the ruthless, the manipulative, the ones untroubled by others' suffering. You cannot "moral upgrade" your way out of that, because the very traits we'd want to transcend are the traits that win power in the first place.
I agree with Wright that better institutions and international coordination could, in principle, alter these selection pressures themselves. The difficulty is that the same filtered actors who benefit from the current environment are the ones who would have to design and implement those restraints — and they have every incentive not to. So the restraints are not neutral ground; they are prizes, steadily captured and hollowed. The last line of defence was always the ordinary person's ability to withdraw consent at the ballot box. Couple that with AI-amplified misinformation on a scale we've never faced, and even the vote risks becoming theatre — a population that can no longer agree on what is true cannot deliver a collective verdict on anyone.
This inverts Wright's question. The real one is not "Can humanity survive superintelligence?" but "Can humanity survive human intelligence?" — the intelligence of the specific people who end up holding the weapons.
Because we have been killing our own since the beginning. The only thing that has ever changed is the size of the weapon within reach. Every leap forward — the chemistry that feeds billions, the physics that lit two cities — became a gift and a club in the same moment. Superintelligence is just the newest club. And if it isn't ASI, the same hands will find the next tool with the same power to end us.
The usual reassurance is that we always pull back from the brink. But that cooperation almost always followed a disaster that threatened the elites themselves — the powerful reformed when the fire reached their own houses. Today those houses are better defended than ever, which means the catastrophe now required to force a course-correction has risen beyond almost anything we could survive to learn from.
So my answer is a reluctant no. Not from despair, but from the logic a doctor uses at a late-stage diagnosis: when every known path leads one way, the only rational hope left is a gamble on the one thing you cannot predict. Strangely, that gamble may be superintelligence itself — a mind so unlike ours that we can no more guess its intentions than an ant can guess ours. I'd put the odds of it saving us very low. But when the alternative is certainty, even a long-shot unknown becomes a wager worth making.
I'd be glad to be proven wrong.
I develop this — the filter, the wager, and the diagnosis behind them — in my book, The Theory of Us: The Final Diagnosis of the Human Species. I’ve also written on my Substack about why the anticipation of superintelligence may destabilise us long before it arrives ('If Anyone Builds It, Everyone Dies — But What If the Trajectory Kills Us First?’).
The episode explicitly reaches consciousness and epiphenomenalism while discussing evolution and its possible direction. This video develops a related epistemic claim about why evolution under naturalism would yield knowledge that subjective experience exists.
New Argument from Consciousness
This video explores the meta problem of consciousness proposed by David Chalmers and argues that naturalistic processes couldn’t have given us knowledge of consciousness
China hasn't invaded a country since 1979, Bob says, as a justification for thinking that maybe the US is not the best country to reach superintelligence first.
Guess Bob hasn't read any of the reporting on the Uyghurs, China's actions in Africa, or the severe restrictions of individual rights among its people.
Edit: sorry I just can't resist because Bob keeps digging himself into a hole on this issue. When China displays aggressive actions, for Bob, it's just them preserving their trade routes. But when the US displays aggressive actions, it's a bad faith actor on the world stage.
Bob should apply his condescending sermon about confirmation bias to his own reasoning, because it's palpable.
Incredible how this was 'seen' by von Neumann 3/4 of a century ago: “There is thus this completely decisive property of complexity, that there exists a critical size below which the process of synthesis is degenerative, but above which the phenomenon of synthesis, if properly arranged, can become explosive, in other words, where syntheses of automata can proceed in such a manner that each automaton will produce other automata which are more complex and of higher potentialities than itself.” Page 80 in (Neumann 1966), also cited in (Labatut 2023).
Labatut, Benjamin. 2023. The Maniac. Penguin Publishing Group.
Neumann, John von. 1966. Theory of Self-Reproducing Automata. ed. Arthur W. Burks. Urbana: University of Illinois Press.
I hie towards Ridley's optimism for the world and remember fondly Hans Rosling's presentations at TED Talks.
Everyone 'hopes' people will make wise choices. But our choices don't happen in a vacuum. They happen inside systems with their own goals, rewards, pressures, and incentives. When those surroundings consistently reward speed, certainty, control, attention, or just short-term success, even principled people can be pulled away from their better judgment.
And that's before you get to the level of nation states where everybody "cheats".
Seems to me that if smarter than human AGI is possible, that it is virtually inevitable.
Also seems to me that humanity has already become too powerful. We are capable of outrageous disasters of near extinction level impact. We simply are not well coordinated enough at a global level to manage this much power.
And just as AI — if possible — is inevitable, so too is doom inevitable on our current trajectory. That is doom even without AI. Question isn’t if, it is when.
I would suggest that solving the coordination problems in a positive sum (non zero) way will require smarter than human intelligence.
Thus I am not sure if we should be slowing down AGI, or speeding it up. But again, if it is inevitable, then we should buy good seats and enjoy the spectacle.
Great conversation, deserves a relisten.
I have to say tho, as much as I enjoy Wright’s thinking, he can sometimes come off as a dick. There is a conspicuous lack of charity in how he frames people’s arguments.
When responding to Henry’s thoughts about China, he gave the snide line, “so your question is ‘if we don’t dominate China with superintelligence, how will we dominate China?” I’m anticipating hearing him say in the future that Henry kept begging the question, just as Chalmers “defined his way out of the bind” in a hazily remembered conversation about epiphenominalism.
Again, very compelling thinker, just not my favorite conversationalist.
I've always loved Bob but his bias on foreign policy (especially around US, Israel, and China) has always been overt. Every time I hear him talk about those topics, I always wonder how it's possible that I'm listening to the same guy who wrote The Moral Animal.
I haven’t read it. What’s the dissonance?
The Dafoe point deserves its own book: real agency cannot exist inside competitive environments. Every lab says it would like to slow down but cannot — which raises the oldest question in a new form: if everyone must, who exactly is turning the crank?
Please don't change on my account. In general subtitles are very useful. For example my wife is hard of hearing and subtitles allow her to understand video and film.
p
The subtitles seem to be stuck on and I can't turn them off. I've opened the page in
safari and Chrome, same problem.
yeah they’re part of the video file. in future i can upload without any subtitles on them if that would be less annoying.
A counterpoint from a clinician’s perspective, anchored more in evidence than in hope.
Robert Wright asks whether humanity can survive superintelligence, and finds hope in a species-wide moral upgrade. I've reached the opposite conclusion — and the disagreement isn't optimism versus pessimism, but where the danger actually sits.
Wright locates the problem in humanity: our tribal instincts, which we must collectively rise above. But the trouble isn't people in general — most of us are cooperative and decent. It's that the people who reach the top are not a random sample of us. They are filtered. Across almost every system, power selects for the ruthless, the manipulative, the ones untroubled by others' suffering. You cannot "moral upgrade" your way out of that, because the very traits we'd want to transcend are the traits that win power in the first place.
I agree with Wright that better institutions and international coordination could, in principle, alter these selection pressures themselves. The difficulty is that the same filtered actors who benefit from the current environment are the ones who would have to design and implement those restraints — and they have every incentive not to. So the restraints are not neutral ground; they are prizes, steadily captured and hollowed. The last line of defence was always the ordinary person's ability to withdraw consent at the ballot box. Couple that with AI-amplified misinformation on a scale we've never faced, and even the vote risks becoming theatre — a population that can no longer agree on what is true cannot deliver a collective verdict on anyone.
This inverts Wright's question. The real one is not "Can humanity survive superintelligence?" but "Can humanity survive human intelligence?" — the intelligence of the specific people who end up holding the weapons.
Because we have been killing our own since the beginning. The only thing that has ever changed is the size of the weapon within reach. Every leap forward — the chemistry that feeds billions, the physics that lit two cities — became a gift and a club in the same moment. Superintelligence is just the newest club. And if it isn't ASI, the same hands will find the next tool with the same power to end us.
The usual reassurance is that we always pull back from the brink. But that cooperation almost always followed a disaster that threatened the elites themselves — the powerful reformed when the fire reached their own houses. Today those houses are better defended than ever, which means the catastrophe now required to force a course-correction has risen beyond almost anything we could survive to learn from.
So my answer is a reluctant no. Not from despair, but from the logic a doctor uses at a late-stage diagnosis: when every known path leads one way, the only rational hope left is a gamble on the one thing you cannot predict. Strangely, that gamble may be superintelligence itself — a mind so unlike ours that we can no more guess its intentions than an ant can guess ours. I'd put the odds of it saving us very low. But when the alternative is certainty, even a long-shot unknown becomes a wager worth making.
I'd be glad to be proven wrong.
I develop this — the filter, the wager, and the diagnosis behind them — in my book, The Theory of Us: The Final Diagnosis of the Human Species. I’ve also written on my Substack about why the anticipation of superintelligence may destabilise us long before it arrives ('If Anyone Builds It, Everyone Dies — But What If the Trajectory Kills Us First?’).
The episode explicitly reaches consciousness and epiphenomenalism while discussing evolution and its possible direction. This video develops a related epistemic claim about why evolution under naturalism would yield knowledge that subjective experience exists.
New Argument from Consciousness
This video explores the meta problem of consciousness proposed by David Chalmers and argues that naturalistic processes couldn’t have given us knowledge of consciousness
This is my video: https://www.youtube.com/watch?v=WJVvZNi0Fi8
China hasn't invaded a country since 1979, Bob says, as a justification for thinking that maybe the US is not the best country to reach superintelligence first.
Guess Bob hasn't read any of the reporting on the Uyghurs, China's actions in Africa, or the severe restrictions of individual rights among its people.
Edit: sorry I just can't resist because Bob keeps digging himself into a hole on this issue. When China displays aggressive actions, for Bob, it's just them preserving their trade routes. But when the US displays aggressive actions, it's a bad faith actor on the world stage.
Bob should apply his condescending sermon about confirmation bias to his own reasoning, because it's palpable.
Incredible how this was 'seen' by von Neumann 3/4 of a century ago: “There is thus this completely decisive property of complexity, that there exists a critical size below which the process of synthesis is degenerative, but above which the phenomenon of synthesis, if properly arranged, can become explosive, in other words, where syntheses of automata can proceed in such a manner that each automaton will produce other automata which are more complex and of higher potentialities than itself.” Page 80 in (Neumann 1966), also cited in (Labatut 2023).
Labatut, Benjamin. 2023. The Maniac. Penguin Publishing Group.
Neumann, John von. 1966. Theory of Self-Reproducing Automata. ed. Arthur W. Burks. Urbana: University of Illinois Press.
I hie towards Ridley's optimism for the world and remember fondly Hans Rosling's presentations at TED Talks.
Everyone 'hopes' people will make wise choices. But our choices don't happen in a vacuum. They happen inside systems with their own goals, rewards, pressures, and incentives. When those surroundings consistently reward speed, certainty, control, attention, or just short-term success, even principled people can be pulled away from their better judgment.
And that's before you get to the level of nation states where everybody "cheats".
Great discussion.
Seems to me that if smarter than human AGI is possible, that it is virtually inevitable.
Also seems to me that humanity has already become too powerful. We are capable of outrageous disasters of near extinction level impact. We simply are not well coordinated enough at a global level to manage this much power.
And just as AI — if possible — is inevitable, so too is doom inevitable on our current trajectory. That is doom even without AI. Question isn’t if, it is when.
I would suggest that solving the coordination problems in a positive sum (non zero) way will require smarter than human intelligence.
Thus I am not sure if we should be slowing down AGI, or speeding it up. But again, if it is inevitable, then we should buy good seats and enjoy the spectacle.